home-network-k3s-impl/runbooks

Alert runbooks — home-network-k3s prod

This is the index page served at https://home.keithjackson.dev/runbooks/.

Every alert rule in this platform carries an annotation of the form

runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"

46 of them, over 35 names. Until this directory existed, that path returned HTTP 404 on the live cluster: every alert that reached a human handed them a dead link at the exact moment they needed help. These pages are what makes the link resolve.

How to run the commands in these runbooks

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down.

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — k3s-dev until dev was decommissioned on 2026-09-13, k3s-prod now. A command that names its context still means the same thing next month; one that relies on the default does not, and the failure is silent. CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster — read it. Do not run two dev.sh invocations at once; they share one container.

Use fully-qualified resource names, always. Three CRDs on this cluster register the kind Backup — longhorn.io/v1beta2, postgresql.cnpg.io/v1, velero.io/v1 — and the bare short name resolves to Longhorn's, which holds zero objects. kubectl get backup -A therefore prints "No resources found" on a cluster holding 6 Velero and 27 CNPG backups (verified on prod 2026-09-13). Acting on that false negative is how a CLI quirk becomes real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Merging a PR into main is the deploy; there is no other apply step. main is protected, so a direct push is rejected — durable changes go through a branch, a PR, and a green gate.

Alerts you should not trust the silence of (measured 2026-09-13)

The Longhorn and cert-manager scrapes were ENABLED on 2026-09-13, so the Longhorn row below is no longer inert — and re-keying the two robustness rules the same day fixed a SECOND inertness the scrape had left in place (the metric is one-hot over a state label, not a 0–3 enum). Take the Longhorn row as history; the rest of this table still holds, and Flux-controller and Hubble/Cilium rule groups still evaluate against nothing and cannot fire:

Missing scrape job	Inert alerts
Longhorn	LonghornVolumeDegraded, LonghornVolumeFaulted, LonghornDiskAllocationHigh, LonghornDiskUnschedulable
cert-manager	CertificateExpiringSoon, CertificateExpiryImminent, CertificateNotReady
Flux (gotk_*)	FluxReconciliationFailure, FluxResourceSuspended, ImageUpdateAutomationFailing, ImagePolicyStalled, UpdateHealthGateFailed
Hubble (hubble_drop_total)	HubbleDeniedFlowSpike
etcd (kubeEtcd disabled for k3s, deliberately)	no etcd fsync alerting exists at all
per-app oauth2-proxy sidecars (oauth2_proxy_*) — updated 2026-10-01	no authentication alerting exists at all for crafty, jackett, pihole, qbittorrent, radarr, sonarr

The oauth2-proxy row is the newest, and it has moved twice in two days — worth reading in order, because each step looked like the end of it.

2026-09-30 — nothing to scrape. count by (__name__) ({__name__=~"oauth2_proxy.*"}) returned exactly one family, oauth2_proxy_requests_in_flight, from the two replicas of the shared forward-auth instance. All six per-app gate sidecars ran without --metrics-address, so their metrics listener never started, and no ServiceMonitor or PodMonitor selected them.

2026-10-01 — the listener exists, and the scrape is still blocked. PR #28 added --metrics-address=0.0.0.0:44180 and a named metrics port to the five fleet sidecars, and the endpoint genuinely serves (port-forwarded: http=200, 10208 bytes, all three oauth2_proxy_* families, oauth2_proxy_requests_total{code="302"} present). platform/oauth2-proxy/podmonitor.yaml now discovers them by pod label. But the app namespaces' default-deny NetworkPolicy drops the scrape: from a pod in ns monitoring, both :44180 and :4180 on the jackett pod time out, while the shared proxy's :44180 answers — the only difference being that ns oauth2-proxy has no NetworkPolicy at all. So the targets are discovered and up == 0, and the TSDB is still empty for those namespaces. This is the same bug Longhorn had, with the same fingerprint; the fix idiom is platform/longhorn/networkpolicy-metrics.yaml (ns monitoring → one pod label → one port), and for these apps it has to be rendered alongside each app's own policy rather than shipped from the oauth2-proxy tier, which is a sibling of apps-wave-0 in the Flux graph and so cannot rely on those namespaces existing yet.

OAuth2ProxyGateUnobserved (platform/oauth2-proxy/auth-visibility-alerts.yaml) alerts on the gap itself from kube_pod_container_info and up — two families that are scraped. Note its up == 1: a discovered-but-failing target publishes up with value 0, so the earlier bare form would have gone quiet the moment the PodMonitor landed, while nothing was being scraped. That near-miss is the whole reason to distrust a cleared alert here. Crafty remains outside all of this — no metrics listener at all. See oauth2-proxy-auth-failures.

Each affected runbook says so on its own page and gives the by-hand equivalent. See targets-down.

The runbooks

Nodes

• node-down — a node's node-exporter is unscrapable; the node may be gone, or its kubelet may have stopped posting status while the host stays up (the 2026-09-07 shape).
• node-io — disk I/O pressure/starvation. The etcd WAL shares a partition with Longhorn replicas; this is the alert that would have called the 2026-09-07 outage and did not exist at the time.
• node-memory — memory pressure. Includes why SwapTotal == 0 is not evidence swap is off.
• node-disk — a node filesystem filling up. /data is the one that matters.
• node-cpu — CPU PSI and run-queue depth, and how to tell CPU contention from I/O contention.

Storage

• longhorn-degraded — a volume has lost redundancy. Normal after any reboot, and the rebuild is the expensive part. Freeze/thaw commands here.
• longhorn-faulted — no usable replica. Data at risk; stop before anything destructive.
• longhorn-capacity — a node's Longhorn disk running out of room for replicas (allocation, not PVC fill; for PVC fill see pvc-capacity / monitoring-capacity).
• pvc-capacity — PVC filling up / almost full / predicted to fill. These rules work.
• monitoring-capacity — Prometheus or Loki running out of disk; don't let the monitor blind itself.

GitOps and updates

• flux-reconcile — a Flux resource stuck at Ready=False. Which tier decides the blast radius.
• flux-suspended — reconciliation turned off and forgotten. Drift becomes silent.
• update-rollback — a HelmRelease failed its health gate and is rolling back.
• image-automation — the git write-back engine is failing (feature is disabled here).
• image-policy-stalled — an ImagePolicy/ImageRepository cannot resolve a tag.

Certificates

• cert-issuance — ACME issuance failing. The DNS-01 self-check is the usual cause.
• cert-expiry — a certificate is expiring. Almost always a late symptom of cert-issuance.

Networking

• vpn-down — the egress VPN tunnel. Explains what the kill-switch actually does, and the silent egress-gateway staleness this alert cannot see — which now has its own alerts and page below.
• vpn-egress-stale — the tunnel is UP but egress-class=vpn pods have no egress: Cilium's cached ifindex outlived a reconnect. Also covers the watchdog for that rule going inert.
• denied-flows — policy-denied flow spike. The Cilium hairpin/identity regression class.
• vpn-remote-access — the INBOUND remote-access VPN (WireGuard, udp/51820): standing it up, enrolling and revoking devices, and the off-LAN checks that are the only honest proof it works. The opposite direction to the two pages above — do not confuse them.

Backups and DR

• backup-failure — a Velero backup failed. The R2 object-tagging incompatibility is documented here.
• backup-partial — a Velero backup completed with partial failures, and which skips are deliberate.
• backup-missed — no successful Velero backup in 24h.
• cnpg-archiving — WAL archiving stuck / no recoverability point. An availability incident, not just a backup one.
• cnpg-backup-missed — no recent CNPG base backup. Velero does not cover these databases.
• cnpg-wal-growth — pg_wal not being recycled. Never delete WAL by hand.
• cnpg-exporter — the CNPG SQL exporter erroring: a blind spot that looks exactly like health.
• cnpg-primary-spread — two or more CNPG primaries on one node. Blast radius, not an outage, and the rebalance costs ~9s of writes per database — one cluster at a time.

Identity

• keycloak-down — the IdP is down. Blast radius, and the break-glass path that still works.
• keycloak-db — Keycloak's CloudNativePG database is unavailable.
• keycloak-client-secrets — the client/URL reconciler failing or silent. The 2026-09-06 lockout that every green signal missed.
• oauth2-proxy-auth-failures — a gated app "is up but does not work". Linked from OAuth2ProxyGateUnobserved. The gate answers an unauthenticated XHR with a 302 to Keycloak rather than a 401, the browser follows it silently against a live SSO session, and the call resolves 200 carrying HTML where the app's JavaScript wanted JSON — so the user is never re-prompted and every uptime check stays green. Tells the gate's refusal from the group check from the app's own login, from the sidecar log and from Loki, with the 2026-09-30 jackett trace as the worked example. Read the caveat: the per-app gate sidecars export no metrics at all, so these logs (72h of them) are the only record this class of failure leaves.

Operator procedures (no alert links here)

• plex-claim — claiming the Plex Media Server. Claimed 2026-09-29; the page now only applies after a /config wipe. The claim token is short-lived — read the countdown on plex.tv/claim — so it is a timed sequence, scripted as infra/plex/claim-ship.sh: token → sops → PR → gate → merge → forced flux reconcile → pod roll (~2m45s measured). Includes why a changed Secret does not restart anything, and why an expired token is harmless.
• plex-quicksync — hardware transcoding on the host iGPU. Confirmed working 2026-10-02 (VAAPI decode and encode, zero-copy). GitOps requests gpu.intel.com/i915 on the pod (hardware.gpu: intel-quicksync, satisfied by the intel-gpu-plugin DaemonSet — a /dev/dri hostPath was measured not to work: device-cgroup EPERM); a human still has to tick Settings → Transcoder → "Use hardware acceleration when available", and no environment variable can do it (the image supports no PLEX_PREFERENCE*, verified). The proof is Plex Media Server.log's final decoder: / final encoder: line, not the Plex dashboard — the dashboard showed nothing while hardware transcoding was working, so its silence proves nothing; the page carries the redacting grep (X-Plex-Token lives in that log), the WRONG log file to avoid, and two more documented false leads (an empty /proc/*/fd scan, and a missing HardwareAcceleratedCodecs key). Plex renders no ServiceMonitor, so no alert could ever fire. Also: Pending stalls the whole app tier and its one-field escape hatch, and why a :latest pull can silently break it.
• oauth2-proxy-session — a gated UI loads but its saves quietly do nothing, and you are never re-prompted to log in. Tells the three stacked auth layers apart from the sidecar's own access log (the four lines that matter, including the upstream - that means the proxy answered a request the app never saw). Covers the cookie collision between per-app sidecars sharing the default _oauth2_proxy name on .home.keithjackson.dev — which deletes sessions belonging to apps in other namespaces and is only legible with two logs side by side — the 5-minute access-token session with no --cookie-refresh, and why an XHR answered with a 302 instead of a 401 loses a POST body silently.
• crafty-admin-credential — you signed into Keycloak and Crafty still asks for a password. Expected: Crafty 4 has no OIDC, so its own local login sits behind the oauth2-proxy gate as a second factor. The page tells the two layers apart from the sidecar log, and explains that the bootstrap password exists only on the PVC (default-creds.txt, generated once on first boot, never in git). Includes the argon2 reset for a lost password, why the 3-strikes lockout cannot fire behind the sidecar, and why deleting crafty.sqlite to force a new password destroys every server definition.

Virtualisation

• kubevirt-control-plane — VM lifecycle operations will fail; running VMs keep running.
• kubevirt-launch-failed — a VMI reached the Failed phase. The three k3s-specific launch traps.
• kubevirt-crashloop — a virt-launcher pod restarting repeatedly.
• cdi-import-stuck — a DataVolume import not progressing. Check node I/O before restarting it.

Editing a runbook

Each page is runbooks/<name>.md in the home-network-k3s-impl repo. Edit it, push to main, and Flux re-renders the ConfigMap and rolls the two-replica server; kustomize's content hash on the generated ConfigMap name is what triggers the roll.

Adding a new alert with a runbook_url requires the matching runbooks/<name>.md and its <name>=<name>.md entry in runbooks/kustomization.yaml. Both are enforced by conformance/tests/l0-static/alert-runbooks-resolvable, which runs unconditionally in the merge gate — "the file exists in git" and "the URL resolves in the cluster" are different claims, and it checks both.

Every page must carry the four operator sections (What this means here, First commands, Likely causes, most likely first, When to escalate or stop), mention dev.sh, and pass --context k3s-prod. The check enforces that too, because a stub that makes the link resolve while helping nobody is worse than the 404 it replaced — it looks fixed.

Why this route has no login

Deliberately public. The UIs that declare forward-auth (pihole, jackett, qbittorrent, radarr, sonarr) really are closed: enforcement is per-pod, each one's HTTPRoute targeting its oauth2-proxy sidecar on :4180 rather than the app port. If these pages did too, keycloak-down would be unreadable exactly when Keycloak is down, keycloak-db unreadable when its database is down, and keycloak-client-secrets unreadable during the precise failure that broke all 12 OIDC clients on 2026-09-06 behind an entirely green cluster. An operator who cannot log in is this route's primary reader. The content is operational procedure — metric names, thresholds, kubectl invocations, topology — and no secrets. See runbooks/serving/httproute.yaml for the full reasoning, including how forward-auth is actually enforced here and the instruction to leave this route out of any Gateway-level ext-authz attachment if one is ever introduced.

The running narrative of real incidents on this cluster is DECISIONS-LOG.md.