# Alert runbooks — home-network-k3s prod This is the index page served at `https://home.keithjackson.dev/runbooks/`. Every alert rule in this platform carries an annotation of the form runbook_url: "https://${BASE_DOMAIN}/runbooks/" 46 of them, over 35 names. Until this directory existed, that path returned **HTTP 404** on the live cluster: every alert that reached a human handed them a dead link at the exact moment they needed help. These pages are what makes the link resolve. ## How to run the commands in these runbooks `kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that hang is not the cluster being down. infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' Always pass `--context k3s-prod`. `dev.sh` merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and **that default has already changed once** — k3s-dev until dev was decommissioned on 2026-09-13, k3s-prod now. A command that names its context still means the same thing next month; one that relies on the default does not, and the failure is silent. `CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster — read it. Do not run two `dev.sh` invocations at once; they share one container. **Use fully-qualified resource names, always.** Three CRDs on this cluster register the kind `Backup` — `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1`, `velero.io/v1` — and the bare short name resolves to Longhorn's, which holds zero objects. `kubectl get backup -A` therefore prints **"No resources found"** on a cluster holding 6 Velero and 27 CNPG backups (verified on prod 2026-09-13). Acting on that false negative is how a CLI quirk becomes real data loss. Write `backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Merging a PR into `main` **is** the deploy; there is no other apply step. `main` is protected, so a direct push is rejected — durable changes go through a branch, a PR, and a green gate. ## Alerts you should not trust the silence of (measured 2026-09-13) The Longhorn and cert-manager scrapes were ENABLED on 2026-09-13, so the Longhorn row below is no longer inert — and re-keying the two robustness rules the same day fixed a SECOND inertness the scrape had left in place (the metric is one-hot over a `state` label, not a 0–3 enum). Take the Longhorn row as history; the rest of this table still holds, and Flux-controller and Hubble/Cilium rule groups still evaluate against nothing and **cannot fire**: | Missing scrape job | Inert alerts | |---|---| | Longhorn | `LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `LonghornDiskAllocationHigh`, `LonghornDiskUnschedulable` | | cert-manager | `CertificateExpiringSoon`, `CertificateExpiryImminent`, `CertificateNotReady` | | Flux (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` | | Hubble (`hubble_drop_total`) | `HubbleDeniedFlowSpike` | | etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all | | per-app oauth2-proxy sidecars (`oauth2_proxy_*`) — updated 2026-10-01 | no authentication alerting exists at all for crafty, jackett, pihole, qbittorrent, radarr, sonarr | The oauth2-proxy row is the newest, and it has **moved twice in two days** — worth reading in order, because each step looked like the end of it. **2026-09-30 — nothing to scrape.** `count by (__name__) ({__name__=~"oauth2_proxy.*"})` returned exactly one family, `oauth2_proxy_requests_in_flight`, from the two replicas of the **shared** forward-auth instance. All six per-app gate sidecars ran without `--metrics-address`, so their metrics listener never started, and no ServiceMonitor or PodMonitor selected them. **2026-10-01 — the listener exists, and the scrape is still blocked.** PR #28 added `--metrics-address=0.0.0.0:44180` and a named `metrics` port to the five fleet sidecars, and the endpoint genuinely serves (port-forwarded: `http=200`, 10208 bytes, all three `oauth2_proxy_*` families, `oauth2_proxy_requests_total{code="302"}` present). `platform/oauth2-proxy/podmonitor.yaml` now discovers them by pod label. **But the app namespaces' `default-deny` NetworkPolicy drops the scrape**: from a pod in ns `monitoring`, both `:44180` and `:4180` on the jackett pod time out, while the shared proxy's `:44180` answers — the only difference being that ns `oauth2-proxy` has no NetworkPolicy at all. So the targets are discovered and `up == 0`, and the TSDB is still empty for those namespaces. **This is the same bug Longhorn had**, with the same fingerprint; the fix idiom is `platform/longhorn/networkpolicy-metrics.yaml` (ns monitoring → one pod label → one port), and for these apps it has to be rendered alongside each app's own policy rather than shipped from the oauth2-proxy tier, which is a *sibling* of `apps-wave-0` in the Flux graph and so cannot rely on those namespaces existing yet. `OAuth2ProxyGateUnobserved` (`platform/oauth2-proxy/auth-visibility-alerts.yaml`) alerts on the gap itself from `kube_pod_container_info` and `up` — two families that *are* scraped. Note its `up == 1`: a discovered-but-failing target publishes `up` with value **0**, so the earlier bare form would have gone quiet the moment the PodMonitor landed, while nothing was being scraped. That near-miss is the whole reason to distrust a cleared alert here. Crafty remains outside all of this — no metrics listener at all. See `oauth2-proxy-auth-failures`. Each affected runbook says so on its own page and gives the by-hand equivalent. See `targets-down`. ## The runbooks ### Nodes - **`node-down`** — a node's node-exporter is unscrapable; the node may be gone, or its kubelet may have stopped posting status while the host stays up (the 2026-09-07 shape). - **`node-io`** — disk I/O pressure/starvation. The etcd WAL shares a partition with Longhorn replicas; this is the alert that would have called the 2026-09-07 outage and did not exist at the time. - **`node-memory`** — memory pressure. Includes why `SwapTotal == 0` is not evidence swap is off. - **`node-disk`** — a node filesystem filling up. `/data` is the one that matters. - **`node-cpu`** — CPU PSI and run-queue depth, and how to tell CPU contention from I/O contention. ### Storage - **`longhorn-degraded`** — a volume has lost redundancy. Normal after any reboot, and the rebuild is the expensive part. Freeze/thaw commands here. - **`longhorn-faulted`** — no usable replica. Data at risk; stop before anything destructive. - **`longhorn-capacity`** — a node's Longhorn disk running out of room for replicas (allocation, not PVC fill; for PVC fill see `pvc-capacity` / `monitoring-capacity`). - **`pvc-capacity`** — PVC filling up / almost full / predicted to fill. These rules work. - **`monitoring-capacity`** — Prometheus or Loki running out of disk; don't let the monitor blind itself. ### GitOps and updates - **`flux-reconcile`** — a Flux resource stuck at Ready=False. Which tier decides the blast radius. - **`flux-suspended`** — reconciliation turned off and forgotten. Drift becomes silent. - **`update-rollback`** — a HelmRelease failed its health gate and is rolling back. - **`image-automation`** — the git write-back engine is failing (feature is disabled here). - **`image-policy-stalled`** — an ImagePolicy/ImageRepository cannot resolve a tag. ### Certificates - **`cert-issuance`** — ACME issuance failing. The DNS-01 self-check is the usual cause. - **`cert-expiry`** — a certificate is expiring. Almost always a late symptom of `cert-issuance`. ### Networking - **`vpn-down`** — the egress VPN tunnel. Explains what the kill-switch actually does, and the silent egress-gateway staleness this alert cannot see — which now has its own alerts and page below. - **`vpn-egress-stale`** — the tunnel is UP but `egress-class=vpn` pods have no egress: Cilium's cached ifindex outlived a reconnect. Also covers the watchdog for that rule going inert. - **`denied-flows`** — policy-denied flow spike. The Cilium hairpin/identity regression class. - **`vpn-remote-access`** — the INBOUND remote-access VPN (WireGuard, udp/51820): standing it up, enrolling and revoking devices, and the off-LAN checks that are the only honest proof it works. The opposite direction to the two pages above — do not confuse them. ### Backups and DR - **`backup-failure`** — a Velero backup failed. The R2 object-tagging incompatibility is documented here. - **`backup-partial`** — a Velero backup completed with partial failures, and which skips are deliberate. - **`backup-missed`** — no successful Velero backup in 24h. - **`cnpg-archiving`** — WAL archiving stuck / no recoverability point. **An availability incident, not just a backup one.** - **`cnpg-backup-missed`** — no recent CNPG base backup. Velero does not cover these databases. - **`cnpg-wal-growth`** — `pg_wal` not being recycled. Never delete WAL by hand. - **`cnpg-exporter`** — the CNPG SQL exporter erroring: a blind spot that looks exactly like health. - **`cnpg-primary-spread`** — two or more CNPG primaries on one node. Blast radius, not an outage, and the rebalance costs ~9s of writes per database — one cluster at a time. ### Identity - **`keycloak-down`** — the IdP is down. Blast radius, and the break-glass path that still works. - **`keycloak-db`** — Keycloak's CloudNativePG database is unavailable. - **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that every green signal missed. - **`keycloak-passkey-reconciler`** — the passkey policy / enrolment action / browser-flow reconciler failing. Never writes the flow you log in through; a missing `CONFIGURE_TOTP` is the cause that bites. - **`oauth2-proxy-auth-failures`** — a gated app "is up but does not work". Linked from `OAuth2ProxyGateUnobserved`. The gate answers an unauthenticated **XHR** with a 302 to Keycloak rather than a 401, the browser follows it silently against a live SSO session, and the call resolves **200 carrying HTML** where the app's JavaScript wanted JSON — so the user is never re-prompted and every uptime check stays green. Tells the gate's refusal from the group check from the app's own login, from the sidecar log and from Loki, with the 2026-09-30 jackett trace as the worked example. **Read the caveat**: the per-app gate sidecars export no metrics at all, so these logs (72h of them) are the only record this class of failure leaves. ### Operator procedures (no alert links here) - **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only applies after a `/config` wipe. The claim token is short-lived — read the countdown on plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`: token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured). Includes why a changed Secret does not restart anything, and why an expired token is harmless. - **`plex-quicksync`** — hardware transcoding on the host iGPU. **Confirmed working 2026-10-02** (VAAPI decode *and* encode, zero-copy). GitOps requests `gpu.intel.com/i915` on the pod (`hardware.gpu: intel-quicksync`, satisfied by the `intel-gpu-plugin` DaemonSet — a `/dev/dri` hostPath was measured not to work: device-cgroup `EPERM`); **a human still has to tick Settings → Transcoder → "Use hardware acceleration when available"**, and no environment variable can do it (the image supports no `PLEX_PREFERENCE*`, verified). **The proof is `Plex Media Server.log`'s `final decoder: / final encoder:` line, not the Plex dashboard** — the dashboard showed nothing while hardware transcoding was working, so its silence proves nothing; the page carries the redacting grep (`X-Plex-Token` lives in that log), the WRONG log file to avoid, and two more documented false leads (an empty `/proc/*/fd` scan, and a missing `HardwareAcceleratedCodecs` key). Plex renders no ServiceMonitor, so no alert could ever fire. Also: **`Pending` stalls the whole app tier** and its one-field escape hatch, and why a `:latest` pull can silently break it. - **`oauth2-proxy-session`** — a gated UI loads but its saves quietly do nothing, and you are never re-prompted to log in. Tells the three stacked auth layers apart from the sidecar's own access log (the four lines that matter, including the upstream `-` that means the proxy answered a request the app never saw). Covers the cookie collision between per-app sidecars sharing the default `_oauth2_proxy` name on `.home.keithjackson.dev` — which deletes sessions belonging to apps in *other* namespaces and is only legible with two logs side by side — the 5-minute access-token session with no `--cookie-refresh`, and why an XHR answered with a 302 instead of a 401 loses a POST body silently. - **`crafty-admin-credential`** — you signed into Keycloak and Crafty *still* asks for a password. Expected: Crafty 4 has no OIDC, so its own local login sits behind the oauth2-proxy gate as a second factor. The page tells the two layers apart from the sidecar log, and explains that the bootstrap password exists **only** on the PVC (`default-creds.txt`, generated once on first boot, never in git). Includes the argon2 reset for a lost password, why the 3-strikes lockout cannot fire behind the sidecar, and why deleting `crafty.sqlite` to force a new password destroys every server definition. ### Virtualisation - **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running. - **`kubevirt-launch-failed`** — a VMI reached the Failed phase. The three k3s-specific launch traps. - **`kubevirt-crashloop`** — a virt-launcher pod restarting repeatedly. - **`cdi-import-stuck`** — a DataVolume import not progressing. Check node I/O before restarting it. ## Editing a runbook Each page is `runbooks/.md` in the `home-network-k3s-impl` repo. Edit it, push to `main`, and Flux re-renders the ConfigMap and rolls the two-replica server; kustomize's content hash on the generated ConfigMap name is what triggers the roll. Adding a **new** alert with a `runbook_url` requires the matching `runbooks/.md` **and** its `=.md` entry in `runbooks/kustomization.yaml`. Both are enforced by `conformance/tests/l0-static/alert-runbooks-resolvable`, which runs unconditionally in the merge gate — "the file exists in git" and "the URL resolves in the cluster" are different claims, and it checks both. Every page must carry the four operator sections (*What this means here*, *First commands*, *Likely causes, most likely first*, *When to escalate or stop*), mention `dev.sh`, and pass `--context k3s-prod`. The check enforces that too, because a stub that makes the link resolve while helping nobody is worse than the 404 it replaced — it looks fixed. ## Why this route has no login Deliberately public. The UIs that declare forward-auth (pihole, jackett, qbittorrent, radarr, sonarr) really are closed: enforcement is per-pod, each one's HTTPRoute targeting its oauth2-proxy sidecar on :4180 rather than the app port. If these pages did too, `keycloak-down` would be unreadable exactly when Keycloak is down, `keycloak-db` unreadable when its database is down, and `keycloak-client-secrets` unreadable during the precise failure that broke all 12 OIDC clients on 2026-09-06 behind an entirely green cluster. An operator who cannot log in is this route's primary reader. The content is operational procedure — metric names, thresholds, kubectl invocations, topology — and no secrets. See `runbooks/serving/httproute.yaml` for the full reasoning, including how forward-auth is actually enforced here and the instruction to leave this route **out** of any Gateway-level ext-authz attachment if one is ever introduced. The running narrative of real incidents on this cluster is `DECISIONS-LOG.md`.