jakman85andClaude Opus 5
fc7ec01628
Jackett failed on 2026-09-30 in the most expensive way available: HTTP 200 and a
plausible page. oauth2-proxy answers an unauthenticated XHR with a 302 to Keycloak
rather than a 401; the browser follows it silently against a live SSO session and
the call resolves 200 carrying a 44 KB HTML dashboard where the app's JavaScript
wanted JSON. The user is never re-prompted, reloading never helps, and every uptime
check stays green. `jackett-down` watches replicas_available, which was 1 throughout.
The reason nothing alerted turns out not to be a missing rule. Measured on prod:
count by (__name__) ({__name__=~"oauth2_proxy.*"})
oauth2_proxy_requests_in_flight => 2
Two series, both from the SHARED forward-auth Deployment. The six per-app gate
sidecars (crafty, jackett, pihole, qbittorrent, radarr, sonarr) publish nothing at
all: no --metrics-address, so the metrics listener never starts; no `metrics`
container port; no ServiceMonitor or PodMonitor selects them. The alert the incident
calls for — sustained auth redirects to an authenticated user — is therefore NOT
WRITABLE, and writing it anyway would have shipped a rule that is present,
evaluating, permanently silent and indistinguishable from health.
So alert on the condition that made the incident invisible, from families that ARE
scraped:
OAuth2ProxyGateUnobserved
count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
unless
count by (namespace) (up{container="oauth2-proxy"})
kube_pod_container_info is kube-state-metrics (a declared scrape owner) and sees the
container whether or not it exports anything; `up` is UNIVERSAL and carries a
`container` label for any target, so the rule needs no advance knowledge of what job
name a future per-app scrape picks. Nothing in it touches oauth2_proxy_*, so it
cannot be inert for the reason it is reporting. Run verbatim against live prod it
returns six namespaces and correctly excludes oauth2-proxy. It clears per namespace
as each gate gains a real scrape.
Also:
- promtool companion test pinning all four states: fires unscraped, quiet when
scraped, honours for:1h, and STOPS once a scrape appears under a different job name.
- runbooks/oauth2-proxy-auth-failures.md — tell the gate's refusal from the group
check from the app's own login from an XHR that silently got HTML, with the
2026-09-30 jackett trace as the worked example and the Loki queries to use while
there are no metrics. Registered in the configMapGenerator and README.
- runbooks/README.md gains an inert-alerts row for the unscraped sidecars.
Rejected: an alert on the shared proxy's oauth2_proxy_requests_total ratio. It needs
a conformance/ SCRAPE_OWNERS entry (human approval) and, decisively, would not have
fired on this incident — the shared instance is not in any sidecar-gated app's data
path and its current pods have proxied nothing. It would have looked like coverage
and been none.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
182 lines
12 KiB
Markdown
182 lines
12 KiB
Markdown
# Alert runbooks — home-network-k3s prod
|
||
|
||
This is the index page served at `https://home.keithjackson.dev/runbooks/`.
|
||
|
||
Every alert rule in this platform carries an annotation of the form
|
||
|
||
runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"
|
||
|
||
46 of them, over 35 names. Until this directory existed, that path returned **HTTP 404** on the
|
||
live cluster: every alert that reached a human handed them a dead link at the exact moment they needed
|
||
help. These pages are what makes the link resolve.
|
||
|
||
## How to run the commands in these runbooks
|
||
|
||
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
||
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that hang
|
||
is not the cluster being down.
|
||
|
||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
||
|
||
Always pass `--context k3s-prod`. `dev.sh` merges BOTH clusters into one kubeconfig and picks a
|
||
current-context per invocation, and **that default has already changed once** — k3s-dev until dev was
|
||
decommissioned on 2026-09-13, k3s-prod now. A command that names its context still means the same thing
|
||
next month; one that relies on the default does not, and the failure is silent. `CLUSTER=prod
|
||
infra/devcontainer/dev.sh run kubectl ...` selects it explicitly, and every invocation prints a banner
|
||
naming the cluster — read it. Do not run two `dev.sh` invocations at once; they share one container.
|
||
|
||
**Use fully-qualified resource names, always.** Three CRDs on this cluster register the kind `Backup` —
|
||
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1`, `velero.io/v1` — and the bare short name resolves to
|
||
Longhorn's, which holds zero objects. `kubectl get backup -A` therefore prints **"No resources found"** on
|
||
a cluster holding 6 Velero and 27 CNPG backups (verified on prod 2026-09-13). Acting on that false
|
||
negative is how a CLI quirk becomes real data loss. Write `backups.velero.io`,
|
||
`backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`,
|
||
`volumes.longhorn.io`, `nodes.longhorn.io`.
|
||
|
||
Merging a PR into `main` **is** the deploy; there is no other apply step. `main` is protected, so a
|
||
direct push is rejected — durable changes go through a branch, a PR, and a green gate.
|
||
|
||
## Alerts you should not trust the silence of (measured 2026-09-13)
|
||
|
||
The Longhorn and cert-manager scrapes were ENABLED on 2026-09-13, so the Longhorn row below is no
|
||
longer inert — and re-keying the two robustness rules the same day fixed a SECOND inertness the
|
||
scrape had left in place (the metric is one-hot over a `state` label, not a 0–3 enum). Take the
|
||
Longhorn row as history; the rest of this table still holds, and Flux-controller and Hubble/Cilium
|
||
rule groups still evaluate against nothing and **cannot fire**:
|
||
|
||
| Missing scrape job | Inert alerts |
|
||
|---|---|
|
||
| Longhorn | `LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `LonghornDiskAllocationHigh`, `LonghornDiskUnschedulable` |
|
||
| cert-manager | `CertificateExpiringSoon`, `CertificateExpiryImminent`, `CertificateNotReady` |
|
||
| Flux (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` |
|
||
| Hubble (`hubble_drop_total`) | `HubbleDeniedFlowSpike` |
|
||
| etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all |
|
||
| per-app oauth2-proxy sidecars (`oauth2_proxy_*`) — measured 2026-09-30 | no authentication alerting exists at all for crafty, jackett, pihole, qbittorrent, radarr, sonarr |
|
||
|
||
The oauth2-proxy row is the newest and the least obvious, because the scrape is not merely missing —
|
||
there is **nothing to scrape**. `count by (__name__) ({__name__=~"oauth2_proxy.*"})` on prod returns
|
||
exactly one family, `oauth2_proxy_requests_in_flight`, from the two replicas of the **shared**
|
||
forward-auth instance in the `oauth2-proxy` namespace. The six per-app gate sidecars run without
|
||
`--metrics-address`, so their metrics listener never starts; they declare no `metrics` container port;
|
||
and no ServiceMonitor or PodMonitor selects them. Any rule written against `oauth2_proxy_*` for those
|
||
namespaces would be inert on arrival. `OAuth2ProxyGateUnobserved`
|
||
(`platform/oauth2-proxy/auth-visibility-alerts.yaml`) alerts on that gap itself, using
|
||
`kube_pod_container_info` and `up` — two families that *are* scraped — and clears when a gate gains a
|
||
real scrape. See `oauth2-proxy-auth-failures`.
|
||
|
||
Each affected runbook says so on its own page and gives the by-hand equivalent. See `targets-down`.
|
||
|
||
## The runbooks
|
||
|
||
### Nodes
|
||
- **`node-down`** — a node's node-exporter is unscrapable; the node may be gone, or its kubelet may have
|
||
stopped posting status while the host stays up (the 2026-09-07 shape).
|
||
- **`node-io`** — disk I/O pressure/starvation. The etcd WAL shares a partition with Longhorn replicas;
|
||
this is the alert that would have called the 2026-09-07 outage and did not exist at the time.
|
||
- **`node-memory`** — memory pressure. Includes why `SwapTotal == 0` is not evidence swap is off.
|
||
- **`node-disk`** — a node filesystem filling up. `/data` is the one that matters.
|
||
- **`node-cpu`** — CPU PSI and run-queue depth, and how to tell CPU contention from I/O contention.
|
||
|
||
### Storage
|
||
- **`longhorn-degraded`** — a volume has lost redundancy. Normal after any reboot, and the rebuild is the
|
||
expensive part. Freeze/thaw commands here.
|
||
- **`longhorn-faulted`** — no usable replica. Data at risk; stop before anything destructive.
|
||
- **`longhorn-capacity`** — a node's Longhorn disk running out of room for replicas (allocation, not PVC fill; for PVC fill see `pvc-capacity` / `monitoring-capacity`).
|
||
- **`pvc-capacity`** — PVC filling up / almost full / predicted to fill. These rules work.
|
||
- **`monitoring-capacity`** — Prometheus or Loki running out of disk; don't let the monitor blind itself.
|
||
|
||
### GitOps and updates
|
||
- **`flux-reconcile`** — a Flux resource stuck at Ready=False. Which tier decides the blast radius.
|
||
- **`flux-suspended`** — reconciliation turned off and forgotten. Drift becomes silent.
|
||
- **`update-rollback`** — a HelmRelease failed its health gate and is rolling back.
|
||
- **`image-automation`** — the git write-back engine is failing (feature is disabled here).
|
||
- **`image-policy-stalled`** — an ImagePolicy/ImageRepository cannot resolve a tag.
|
||
|
||
### Certificates
|
||
- **`cert-issuance`** — ACME issuance failing. The DNS-01 self-check is the usual cause.
|
||
- **`cert-expiry`** — a certificate is expiring. Almost always a late symptom of `cert-issuance`.
|
||
|
||
### Networking
|
||
- **`vpn-down`** — the egress VPN tunnel. Explains what the kill-switch actually does, and the silent
|
||
egress-gateway staleness this alert cannot see — which now has its own alerts and page below.
|
||
- **`vpn-egress-stale`** — the tunnel is UP but `egress-class=vpn` pods have no egress: Cilium's cached
|
||
ifindex outlived a reconnect. Also covers the watchdog for that rule going inert.
|
||
- **`denied-flows`** — policy-denied flow spike. The Cilium hairpin/identity regression class.
|
||
|
||
### Backups and DR
|
||
- **`backup-failure`** — a Velero backup failed. The R2 object-tagging incompatibility is documented here.
|
||
- **`backup-partial`** — a Velero backup completed with partial failures, and which skips are deliberate.
|
||
- **`backup-missed`** — no successful Velero backup in 24h.
|
||
- **`cnpg-archiving`** — WAL archiving stuck / no recoverability point. **An availability incident, not
|
||
just a backup one.**
|
||
- **`cnpg-backup-missed`** — no recent CNPG base backup. Velero does not cover these databases.
|
||
- **`cnpg-wal-growth`** — `pg_wal` not being recycled. Never delete WAL by hand.
|
||
- **`cnpg-exporter`** — the CNPG SQL exporter erroring: a blind spot that looks exactly like health.
|
||
- **`cnpg-primary-spread`** — two or more CNPG primaries on one node. Blast radius, not an outage,
|
||
and the rebalance costs ~9s of writes per database — one cluster at a time.
|
||
|
||
### Identity
|
||
- **`keycloak-down`** — the IdP is down. Blast radius, and the break-glass path that still works.
|
||
- **`keycloak-db`** — Keycloak's CloudNativePG database is unavailable.
|
||
- **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
|
||
every green signal missed.
|
||
- **`oauth2-proxy-auth-failures`** — a gated app "is up but does not work". Linked from
|
||
`OAuth2ProxyGateUnobserved`. The gate answers an unauthenticated **XHR** with a 302 to Keycloak
|
||
rather than a 401, the browser follows it silently against a live SSO session, and the call
|
||
resolves **200 carrying HTML** where the app's JavaScript wanted JSON — so the user is never
|
||
re-prompted and every uptime check stays green. Tells the gate's refusal from the group check
|
||
from the app's own login, from the sidecar log and from Loki, with the 2026-09-30 jackett trace
|
||
as the worked example. **Read the caveat**: the per-app gate sidecars export no metrics at all,
|
||
so these logs (72h of them) are the only record this class of failure leaves.
|
||
|
||
### Operator procedures (no alert links here)
|
||
- **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only
|
||
applies after a `/config` wipe. The claim token is short-lived — read the countdown on
|
||
plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
|
||
token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
|
||
Includes why a changed Secret does not restart anything, and why an expired token is harmless.
|
||
- **`crafty-admin-credential`** — you signed into Keycloak and Crafty *still* asks for a password.
|
||
Expected: Crafty 4 has no OIDC, so its own local login sits behind the oauth2-proxy gate as a second
|
||
factor. The page tells the two layers apart from the sidecar log, and explains that the bootstrap
|
||
password exists **only** on the PVC (`default-creds.txt`, generated once on first boot, never in git).
|
||
Includes the argon2 reset for a lost password, why the 3-strikes lockout cannot fire behind the
|
||
sidecar, and why deleting `crafty.sqlite` to force a new password destroys every server definition.
|
||
|
||
### Virtualisation
|
||
- **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.
|
||
- **`kubevirt-launch-failed`** — a VMI reached the Failed phase. The three k3s-specific launch traps.
|
||
- **`kubevirt-crashloop`** — a virt-launcher pod restarting repeatedly.
|
||
- **`cdi-import-stuck`** — a DataVolume import not progressing. Check node I/O before restarting it.
|
||
|
||
## Editing a runbook
|
||
|
||
Each page is `runbooks/<name>.md` in the `home-network-k3s-impl` repo. Edit it, push to `main`, and Flux
|
||
re-renders the ConfigMap and rolls the two-replica server; kustomize's content hash on the generated
|
||
ConfigMap name is what triggers the roll.
|
||
|
||
Adding a **new** alert with a `runbook_url` requires the matching `runbooks/<name>.md` **and** its
|
||
`<name>=<name>.md` entry in `runbooks/kustomization.yaml`. Both are enforced by
|
||
`conformance/tests/l0-static/alert-runbooks-resolvable`, which runs unconditionally in the merge gate —
|
||
"the file exists in git" and "the URL resolves in the cluster" are different claims, and it checks both.
|
||
|
||
Every page must carry the four operator sections (*What this means here*, *First commands*, *Likely
|
||
causes, most likely first*, *When to escalate or stop*), mention `dev.sh`, and pass `--context k3s-prod`.
|
||
The check enforces that too, because a stub that makes the link resolve while helping nobody is worse than
|
||
the 404 it replaced — it looks fixed.
|
||
|
||
## Why this route has no login
|
||
|
||
Deliberately public. The UIs that declare forward-auth (pihole, jackett, qbittorrent, radarr, sonarr)
|
||
really are closed: enforcement is per-pod, each one's HTTPRoute targeting its oauth2-proxy sidecar on
|
||
:4180 rather than the app port. If
|
||
these pages did too, `keycloak-down` would be unreadable exactly when Keycloak is down,
|
||
`keycloak-db` unreadable when its database is down, and `keycloak-client-secrets` unreadable during the
|
||
precise failure that broke all 12 OIDC clients on 2026-09-06 behind an entirely green cluster. An
|
||
operator who cannot log in is this route's primary reader. The content is operational procedure — metric
|
||
names, thresholds, kubectl invocations, topology — and no secrets. See
|
||
`runbooks/serving/httproute.yaml` for the full reasoning, including how forward-auth is actually
|
||
enforced here and the instruction to leave this route **out** of any Gateway-level ext-authz attachment
|
||
if one is ever introduced.
|
||
|
||
The running narrative of real incidents on this cluster is `DECISIONS-LOG.md`.
|