jakman85andClaude Opus 5
b7341b329b
Three corrections after rebasing onto main. 1. THE GUARD CAUGHT A REAL ONE ON ITS FIRST CONTACT WITH main. main had gained `CNPGPrimariesCoLocated` (platform/cnpg/alerts.yaml) pointing at /runbooks/cnpg-primary-spread, with no runbook. The check failed the rebase exactly as designed — 46 annotations, 35 names, 34 present — naming both the missing file AND the missing configMapGenerator entry. runbooks/cnpg-primary-spread.md now exists: what co-located primaries actually cost here (blast radius, not an outage, because instances: 3 + required anti-affinity), why the drift is uncorrectable-by-declaration (CNPG has no primary-placement field), the readiness gate that keeps the rule quiet during a real node outage, and the ~9s-per-database switchover rebalance, one cluster at a time. The 2026-09-07 outage in which all three primaries sat on one node is the reason the alert exists at all. 2. WHOLE-REPO SCOPE, NOT A HAND-WRITTEN STAGE. main's refactor replaced ci/l0.sh's per-check "UNCONDITIONAL" stages with a declaration: `conformance.home.k3s/scope: "whole-repo"`, unioned into every gate run by Stage 3b's `--include-scope`. My +38-line Stage 3f is removed (ci/l0.sh is now byte-identical to main) and the annotation is declared instead. Assertion (7) of the check was rewritten to match: it no longer asserts its own path appears in ci/l0.sh — it asserts the test DECLARES the scope and that ci/l0.sh still passes --include-scope, since neither half alone makes the check load-bearing. That matters here because ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02", not R-OBS-04, and maps platform/kubevirt/* and runbooks/* to nothing at all. 3. A FACTUAL CORRECTION IN ALL 35 RUNBOOKS. Every page claimed `dev.sh` defaults its current-context to k3s-dev. main's `9b4744d fix(devcontainer): dev.sh defaults to prod` made that false, and a runbook that misstates which cluster a command talks to is worse than one that says nothing. The pages now say what is durably true: the default has already changed once, so naming `--context k3s-prod` is what makes a command mean the same thing next month, and the per-invocation banner is there to be read. The check's own failure message was corrected the same way. Re-verified after all three: check green at 374 assertions (46 annotations, 35/35 runbooks present, 35/35 projected), re-applied to prod, and re-probed in-cluster through the Gateway — node-down, keycloak-down, keycloak-client-secrets, cnpg-archiving, longhorn-faulted, vpn-down, node-io and cnpg-primary-spread all HTTP 200 with text/plain; charset=utf-8 and 8.9-12.1 KB of real body; /runbooks/ index 200; an unknown name 404s with a pointer rather than a blank page; and homepage still answers the apex `/`, confirming Gateway API longest-path precedence. Full merge gate: PASS (spec-immutability, lint, l0, render-golden, l1 all green; prom-rules SKIPPED for want of promtool in the image; L2/L3 SKIPPED, no cluster). Co-Authored-By: Claude Opus 5 <[email protected]>
140 lines
8.6 KiB
Markdown
140 lines
8.6 KiB
Markdown
# Alert runbooks — home-network-k3s prod
|
|
|
|
This is the index page served at `https://home.keithjackson.dev/runbooks/`.
|
|
|
|
Every alert rule in this platform carries an annotation of the form
|
|
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"
|
|
|
|
46 of them, over the 35 names below. Until this directory existed, that path returned **HTTP 404** on the
|
|
live cluster: every alert that reached a human handed them a dead link at the exact moment they needed
|
|
help. These pages are what makes the link resolve.
|
|
|
|
## How to run the commands in these runbooks
|
|
|
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that hang
|
|
is not the cluster being down.
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
|
|
|
Always pass `--context k3s-prod`. `dev.sh` merges BOTH clusters into one kubeconfig and picks a
|
|
current-context per invocation, and **that default has already changed once** — k3s-dev until dev was
|
|
decommissioned on 2026-09-13, k3s-prod now. A command that names its context still means the same thing
|
|
next month; one that relies on the default does not, and the failure is silent. `CLUSTER=prod
|
|
infra/devcontainer/dev.sh run kubectl ...` selects it explicitly, and every invocation prints a banner
|
|
naming the cluster — read it. Do not run two `dev.sh` invocations at once; they share one container.
|
|
|
|
**Use fully-qualified resource names, always.** Three CRDs on this cluster register the kind `Backup` —
|
|
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1`, `velero.io/v1` — and the bare short name resolves to
|
|
Longhorn's, which holds zero objects. `kubectl get backup -A` therefore prints **"No resources found"** on
|
|
a cluster holding 6 Velero and 27 CNPG backups (verified on prod 2026-09-13). Acting on that false
|
|
negative is how a CLI quirk becomes real data loss. Write `backups.velero.io`,
|
|
`backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`,
|
|
`volumes.longhorn.io`, `nodes.longhorn.io`.
|
|
|
|
`git push origin main` **is** the deploy; there is no other apply step. Durable changes go through git.
|
|
|
|
## Alerts you should not trust the silence of (measured 2026-09-13)
|
|
|
|
Prometheus scrapes 19 jobs on prod. There is **no** Longhorn, cert-manager, Flux-controller or
|
|
Hubble/Cilium job, so these rule groups currently evaluate against nothing and **cannot fire**:
|
|
|
|
| Missing scrape job | Inert alerts |
|
|
|---|---|
|
|
| Longhorn | `LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `LonghornVolumeCapacityHigh`, `LonghornNodeStorageHigh` |
|
|
| cert-manager | `CertificateExpiringSoon`, `CertificateExpiryImminent`, `CertificateNotReady` |
|
|
| Flux (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` |
|
|
| Hubble (`hubble_drop_total`) | `HubbleDeniedFlowSpike` |
|
|
| etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all |
|
|
|
|
Each affected runbook says so on its own page and gives the by-hand equivalent. See `targets-down`.
|
|
|
|
## The runbooks
|
|
|
|
### Nodes
|
|
- **`node-down`** — a node's node-exporter is unscrapable; the node may be gone, or its kubelet may have
|
|
stopped posting status while the host stays up (the 2026-09-07 shape).
|
|
- **`node-io`** — disk I/O pressure/starvation. The etcd WAL shares a partition with Longhorn replicas;
|
|
this is the alert that would have called the 2026-09-07 outage and did not exist at the time.
|
|
- **`node-memory`** — memory pressure. Includes why `SwapTotal == 0` is not evidence swap is off.
|
|
- **`node-disk`** — a node filesystem filling up. `/data` is the one that matters.
|
|
- **`node-cpu`** — CPU PSI and run-queue depth, and how to tell CPU contention from I/O contention.
|
|
|
|
### Storage
|
|
- **`longhorn-degraded`** — a volume has lost redundancy. Normal after any reboot, and the rebuild is the
|
|
expensive part. Freeze/thaw commands here.
|
|
- **`longhorn-faulted`** — no usable replica. Data at risk; stop before anything destructive.
|
|
- **`longhorn-capacity`** — a volume or a node's Longhorn disk near capacity.
|
|
- **`pvc-capacity`** — PVC filling up / almost full / predicted to fill. These rules work.
|
|
- **`monitoring-capacity`** — Prometheus or Loki running out of disk; don't let the monitor blind itself.
|
|
|
|
### GitOps and updates
|
|
- **`flux-reconcile`** — a Flux resource stuck at Ready=False. Which tier decides the blast radius.
|
|
- **`flux-suspended`** — reconciliation turned off and forgotten. Drift becomes silent.
|
|
- **`update-rollback`** — a HelmRelease failed its health gate and is rolling back.
|
|
- **`image-automation`** — the git write-back engine is failing (feature is disabled here).
|
|
- **`image-policy-stalled`** — an ImagePolicy/ImageRepository cannot resolve a tag.
|
|
|
|
### Certificates
|
|
- **`cert-issuance`** — ACME issuance failing. The DNS-01 self-check is the usual cause.
|
|
- **`cert-expiry`** — a certificate is expiring. Almost always a late symptom of `cert-issuance`.
|
|
|
|
### Networking
|
|
- **`vpn-down`** — the egress VPN tunnel. Includes the silent egress-gateway staleness that this alert
|
|
cannot see, and what the kill-switch actually does.
|
|
- **`denied-flows`** — policy-denied flow spike. The Cilium hairpin/identity regression class.
|
|
|
|
### Backups and DR
|
|
- **`backup-failure`** — a Velero backup failed. The R2 object-tagging incompatibility is documented here.
|
|
- **`backup-partial`** — a Velero backup completed with partial failures, and which skips are deliberate.
|
|
- **`backup-missed`** — no successful Velero backup in 24h.
|
|
- **`cnpg-archiving`** — WAL archiving stuck / no recoverability point. **An availability incident, not
|
|
just a backup one.**
|
|
- **`cnpg-backup-missed`** — no recent CNPG base backup. Velero does not cover these databases.
|
|
- **`cnpg-wal-growth`** — `pg_wal` not being recycled. Never delete WAL by hand.
|
|
- **`cnpg-exporter`** — the CNPG SQL exporter erroring: a blind spot that looks exactly like health.
|
|
- **`cnpg-primary-spread`** — two or more CNPG primaries on one node. Blast radius, not an outage,
|
|
and the rebalance costs ~9s of writes per database — one cluster at a time.
|
|
|
|
### Identity
|
|
- **`keycloak-down`** — the IdP is down. Blast radius, and the break-glass path that still works.
|
|
- **`keycloak-db`** — Keycloak's CloudNativePG database is unavailable.
|
|
- **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
|
|
every green signal missed.
|
|
|
|
### Virtualisation
|
|
- **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.
|
|
- **`kubevirt-launch-failed`** — a VMI reached the Failed phase. The three k3s-specific launch traps.
|
|
- **`kubevirt-crashloop`** — a virt-launcher pod restarting repeatedly.
|
|
- **`cdi-import-stuck`** — a DataVolume import not progressing. Check node I/O before restarting it.
|
|
|
|
## Editing a runbook
|
|
|
|
Each page is `runbooks/<name>.md` in the `home-network-k3s-impl` repo. Edit it, push to `main`, and Flux
|
|
re-renders the ConfigMap and rolls the two-replica server; kustomize's content hash on the generated
|
|
ConfigMap name is what triggers the roll.
|
|
|
|
Adding a **new** alert with a `runbook_url` requires the matching `runbooks/<name>.md` **and** its
|
|
`<name>=<name>.md` entry in `runbooks/kustomization.yaml`. Both are enforced by
|
|
`conformance/tests/l0-static/alert-runbooks-resolvable`, which runs unconditionally in the merge gate —
|
|
"the file exists in git" and "the URL resolves in the cluster" are different claims, and it checks both.
|
|
|
|
Every page must carry the four operator sections (*What this means here*, *First commands*, *Likely
|
|
causes, most likely first*, *When to escalate or stop*), mention `dev.sh`, and pass `--context k3s-prod`.
|
|
The check enforces that too, because a stub that makes the link resolve while helping nobody is worse than
|
|
the 404 it replaced — it looks fixed.
|
|
|
|
## Why this route has no login
|
|
|
|
Deliberately public. Every other exposed UI here declares forward-auth via oauth2-proxy → Keycloak; if
|
|
these pages did too, `keycloak-down` would be unreadable exactly when Keycloak is down,
|
|
`keycloak-db` unreadable when its database is down, and `keycloak-client-secrets` unreadable during the
|
|
precise failure that broke all 12 OIDC clients on 2026-09-06 behind an entirely green cluster. An
|
|
operator who cannot log in is this route's primary reader. The content is operational procedure — metric
|
|
names, thresholds, kubectl invocations, topology — and no secrets. See
|
|
`runbooks/serving/httproute.yaml` for the full reasoning, including the instruction to leave this route
|
|
**out** of the Gateway ext-authz attachment when forward-auth enforcement lands.
|
|
|
|
The running narrative of real incidents on this cluster is `DECISIONS-LOG.md`.
|