Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 4 Actions Packages Projects Releases Wiki Activity
Files
a92fa117012b8e7d5ed4ae8eecd7b83dbf72a4e3
home-network-k3s-impl/runbooks/backup-missed.md
T
jakman85andClaude Opus 5 b7341b329b fix(runbooks): declare whole-repo scope, drop the bespoke l0 stage, +cnpg-primary-spread 
Three corrections after rebasing onto main.

1. THE GUARD CAUGHT A REAL ONE ON ITS FIRST CONTACT WITH main. main had gained
   `CNPGPrimariesCoLocated` (platform/cnpg/alerts.yaml) pointing at
   /runbooks/cnpg-primary-spread, with no runbook. The check failed the rebase exactly as
   designed — 46 annotations, 35 names, 34 present — naming both the missing file AND the
   missing configMapGenerator entry. runbooks/cnpg-primary-spread.md now exists: what
   co-located primaries actually cost here (blast radius, not an outage, because
   instances: 3 + required anti-affinity), why the drift is uncorrectable-by-declaration
   (CNPG has no primary-placement field), the readiness gate that keeps the rule quiet
   during a real node outage, and the ~9s-per-database switchover rebalance, one cluster
   at a time. The 2026-09-07 outage in which all three primaries sat on one node is the
   reason the alert exists at all.

2. WHOLE-REPO SCOPE, NOT A HAND-WRITTEN STAGE. main's refactor replaced ci/l0.sh's
   per-check "UNCONDITIONAL" stages with a declaration:
   `conformance.home.k3s/scope: "whole-repo"`, unioned into every gate run by Stage 3b's
   `--include-scope`. My +38-line Stage 3f is removed (ci/l0.sh is now byte-identical to
   main) and the annotation is declared instead. Assertion (7) of the check was rewritten
   to match: it no longer asserts its own path appears in ci/l0.sh — it asserts the test
   DECLARES the scope and that ci/l0.sh still passes --include-scope, since neither half
   alone makes the check load-bearing. That matters here because
   ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02", not
   R-OBS-04, and maps platform/kubevirt/* and runbooks/* to nothing at all.

3. A FACTUAL CORRECTION IN ALL 35 RUNBOOKS. Every page claimed `dev.sh` defaults its
   current-context to k3s-dev. main's `9b4744d fix(devcontainer): dev.sh defaults to
   prod` made that false, and a runbook that misstates which cluster a command talks to is
   worse than one that says nothing. The pages now say what is durably true: the default
   has already changed once, so naming `--context k3s-prod` is what makes a command mean
   the same thing next month, and the per-invocation banner is there to be read. The
   check's own failure message was corrected the same way.

Re-verified after all three: check green at 374 assertions (46 annotations, 35/35 runbooks
present, 35/35 projected), re-applied to prod, and re-probed in-cluster through the
Gateway — node-down, keycloak-down, keycloak-client-secrets, cnpg-archiving,
longhorn-faulted, vpn-down, node-io and cnpg-primary-spread all HTTP 200 with
text/plain; charset=utf-8 and 8.9-12.1 KB of real body; /runbooks/ index 200; an unknown
name 404s with a pointer rather than a blank page; and homepage still answers the apex `/`,
confirming Gateway API longest-path precedence.

Full merge gate: PASS (spec-immutability, lint, l0, render-golden, l1 all green;
prom-rules SKIPPED for want of promtool in the image; L2/L3 SKIPPED, no cluster).

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 00:00:24 -05:00

162 lines
9.1 KiB
Markdown
Raw Blame History

# backup-missed — No successful Velero backup in 24 hours
**Alert:** `VeleroNoRecentBackup` (critical, 1h)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.backups`
**Expression:** `(time() - velero_backup_last_successful_timestamp{schedule!=""}) > (24 * 3600)`
## What this means here
A Velero **schedule** has not produced a successful backup in over a day. This is the alert that catches
the quiet failure — not "a backup broke" (that is `/runbooks/backup-failure`) but "backups stopped
happening and nobody noticed". The `{schedule!=""}` filter means it only watches scheduled backups, so
ad-hoc ones do not mask a dead schedule.
Live shape on prod: two schedules, both `0 3 * * *` —
`schedule.velero.io/velero-daily` and `schedule.velero.io/velero-daily-resources-only` — retention
`720h`, target Cloudflare R2 (bucket `k3s-prod-backup`, prefix `velero`, region `auto`).
**Why this alert is worth its `critical`:** the precedent is that these rules *could not fire at all*.
Velero shipped with three backup alerts and the chart default `metrics.serviceMonitor.enabled: false`,
confirmed live on prod — `kubectl -n velero get servicemonitor` returned no resources. Recorded verbatim:
*"A nightly backup could fail — or silently stop running — for weeks with nothing to say so."* The
ServiceMonitor now exists (`velero` and `velero/node-agent` are both live scrape jobs as of 2026-09-13),
which is the only reason this alert is real.
## READ THIS BEFORE YOU CONCLUDE THE BACKUPS ARE GONE
`kubectl get backup -A` prints **"No resources found"** on this cluster **even when backups exist**.
Three CRDs register the kind `Backup` — `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and
`velero.io/v1` — and the bare short name resolves to **Longhorn's**, which holds zero objects. Verified
on prod 2026-09-13: the fully-qualified queries returned **6 Velero backups** and **27 CNPG backups**
while the short name returned nothing. This false negative has already produced two wrong "the backups
are missing" conclusions in one night. **A destructive recovery action taken on it is how a CLI quirk
becomes real data loss.** Always:
```
backups.velero.io backups.postgresql.cnpg.io backups.longhorn.io
schedules.velero.io scheduledbackups.postgresql.cnpg.io clusters.postgresql.cnpg.io
```
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
Are the schedules enabled, and when did each last run?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n velero get schedules.velero.io \
-o custom-columns=NAME:.metadata.name,SCHEDULE:.spec.schedule,PAUSED:.spec.paused,LASTBACKUP:.status.lastBackup,PHASE:.status.phase'
```
What backups actually exist (fully-qualified — the short name lies):
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get backups.velero.io -A \
-o custom-columns=NAME:.metadata.name,PHASE:.status.phase,SCHEDULE:.metadata.labels.velero\\.io/schedule-name,START:.status.startTimestamp --sort-by=.status.startTimestamp'
```
Is the storage location usable at all?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n velero get backupstoragelocations.velero.io \
-o custom-columns=NAME:.metadata.name,PHASE:.status.phase,LASTVALIDATED:.status.lastValidationTime,DEFAULT:.spec.default'
```
Is Velero's own Flux tier reconciling, or was it suspended?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \
-o custom-columns=NAME:.metadata.name,SUSPENDED:.spec.suspend,READY:.status.conditions[?\(@.type==\"Ready\"\)].status | grep -Ei "velero|NAME"'
```
And the controller:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n velero get pods; \
kubectl --context k3s-prod -n velero logs deploy/velero --tail=120 2>&1 | tail -40'
```
## Likely causes, most likely first
### 1. The Velero Flux tier was suspended and nobody resumed it
This has happened: **Velero was found suspended live, with the suspension never recorded in git**, and
was resumed with `flux resume`. A suspended backup controller is a silent DR gap and is precisely the
class this alert exists for. Remediation (operator): `flux resume kustomization velero`, and see
`/runbooks/flux-suspended` for why an undocumented live suspension is itself a problem.
### 2. The schedule is paused
`schedules.velero.io` has `spec.paused`. Note from the live listing that
`velero-daily-resources-only` shows an empty PAUSED column while `velero-daily` shows `false` — read the
field, not the column width. A paused schedule is a git change to unpause.
### 3. Every run is failing rather than not running
Then `backups.velero.io` will show recent `Failed` entries and this alert is a downstream symptom. Go to
`/runbooks/backup-failure` — and in particular check the plugin pin
(`velero/velero-plugin-for-aws:v1.14.3-rc.2`), because the R2 tagging bug produces exactly this: volume
data uploads, the backup goes Failed, and no successful backup ever lands.
### 4. The BackupStorageLocation is not Available
A wrong endpoint, a bad credential, or the placeholder. `clusters/prod` deliberately carried the loud
placeholder `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` rather than inheriting the base (dev)
value, because *"an omitted op means prod silently inherits a dev address — far worse than a failed
apply"*. Check `S3_BACKUP_ENDPOINT`, `S3_BACKUP_REGION` (`auto`), `VELERO_BACKUP_BUCKET`,
`VELERO_BACKUP_PREFIX`, `BACKUP_CREDENTIALS_SUFFIX` (`-prod`).
### 5. Velero cannot schedule a pod
Node pressure, or the node-agent DaemonSet unhealthy. Cross-read `/runbooks/node-io`.
### 6. The metric, not the backup, is missing
If `velero_backup_last_successful_timestamp` has no series at all the rule cannot fire — which is the
historical state. Confirm the `velero` scrape job is up (`/runbooks/targets-down`) before trusting either
its silence or its firing.
## When to escalate or stop
- **Escalate: this is a DR alarm and 24h is already a real window of exposure.**
- **Do not create an ad-hoc backup and consider the alert answered.** The expression filters
`{schedule!=""}`, so a manual backup will not clear it, and more importantly it does not fix the
schedule. (An operator's ad-hoc run for reassurance is
`velero backup create --from-schedule velero-daily`.)
- **Do not conclude "no backups exist" from `kubectl get backup`.** See the trap above.
- **Remember the databases are separate.** `keycloak-db`, `nextcloud-db` and `semaphore-db` are covered
by CNPG's Barman archive, not by Velero. This alert being clear says nothing about them, and vice
versa — check `/runbooks/cnpg-backup-missed` too.
- **Be honest about restore:** `dr-restore` is authored and has **never been executed**, so the R2
restore path is unrehearsed. Free R2 egress makes a rehearsal cheap; it is still outstanding.
---
Source of truth: `runbooks/backup-missed.md`. Index: `/runbooks/`
Records: commit `2d5171e` (Velero had no ServiceMonitor, so these alerts could never fire);
`DECISIONS-LOG.md` (Velero found suspended with the suspension never in git; `dr-restore` never
executed); commit `32d6766` (R2 decision and endpoint knobs).
Reference in new issue View Git Blame Copy Permalink