jakman85andClaude Opus 5
8e017fad91
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
164 lines
9.2 KiB
Markdown
164 lines
9.2 KiB
Markdown
# backup-missed — No successful Velero backup in 24 hours
|
|
|
|
**Alert:** `VeleroNoRecentBackup` (critical, 1h)
|
|
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.backups`
|
|
**Expression:** `(time() - velero_backup_last_successful_timestamp{schedule!=""}) > (24 * 3600)`
|
|
|
|
## What this means here
|
|
|
|
A Velero **schedule** has not produced a successful backup in over a day. This is the alert that catches
|
|
the quiet failure — not "a backup broke" (that is `/runbooks/backup-failure`) but "backups stopped
|
|
happening and nobody noticed". The `{schedule!=""}` filter means it only watches scheduled backups, so
|
|
ad-hoc ones do not mask a dead schedule.
|
|
|
|
Live shape on prod: two schedules, both `0 3 * * *` —
|
|
`schedule.velero.io/velero-daily` and `schedule.velero.io/velero-daily-resources-only` — retention
|
|
`720h`, target Cloudflare R2 (bucket `k3s-prod-backup`, prefix `velero`, region `auto`).
|
|
|
|
**Why this alert is worth its `critical`:** the precedent is that these rules *could not fire at all*.
|
|
Velero shipped with three backup alerts and the chart default `metrics.serviceMonitor.enabled: false`,
|
|
confirmed live on prod — `kubectl -n velero get servicemonitor` returned no resources. Recorded verbatim:
|
|
*"A nightly backup could fail — or silently stop running — for weeks with nothing to say so."* The
|
|
ServiceMonitor now exists (`velero` and `velero/node-agent` are both live scrape jobs as of 2026-09-13),
|
|
which is the only reason this alert is real.
|
|
## READ THIS BEFORE YOU CONCLUDE THE BACKUPS ARE GONE
|
|
|
|
`kubectl get backup -A` prints **"No resources found"** on this cluster **even when backups exist**.
|
|
Three CRDs register the kind `Backup` — `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and
|
|
`velero.io/v1` — and the bare short name resolves to **Longhorn's**, which holds zero objects. Verified
|
|
on prod 2026-09-13: the fully-qualified queries returned **6 Velero backups** and **27 CNPG backups**
|
|
while the short name returned nothing. This false negative has already produced two wrong "the backups
|
|
are missing" conclusions in one night. **A destructive recovery action taken on it is how a CLI quirk
|
|
becomes real data loss.** Always:
|
|
|
|
```
|
|
backups.velero.io backups.postgresql.cnpg.io backups.longhorn.io
|
|
schedules.velero.io scheduledbackups.postgresql.cnpg.io clusters.postgresql.cnpg.io
|
|
```
|
|
## Running the commands in this runbook
|
|
|
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
|
that hang is not the cluster being down. Every command below runs as:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
|
|
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
|
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
|
|
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
|
|
the same thing next month as it does today; one that relies on the default does not, and the failure is
|
|
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
|
|
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
|
|
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
|
|
interfere.
|
|
|
|
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
|
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
|
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
|
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
|
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
|
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
|
|
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
|
and it says so.
|
|
|
|
## First commands
|
|
|
|
Are the schedules enabled, and when did each last run?
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n velero get schedules.velero.io \
|
|
-o custom-columns=NAME:.metadata.name,SCHEDULE:.spec.schedule,PAUSED:.spec.paused,LASTBACKUP:.status.lastBackup,PHASE:.status.phase'
|
|
```
|
|
|
|
What backups actually exist (fully-qualified — the short name lies):
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get backups.velero.io -A \
|
|
-o custom-columns=NAME:.metadata.name,PHASE:.status.phase,SCHEDULE:.metadata.labels.velero\\.io/schedule-name,START:.status.startTimestamp --sort-by=.status.startTimestamp'
|
|
```
|
|
|
|
Is the storage location usable at all?
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n velero get backupstoragelocations.velero.io \
|
|
-o custom-columns=NAME:.metadata.name,PHASE:.status.phase,LASTVALIDATED:.status.lastValidationTime,DEFAULT:.spec.default'
|
|
```
|
|
|
|
Is Velero's own Flux tier reconciling, or was it suspended?
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \
|
|
-o custom-columns=NAME:.metadata.name,SUSPENDED:.spec.suspend,READY:.status.conditions[?\(@.type==\"Ready\"\)].status | grep -Ei "velero|NAME"'
|
|
```
|
|
|
|
And the controller:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n velero get pods; \
|
|
kubectl --context k3s-prod -n velero logs deploy/velero --tail=120 2>&1 | tail -40'
|
|
```
|
|
|
|
## Likely causes, most likely first
|
|
|
|
### 1. The Velero Flux tier was suspended and nobody resumed it
|
|
This has happened: **Velero was found suspended live, with the suspension never recorded in git**, and
|
|
was resumed with `flux resume`. A suspended backup controller is a silent DR gap and is precisely the
|
|
class this alert exists for. Remediation (operator): `flux resume kustomization velero`, and see
|
|
`/runbooks/flux-suspended` for why an undocumented live suspension is itself a problem.
|
|
|
|
### 2. The schedule is paused
|
|
`schedules.velero.io` has `spec.paused`. Note from the live listing that
|
|
`velero-daily-resources-only` shows an empty PAUSED column while `velero-daily` shows `false` — read the
|
|
field, not the column width. A paused schedule is a git change to unpause.
|
|
|
|
### 3. Every run is failing rather than not running
|
|
Then `backups.velero.io` will show recent `Failed` entries and this alert is a downstream symptom. Go to
|
|
`/runbooks/backup-failure` — and in particular check the plugin pin
|
|
(`velero/velero-plugin-for-aws:v1.14.3-rc.2`), because the R2 tagging bug produces exactly this: volume
|
|
data uploads, the backup goes Failed, and no successful backup ever lands.
|
|
|
|
### 4. The BackupStorageLocation is not Available
|
|
A wrong endpoint, a bad credential, or the placeholder. `clusters/prod` deliberately carried the loud
|
|
placeholder `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` rather than inheriting the base (dev)
|
|
value, because *"an omitted op means prod silently inherits a dev address — far worse than a failed
|
|
apply"*. Check `S3_BACKUP_ENDPOINT`, `S3_BACKUP_REGION` (`auto`), `VELERO_BACKUP_BUCKET`,
|
|
`VELERO_BACKUP_PREFIX`, `BACKUP_CREDENTIALS_SUFFIX` (`-prod`).
|
|
|
|
### 5. Velero cannot schedule a pod
|
|
Node pressure, or the node-agent DaemonSet unhealthy. Cross-read `/runbooks/node-io`.
|
|
|
|
### 6. The metric, not the backup, is missing
|
|
If `velero_backup_last_successful_timestamp` has no series at all the rule cannot fire — which is the
|
|
historical state. Confirm the `velero` scrape job is up (`/runbooks/targets-down`) before trusting either
|
|
its silence or its firing.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **Escalate: this is a DR alarm and 24h is already a real window of exposure.**
|
|
- **Do not create an ad-hoc backup and consider the alert answered.** The expression filters
|
|
`{schedule!=""}`, so a manual backup will not clear it, and more importantly it does not fix the
|
|
schedule. (An operator's ad-hoc run for reassurance is
|
|
`velero backup create --from-schedule velero-daily`.)
|
|
- **Do not conclude "no backups exist" from `kubectl get backup`.** See the trap above.
|
|
- **Remember the databases are separate.** `keycloak-db`, `nextcloud-db` and `semaphore-db` are covered
|
|
by CNPG's Barman archive, not by Velero. This alert being clear says nothing about them, and vice
|
|
versa — check `/runbooks/cnpg-backup-missed` too.
|
|
- **Be honest about restore:** `dr-restore` is authored and has **never been executed**, so the R2
|
|
restore path is unrehearsed. Free R2 egress makes a rehearsal cheap; it is still outstanding.
|
|
|
|
---
|
|
Source of truth: `runbooks/backup-missed.md`. Index: `/runbooks/`
|
|
Records: commit `2d5171e` (Velero had no ServiceMonitor, so these alerts could never fire);
|
|
`DECISIONS-LOG.md` (Velero found suspended with the suspension never in git; `dr-restore` never
|
|
executed); commit `32d6766` (R2 decision and endpoint knobs).
|