# flux-suspended — A Flux resource has been suspended for an hour **Alert:** `FluxResourceSuspended` (warning, 1h) **File:** `platform/monitoring/alerts.yaml`, group `platform-health.flux` **Expression:** `max by (namespace, name, kind) (gotk_suspend_status) == 1` ## What this means here Someone turned reconciliation off for a resource and it is still off an hour later. While suspended, Flux neither applies new commits nor corrects drift — **git stops being the truth** for that subtree, silently, and everything downstream of a suspended gating tier is frozen with it. Suspension is a legitimate emergency valve here and is used deliberately: `cilium` was split into its own Kustomization partly so that `flux suspend kustomization cilium` becomes an option that does not freeze the other eleven platform components. The problem is never the suspend; it is the **forgotten** suspend. Two real cases: - After the 2026-09-04 external-dns incident, prod's whole `platform` tier and everything downstream were suspended **and left suspended** (`infra/PROD-PROVISIONING.md` §10). That suspension is also the only reason a second bug never fired: `platform-config` would otherwise have requested a `*.home.keithjackson.dev` wildcard from letsencrypt-prod for a certificate dev already held, and the two clusters would have raced each other's `_acme-challenge` TXT records on one zone — neither would reliably solve DNS-01. - **Velero was found suspended live, with the suspension never recorded in git**, and resumed with `flux resume`. A backup controller that is not reconciling is the definition of a silent DR gap. The 1h `for` is the point: short suspensions during maintenance are expected and should not notify. **This alert is currently INERT.** `count(gotk_reconcile_condition)` and `count(gotk_suspend_status)` both returned **0 series** on prod on 2026-09-13: there is no scrape job for the Flux controllers in Prometheus's 19 jobs, so every `gotk_*`-based rule (`FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed`) evaluates against nothing. **Flux health is therefore unmonitored today** and must be read from the CRs directly, with the commands below. See `/runbooks/targets-down` for the full list of inert rule groups. Adding the Flux controllers' ServiceMonitor is open work. ## Running the commands in this runbook `kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write `backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for agents here: every durable change goes through git. **`main` is a protected branch — a direct push is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, and it says so. ## First commands What is suspended, across every Flux kind: ``` infra/devcontainer/dev.sh run bash -c \ 'for k in kustomizations.kustomize.toolkit.fluxcd.io helmreleases.helm.toolkit.fluxcd.io \ gitrepositories.source.toolkit.fluxcd.io helmrepositories.source.toolkit.fluxcd.io \ imagerepositories.image.toolkit.fluxcd.io imagepolicies.image.toolkit.fluxcd.io \ imageupdateautomations.image.toolkit.fluxcd.io; do \ echo "== $k"; kubectl --context k3s-prod get "$k" -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SUSPENDED:.spec.suspend 2>/dev/null | grep -v ""; done' ``` Is the suspension declared in git, or is it live-only drift? (This is the important question.) ``` grep -rn "suspend" clusters/ platform/ | grep -v "^Binary" ``` If it is not in git, it was applied by hand and nothing will restore it after a controller restart — which means the resource may un-suspend itself at an unpredictable moment, or stay off forever. How far behind is it? ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \ -o custom-columns=NAME:.metadata.name,SUSPENDED:.spec.suspend,APPLIED:.status.lastAppliedRevision; echo; \ kubectl --context k3s-prod -n flux-system get gitrepositories.source.toolkit.fluxcd.io \ -o jsonpath="{.items[0].status.artifact.revision}{\"\n\"}"' ``` ## Likely causes, most likely first ### 1. A maintenance suspension nobody resumed By far the most common, and the Velero case above is the archetype. Remediation (operator): `flux resume kustomization ` — but **only after** reading the next paragraph. ### 2. It is suspended for a reason that still applies Before resuming, find out what would be applied. A tier suspended during an incident may be holding back a change that caused the incident. The 2026-09-04 case is the cautionary one: resuming `platform-config` while a second cluster still held the same wildcard certificate would have started an ACME DNS-01 race on a shared Cloudflare zone. **Diff first**: compare `status.lastAppliedRevision` against the GitRepository's current revision and read the commits in between. ### 3. A crashlooping leaf was suspended instead of being moved The correct fix for "a leaf UI keeps failing and stalls the tier" is a `wait: false` tier of its own — that is what happened to `oauth2-proxy`, `headlamp`, `homepage`, `crafty`, `velero` and `runbooks`. Suspension as a permanent substitute leaves drift correction off for that component forever. ### 4. Image automation `ImageUpdateAutomation`/`ImagePolicy`/`ImageRepository` are **authored and deliberately disabled** here (pending a Gitea write credential and `flux install --components-extra`), so if those show up, this is expected state, not an incident. See `/runbooks/image-automation`. ## When to escalate or stop - **Do not resume a gating tier (`cilium`, `platform`, `platform-config`) without the operator.** Resuming applies every commit since the suspension in one pass, to the tier everything depends on. - **Do resume `velero` promptly** if it is the suspended resource — a suspended backup controller is a DR gap, and it has happened before. - If the suspension is not in git, record it in git (or resume it). A live-only suspension is undocumented state, which is the failure class this repo keeps finding. - If you cannot determine *why* something was suspended, stop and ask. That is a safer outcome than either resuming blind or leaving it off. --- Source of truth: `runbooks/flux-suspended.md`. Index: `/runbooks/` Records: `infra/PROD-PROVISIONING.md` §10 (the 2026-09-04 suspension and the queued ACME race); `DECISIONS-LOG.md` (Velero found suspended with the suspension never in git).