jakman85andClaude Opus 5
8e017fad91
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
137 lines
8.2 KiB
Markdown
137 lines
8.2 KiB
Markdown
# flux-suspended — A Flux resource has been suspended for an hour
|
|
|
|
**Alert:** `FluxResourceSuspended` (warning, 1h)
|
|
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.flux`
|
|
**Expression:** `max by (namespace, name, kind) (gotk_suspend_status) == 1`
|
|
|
|
## What this means here
|
|
|
|
Someone turned reconciliation off for a resource and it is still off an hour later. While suspended,
|
|
Flux neither applies new commits nor corrects drift — **git stops being the truth** for that subtree,
|
|
silently, and everything downstream of a suspended gating tier is frozen with it.
|
|
|
|
Suspension is a legitimate emergency valve here and is used deliberately: `cilium` was split into its
|
|
own Kustomization partly so that `flux suspend kustomization cilium` becomes an option that does not
|
|
freeze the other eleven platform components. The problem is never the suspend; it is the **forgotten**
|
|
suspend. Two real cases:
|
|
|
|
- After the 2026-09-04 external-dns incident, prod's whole `platform` tier and everything downstream
|
|
were suspended **and left suspended** (`infra/PROD-PROVISIONING.md` §10). That suspension is also
|
|
the only reason a second bug never fired: `platform-config` would otherwise have requested a
|
|
`*.home.keithjackson.dev` wildcard from letsencrypt-prod for a certificate dev already held, and the
|
|
two clusters would have raced each other's `_acme-challenge` TXT records on one zone — neither would
|
|
reliably solve DNS-01.
|
|
- **Velero was found suspended live, with the suspension never recorded in git**, and resumed with
|
|
`flux resume`. A backup controller that is not reconciling is the definition of a silent DR gap.
|
|
|
|
The 1h `for` is the point: short suspensions during maintenance are expected and should not notify.
|
|
**This alert is currently INERT.** `count(gotk_reconcile_condition)` and `count(gotk_suspend_status)`
|
|
both returned **0 series** on prod on 2026-09-13: there is no scrape job for the Flux controllers in
|
|
Prometheus's 19 jobs, so every `gotk_*`-based rule (`FluxReconciliationFailure`,
|
|
`FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`,
|
|
`UpdateHealthGateFailed`) evaluates against nothing. **Flux health is therefore unmonitored today** and
|
|
must be read from the CRs directly, with the commands below. See `/runbooks/targets-down` for the full
|
|
list of inert rule groups. Adding the Flux controllers' ServiceMonitor is open work.
|
|
## Running the commands in this runbook
|
|
|
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
|
that hang is not the cluster being down. Every command below runs as:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
|
|
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
|
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
|
|
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
|
|
the same thing next month as it does today; one that relies on the default does not, and the failure is
|
|
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
|
|
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
|
|
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
|
|
interfere.
|
|
|
|
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
|
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
|
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
|
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
|
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
|
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
|
|
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
|
and it says so.
|
|
|
|
## First commands
|
|
|
|
What is suspended, across every Flux kind:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'for k in kustomizations.kustomize.toolkit.fluxcd.io helmreleases.helm.toolkit.fluxcd.io \
|
|
gitrepositories.source.toolkit.fluxcd.io helmrepositories.source.toolkit.fluxcd.io \
|
|
imagerepositories.image.toolkit.fluxcd.io imagepolicies.image.toolkit.fluxcd.io \
|
|
imageupdateautomations.image.toolkit.fluxcd.io; do \
|
|
echo "== $k"; kubectl --context k3s-prod get "$k" -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SUSPENDED:.spec.suspend 2>/dev/null | grep -v "<none>"; done'
|
|
```
|
|
|
|
Is the suspension declared in git, or is it live-only drift? (This is the important question.)
|
|
|
|
```
|
|
grep -rn "suspend" clusters/ platform/ | grep -v "^Binary"
|
|
```
|
|
|
|
If it is not in git, it was applied by hand and nothing will restore it after a controller restart —
|
|
which means the resource may un-suspend itself at an unpredictable moment, or stay off forever.
|
|
|
|
How far behind is it?
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \
|
|
-o custom-columns=NAME:.metadata.name,SUSPENDED:.spec.suspend,APPLIED:.status.lastAppliedRevision; echo; \
|
|
kubectl --context k3s-prod -n flux-system get gitrepositories.source.toolkit.fluxcd.io \
|
|
-o jsonpath="{.items[0].status.artifact.revision}{\"\n\"}"'
|
|
```
|
|
|
|
## Likely causes, most likely first
|
|
|
|
### 1. A maintenance suspension nobody resumed
|
|
By far the most common, and the Velero case above is the archetype. Remediation (operator):
|
|
`flux resume kustomization <name>` — but **only after** reading the next paragraph.
|
|
|
|
### 2. It is suspended for a reason that still applies
|
|
Before resuming, find out what would be applied. A tier suspended during an incident may be holding
|
|
back a change that caused the incident. The 2026-09-04 case is the cautionary one: resuming
|
|
`platform-config` while a second cluster still held the same wildcard certificate would have started an
|
|
ACME DNS-01 race on a shared Cloudflare zone. **Diff first**: compare
|
|
`status.lastAppliedRevision` against the GitRepository's current revision and read the commits in
|
|
between.
|
|
|
|
### 3. A crashlooping leaf was suspended instead of being moved
|
|
The correct fix for "a leaf UI keeps failing and stalls the tier" is a `wait: false` tier of its own —
|
|
that is what happened to `oauth2-proxy`, `headlamp`, `homepage`, `crafty`, `velero` and `runbooks`.
|
|
Suspension as a permanent substitute leaves drift correction off for that component forever.
|
|
|
|
### 4. Image automation
|
|
`ImageUpdateAutomation`/`ImagePolicy`/`ImageRepository` are **authored and deliberately disabled** here
|
|
(pending a Gitea write credential and `flux install --components-extra`), so if those show up, this is
|
|
expected state, not an incident. See `/runbooks/image-automation`.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **Do not resume a gating tier (`cilium`, `platform`, `platform-config`) without the operator.**
|
|
Resuming applies every commit since the suspension in one pass, to the tier everything depends on.
|
|
- **Do resume `velero` promptly** if it is the suspended resource — a suspended backup controller is a
|
|
DR gap, and it has happened before.
|
|
- If the suspension is not in git, record it in git (or resume it). A live-only suspension is
|
|
undocumented state, which is the failure class this repo keeps finding.
|
|
- If you cannot determine *why* something was suspended, stop and ask. That is a safer outcome than
|
|
either resuming blind or leaving it off.
|
|
|
|
---
|
|
Source of truth: `runbooks/flux-suspended.md`. Index: `/runbooks/`
|
|
Records: `infra/PROD-PROVISIONING.md` §10 (the 2026-09-04 suspension and the queued ACME race);
|
|
`DECISIONS-LOG.md` (Velero found suspended with the suspension never in git).
|