Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 3 Actions Packages Projects Releases Wiki Activity
Files
a1cc0a55560f27e698f00a5bb00baaf12bb719fd
home-network-k3s-impl/runbooks/flux-suspended.md
T
jakman85andClaude Opus 5 b7341b329b fix(runbooks): declare whole-repo scope, drop the bespoke l0 stage, +cnpg-primary-spread 
Three corrections after rebasing onto main.

1. THE GUARD CAUGHT A REAL ONE ON ITS FIRST CONTACT WITH main. main had gained
   `CNPGPrimariesCoLocated` (platform/cnpg/alerts.yaml) pointing at
   /runbooks/cnpg-primary-spread, with no runbook. The check failed the rebase exactly as
   designed — 46 annotations, 35 names, 34 present — naming both the missing file AND the
   missing configMapGenerator entry. runbooks/cnpg-primary-spread.md now exists: what
   co-located primaries actually cost here (blast radius, not an outage, because
   instances: 3 + required anti-affinity), why the drift is uncorrectable-by-declaration
   (CNPG has no primary-placement field), the readiness gate that keeps the rule quiet
   during a real node outage, and the ~9s-per-database switchover rebalance, one cluster
   at a time. The 2026-09-07 outage in which all three primaries sat on one node is the
   reason the alert exists at all.

2. WHOLE-REPO SCOPE, NOT A HAND-WRITTEN STAGE. main's refactor replaced ci/l0.sh's
   per-check "UNCONDITIONAL" stages with a declaration:
   `conformance.home.k3s/scope: "whole-repo"`, unioned into every gate run by Stage 3b's
   `--include-scope`. My +38-line Stage 3f is removed (ci/l0.sh is now byte-identical to
   main) and the annotation is declared instead. Assertion (7) of the check was rewritten
   to match: it no longer asserts its own path appears in ci/l0.sh — it asserts the test
   DECLARES the scope and that ci/l0.sh still passes --include-scope, since neither half
   alone makes the check load-bearing. That matters here because
   ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02", not
   R-OBS-04, and maps platform/kubevirt/* and runbooks/* to nothing at all.

3. A FACTUAL CORRECTION IN ALL 35 RUNBOOKS. Every page claimed `dev.sh` defaults its
   current-context to k3s-dev. main's `9b4744d fix(devcontainer): dev.sh defaults to
   prod` made that false, and a runbook that misstates which cluster a command talks to is
   worse than one that says nothing. The pages now say what is durably true: the default
   has already changed once, so naming `--context k3s-prod` is what makes a command mean
   the same thing next month, and the per-invocation banner is there to be read. The
   check's own failure message was corrected the same way.

Re-verified after all three: check green at 374 assertions (46 annotations, 35/35 runbooks
present, 35/35 projected), re-applied to prod, and re-probed in-cluster through the
Gateway — node-down, keycloak-down, keycloak-client-secrets, cnpg-archiving,
longhorn-faulted, vpn-down, node-io and cnpg-primary-spread all HTTP 200 with
text/plain; charset=utf-8 and 8.9-12.1 KB of real body; /runbooks/ index 200; an unknown
name 404s with a pointer rather than a blank page; and homepage still answers the apex `/`,
confirming Gateway API longest-path precedence.

Full merge gate: PASS (spec-immutability, lint, l0, render-golden, l1 all green;
prom-rules SKIPPED for want of promtool in the image; L2/L3 SKIPPED, no cluster).

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 00:00:24 -05:00

135 lines
8.0 KiB
Markdown
Raw Blame History

# flux-suspended — A Flux resource has been suspended for an hour
**Alert:** `FluxResourceSuspended` (warning, 1h)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.flux`
**Expression:** `max by (namespace, name, kind) (gotk_suspend_status) == 1`
## What this means here
Someone turned reconciliation off for a resource and it is still off an hour later. While suspended,
Flux neither applies new commits nor corrects drift — **git stops being the truth** for that subtree,
silently, and everything downstream of a suspended gating tier is frozen with it.
Suspension is a legitimate emergency valve here and is used deliberately: `cilium` was split into its
own Kustomization partly so that `flux suspend kustomization cilium` becomes an option that does not
freeze the other eleven platform components. The problem is never the suspend; it is the **forgotten**
suspend. Two real cases:
- After the 2026-09-04 external-dns incident, prod's whole `platform` tier and everything downstream
were suspended **and left suspended** (`infra/PROD-PROVISIONING.md` §10). That suspension is also
the only reason a second bug never fired: `platform-config` would otherwise have requested a
`*.home.keithjackson.dev` wildcard from letsencrypt-prod for a certificate dev already held, and the
two clusters would have raced each other's `_acme-challenge` TXT records on one zone — neither would
reliably solve DNS-01.
- **Velero was found suspended live, with the suspension never recorded in git**, and resumed with
`flux resume`. A backup controller that is not reconciling is the definition of a silent DR gap.
The 1h `for` is the point: short suspensions during maintenance are expected and should not notify.
**This alert is currently INERT.** `count(gotk_reconcile_condition)` and `count(gotk_suspend_status)`
both returned **0 series** on prod on 2026-09-13: there is no scrape job for the Flux controllers in
Prometheus's 19 jobs, so every `gotk_*`-based rule (`FluxReconciliationFailure`,
`FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`,
`UpdateHealthGateFailed`) evaluates against nothing. **Flux health is therefore unmonitored today** and
must be read from the CRs directly, with the commands below. See `/runbooks/targets-down` for the full
list of inert rule groups. Adding the Flux controllers' ServiceMonitor is open work.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
What is suspended, across every Flux kind:
```
infra/devcontainer/dev.sh run bash -c \
'for k in kustomizations.kustomize.toolkit.fluxcd.io helmreleases.helm.toolkit.fluxcd.io \
gitrepositories.source.toolkit.fluxcd.io helmrepositories.source.toolkit.fluxcd.io \
imagerepositories.image.toolkit.fluxcd.io imagepolicies.image.toolkit.fluxcd.io \
imageupdateautomations.image.toolkit.fluxcd.io; do \
echo "== $k"; kubectl --context k3s-prod get "$k" -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SUSPENDED:.spec.suspend 2>/dev/null | grep -v "<none>"; done'
```
Is the suspension declared in git, or is it live-only drift? (This is the important question.)
```
grep -rn "suspend" clusters/ platform/ | grep -v "^Binary"
```
If it is not in git, it was applied by hand and nothing will restore it after a controller restart —
which means the resource may un-suspend itself at an unpredictable moment, or stay off forever.
How far behind is it?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \
-o custom-columns=NAME:.metadata.name,SUSPENDED:.spec.suspend,APPLIED:.status.lastAppliedRevision; echo; \
kubectl --context k3s-prod -n flux-system get gitrepositories.source.toolkit.fluxcd.io \
-o jsonpath="{.items[0].status.artifact.revision}{\"\n\"}"'
```
## Likely causes, most likely first
### 1. A maintenance suspension nobody resumed
By far the most common, and the Velero case above is the archetype. Remediation (operator):
`flux resume kustomization <name>` — but **only after** reading the next paragraph.
### 2. It is suspended for a reason that still applies
Before resuming, find out what would be applied. A tier suspended during an incident may be holding
back a change that caused the incident. The 2026-09-04 case is the cautionary one: resuming
`platform-config` while a second cluster still held the same wildcard certificate would have started an
ACME DNS-01 race on a shared Cloudflare zone. **Diff first**: compare
`status.lastAppliedRevision` against the GitRepository's current revision and read the commits in
between.
### 3. A crashlooping leaf was suspended instead of being moved
The correct fix for "a leaf UI keeps failing and stalls the tier" is a `wait: false` tier of its own —
that is what happened to `oauth2-proxy`, `headlamp`, `homepage`, `crafty`, `velero` and `runbooks`.
Suspension as a permanent substitute leaves drift correction off for that component forever.
### 4. Image automation
`ImageUpdateAutomation`/`ImagePolicy`/`ImageRepository` are **authored and deliberately disabled** here
(pending a Gitea write credential and `flux install --components-extra`), so if those show up, this is
expected state, not an incident. See `/runbooks/image-automation`.
## When to escalate or stop
- **Do not resume a gating tier (`cilium`, `platform`, `platform-config`) without the operator.**
Resuming applies every commit since the suspension in one pass, to the tier everything depends on.
- **Do resume `velero` promptly** if it is the suspended resource — a suspended backup controller is a
DR gap, and it has happened before.
- If the suspension is not in git, record it in git (or resume it). A live-only suspension is
undocumented state, which is the failure class this repo keeps finding.
- If you cannot determine *why* something was suspended, stop and ask. That is a safer outcome than
either resuming blind or leaving it off.
---
Source of truth: `runbooks/flux-suspended.md`. Index: `/runbooks/`
Records: `infra/PROD-PROVISIONING.md` §10 (the 2026-09-04 suspension and the queued ACME race);
`DECISIONS-LOG.md` (Velero found suspended with the suspension never in git).
Reference in new issue View Git Blame Copy Permalink