home-network-k3s-impl/runbooks/flux-rollout.md

flux-rollout — a git change is rolling out, or has stopped halfway

Alerts: FluxRolloutInProgress (info, no for:) and FluxRolloutStuck (warning, 30m) File: platform/monitoring/alerts.yaml, group platform-health.flux Metrics: kube_flux_kustomization_info / kube_flux_kustomization_suspended / kube_flux_gitrepository_info — kube-state-metrics custom-resource-state, configured in platform/monitoring/kube-prometheus-stack.yaml

What this means here

These two are the deploy notification, not a fault report. On this cluster merging a PR into main IS the deploy, and until this pair existed nothing told you a deploy had begun: the Flux event stream emits ReconciliationSucceeded on every 10-minute tick whether or not anything changed (101 such events in one hour, measured 2026-09-29), so "applying a commit" and "idle" looked identical.

The pair reads the one signal that moves only when git moves — the applied revision:

• GitRepository/flux-system .status.artifact.revision — what main is (polled every 1m)
• Kustomization/* .status.lastAppliedRevision — what each tier has actually applied

FluxRolloutInProgress fires the moment any non-suspended Kustomization's applied revision is not the revision fetched from main, and clears when every one of them has caught up. So:

Mail	Subject	Means
firing	[FLUX] applying — FluxRolloutInProgress	a commit started rolling out
resolved	[FLUX] applied — FluxRolloutInProgress	every Kustomization reached it; rollout complete

Both are expected, routine, and not a problem. The severity: info label routes them to the dedicated email-deploy Alertmanager receiver so they never arrive subject-lined [WARNING].

FluxRolloutStuck is the one that matters. Same expression, for: 30m, severity: warning. It means a deploy started and did not converge — the "applying" mail arrived and the "applied" mail never did. The 14-tier chain is dependency-ordered and revision-gated, so a single stalled tier holds every tier below it on "dependency not ready".

First commands

kubectl does not work on the workstation — the toolchain lives only in the pinned container, and the context must always be named explicitly.

Which tiers are behind, and on what:

./infra/devcontainer/dev.sh run bash -c '
  kubectl --context k3s-prod -n flux-system get gitrepository flux-system \
    -o jsonpath="WANTED: {.status.artifact.revision}{\"\n\"}"
  kubectl --context k3s-prod get kustomizations -A \
    -o custom-columns="NAME:.metadata.name,APPLIED:.status.lastAppliedRevision,READY:.status.conditions[?(@.type==\"Ready\")].status"'

Any row whose APPLIED differs from WANTED is a tier this alert is naming. Then ask why that tier is not moving — its Ready condition carries the reason:

./infra/devcontainer/dev.sh run bash -c '
  kubectl --context k3s-prod -n flux-system get kustomization <name> \
    -o jsonpath="{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{\"\n\"}{end}"'

Confirm the alert is reading live data rather than a stale exporter:

./infra/devcontainer/dev.sh run bash -c '
  POD=$(kubectl --context k3s-prod -n monitoring get pod -l app.kubernetes.io/name=prometheus -o name | head -1)
  kubectl --context k3s-prod -n monitoring exec "$POD" -c prometheus -- \
    /bin/promtool query instant http://localhost:9090 "count by (__name__) ({__name__=~\"kube_flux.*\"})"'

Three names must come back (kube_flux_kustomization_info, kube_flux_kustomization_suspended, kube_flux_gitrepository_info). If that query is empty, the alerts are inert — go to the last cause below.

Likely causes, most likely first

1. Nothing is wrong — the rollout is simply in flight. Most Kustomizations reconcile on a 10m interval and the chain is sequential, so several minutes between the "applying" and "applied" mails is normal. cilium is on a 1h interval and can legitimately be the last to converge.
2. A gating tier is unhealthy, so everything below it is parked. Look for DependencyNotReady in the stuck tier's message and walk up the chain to the first tier that is not Ready; that one is the real fault. Its own alert (FluxReconciliationFailure) and runbook flux-reconcile own it.
3. A Kustomization is suspended. A suspended tier is excluded from this alert by design (so it cannot hold the "applied" mail open forever), but anything that dependsOn it still stalls and will show here. FluxResourceSuspended / runbook flux-suspended covers it.
4. A health check is timing out. A wait: true tier only records lastAppliedRevision once its objects pass health checks, so a workload that never goes Ready keeps the tier on the old revision even though the manifests applied fine.
5. kube-state-metrics is not publishing the kube_flux_* family. Then the rules are inert and you would get no mail at all rather than a stuck one — so this is the cause when the complaint is silence. The metrics come from customResourceState plus a read-only RBAC rule for kustomize.toolkit.fluxcd.io / source.toolkit.fluxcd.io; without that RBAC, KSM emits the set empty and logs a forbidden error. Check its logs:

   ./infra/devcontainer/dev.sh run bash -c '
     kubectl --context k3s-prod -n monitoring logs deploy/kube-prometheus-stack-kube-state-metrics \
       --tail=50 | grep -i "forbidden\|custom.resource" || echo "no RBAC complaints"'
   

   Note the chart puts no checksum annotation over that ConfigMap, so a change to the config alone does not restart the pod — that is why podAnnotations carries a homelab.keithjackson.dev/crs-config-rollout counter that must be bumped in the same commit.

When to escalate or stop

• FluxRolloutInProgress alone, cleared within ~15 minutes: do nothing. That is the feature working. Do not suspend, reconcile or apply anything in response to it.
• FluxRolloutStuck firing: fix the upstream fault, not this symptom. This alert names the tier that is behind; it never names the cause. Walk up to the first non-Ready tier and use flux-reconcile.
• Never kubectl apply, edit or patch the change through by hand. Flux reconciles main; a hand-applied object is drift that the next reconcile reverts or, worse, adopts — and it makes the applied-revision signal these alerts read a lie. flux reconcile (an operator action, never an agent's) is the only nudge that is safe, and it only hurries the same pull.
• Escalate to a revert if a stuck rollout is degrading service. Reverting the offending commit on main through the normal PR flow is the rollback path; see runbook update-rollback.
• If both alerts are silent during a deploy you know happened, the notification itself is broken, not the deploy. That is cause 5 above, and it is worth fixing immediately — silence from these rules is indistinguishable from "no deploys happened", which is the exact defect they were written to close.