home-network-k3s-impl/runbooks/flux-rollout.md

# flux-rollout — a git change is rolling out, or has stopped halfway

**Alerts:** `FluxRolloutInProgress` (info, no `for:`) and `FluxRolloutStuck` (warning, 30m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.flux`
**Metrics:** `kube_flux_kustomization_info` / `kube_flux_kustomization_suspended` /
`kube_flux_gitrepository_info` — kube-state-metrics custom-resource-state, configured in
`platform/monitoring/kube-prometheus-stack.yaml`

## What this means here

These two are the **deploy notification**, not a fault report. On this cluster merging a PR into
`main` IS the deploy, and until this pair existed nothing told you a deploy had begun: the Flux event
stream emits `ReconciliationSucceeded` on every 10-minute tick whether or not anything changed (101
such events in one hour, measured 2026-09-29), so "applying a commit" and "idle" looked identical.

The pair reads the one signal that moves only when git moves — the **applied revision**:

- `GitRepository/flux-system` `.status.artifact.revision` — what `main` is (polled every 1m)
- `Kustomization/*` `.status.lastAppliedRevision` — what each tier has actually applied

**`FluxRolloutInProgress` fires** the moment any non-suspended Kustomization's applied revision is
not the revision fetched from `main`, and **clears when every one of them has caught up**. So:

| Mail | Subject | Means |
|---|---|---|
| firing | `[FLUX] applying — FluxRolloutInProgress` | a commit started rolling out |
| resolved | `[FLUX] applied — FluxRolloutInProgress` | every Kustomization reached it; rollout complete |

Both are expected, routine, and not a problem. The `severity: info` label routes them to the
dedicated `email-deploy` Alertmanager receiver so they never arrive subject-lined `[WARNING]`.

**`FluxRolloutStuck` is the one that matters.** Same expression, `for: 30m`, `severity: warning`. It
means a deploy started and did not converge — the "applying" mail arrived and the "applied" mail
never did. The 14-tier chain is dependency-ordered and revision-gated, so a single stalled tier holds
every tier below it on "dependency not ready".

## First commands

`kubectl` does not work on the workstation — the toolchain lives only in the pinned container, and
the context must always be named explicitly.

Which tiers are behind, and on what:

```bash
./infra/devcontainer/dev.sh run bash -c '
  kubectl --context k3s-prod -n flux-system get gitrepository flux-system \
    -o jsonpath="WANTED: {.status.artifact.revision}{\"\n\"}"
  kubectl --context k3s-prod get kustomizations -A \
    -o custom-columns="NAME:.metadata.name,APPLIED:.status.lastAppliedRevision,READY:.status.conditions[?(@.type==\"Ready\")].status"'
```

Any row whose `APPLIED` differs from `WANTED` is a tier this alert is naming. Then ask **why** that
tier is not moving — its Ready condition carries the reason:

```bash
./infra/devcontainer/dev.sh run bash -c '
  kubectl --context k3s-prod -n flux-system get kustomization <name> \
    -o jsonpath="{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{\"\n\"}{end}"'
```

Confirm the alert is reading live data rather than a stale exporter:

```bash
./infra/devcontainer/dev.sh run bash -c '
  POD=$(kubectl --context k3s-prod -n monitoring get pod -l app.kubernetes.io/name=prometheus -o name | head -1)
  kubectl --context k3s-prod -n monitoring exec "$POD" -c prometheus -- \
    /bin/promtool query instant http://localhost:9090 "count by (__name__) ({__name__=~\"kube_flux.*\"})"'
```

Three names must come back (`kube_flux_kustomization_info`, `kube_flux_kustomization_suspended`,
`kube_flux_gitrepository_info`). If that query is **empty**, the alerts are inert — go to the last
cause below.

## Likely causes, most likely first

1. **Nothing is wrong — the rollout is simply in flight.** Most Kustomizations reconcile on a 10m
   interval and the chain is sequential, so several minutes between the "applying" and "applied"
   mails is normal. `cilium` is on a 1h interval and can legitimately be the last to converge.
2. **A gating tier is unhealthy, so everything below it is parked.** Look for `DependencyNotReady`
   in the stuck tier's message and walk *up* the chain to the first tier that is not Ready; that one
   is the real fault. Its own alert (`FluxReconciliationFailure`) and runbook `flux-reconcile` own it.
3. **A Kustomization is suspended.** A suspended tier is excluded from this alert by design (so it
   cannot hold the "applied" mail open forever), but anything that `dependsOn` it still stalls and
   *will* show here. `FluxResourceSuspended` / runbook `flux-suspended` covers it.
4. **A health check is timing out.** A `wait: true` tier only records `lastAppliedRevision` once its
   objects pass health checks, so a workload that never goes Ready keeps the tier on the old
   revision even though the manifests applied fine.
5. **kube-state-metrics is not publishing the `kube_flux_*` family.** Then the rules are inert and
   you would get *no* mail at all rather than a stuck one — so this is the cause when the complaint
   is silence. The metrics come from `customResourceState` plus a read-only RBAC rule for
   `kustomize.toolkit.fluxcd.io` / `source.toolkit.fluxcd.io`; without that RBAC, KSM emits the set
   **empty** and logs a forbidden error. Check its logs:
   ```bash
   ./infra/devcontainer/dev.sh run bash -c '
     kubectl --context k3s-prod -n monitoring logs deploy/kube-prometheus-stack-kube-state-metrics \
       --tail=50 | grep -i "forbidden\|custom.resource" || echo "no RBAC complaints"'
   ```
   Note the chart puts **no checksum annotation** over that ConfigMap, so a change to the config
   alone does not restart the pod — that is why `podAnnotations` carries a
   `homelab.keithjackson.dev/crs-config-rollout` counter that must be bumped in the same commit.

## When to escalate or stop

- **`FluxRolloutInProgress` alone, cleared within ~15 minutes: do nothing.** That is the feature
  working. Do not suspend, reconcile or apply anything in response to it.
- **`FluxRolloutStuck` firing: fix the upstream fault, not this symptom.** This alert names the
  tier that is behind; it never names the cause. Walk up to the first non-Ready tier and use
  `flux-reconcile`.
- **Never `kubectl apply`, `edit` or `patch` the change through by hand.** Flux reconciles `main`;
  a hand-applied object is drift that the next reconcile reverts or, worse, adopts — and it makes
  the applied-revision signal these alerts read a lie. `flux reconcile` (an operator action, never
  an agent's) is the only nudge that is safe, and it only *hurries* the same pull.
- **Escalate to a revert if a stuck rollout is degrading service.** Reverting the offending commit
  on `main` through the normal PR flow is the rollback path; see runbook `update-rollback`.
- **If both alerts are silent during a deploy you know happened,** the notification itself is broken,
  not the deploy. That is cause 5 above, and it is worth fixing immediately — silence from these
  rules is indistinguishable from "no deploys happened", which is the exact defect they were
  written to close.