home-network-k3s-impl/runbooks/flux-reconcile.md

flux-reconcile — A Flux resource has failed to reconcile for 15 minutes

Alert: FluxReconciliationFailure (warning, 15m) File: platform/monitoring/alerts.yaml, group platform-health.flux Expression: max by (namespace, name, kind) (gotk_reconcile_condition{type="Ready", status="False"}) == 1

What this means here

Something in the GitOps chain is stuck at Ready=False. On this cluster that is more consequential than it sounds, because git push origin main IS the deploy — there is no other apply step. A stuck Kustomization means git and the cluster have diverged and nothing will close the gap.

The chain has 14 tiers: flux-system → cilium → platform → platform-config → platform-config-cdi → apps-wave-0/1/2, plus the non-gating leaves apps-vms, oauth2-proxy, headlamp, homepage, crafty, runbooks, backup. Which tier is failing decides the blast radius:

• flux-system, cilium, platform, platform-config are gating (wait: true). One Failed HelmRelease in the platform tier stalls everything downstream on "dependency not ready". That is not theoretical: a crashlooping Headlamp HelmRelease once stalled platform-config, oauth2-proxy and every apps wave (the "Headlamp gating stall"), and oauth2-proxy twice deadlocked by blocking its own fix — it sat in the gating tier while needing Keycloak + trusted TLS + split-horizon DNS, so platform-config, which applies the fix, could never reconcile. Both were resolved by moving the leaf into its own wait: false tier.
• The leaf tiers are wait: false precisely so their unhealth can never gate anything.

Prod's source is http://gitea.gitea.svc.cluster.local:3000 — the in-cluster Service, on purpose. It used to be a public hostname, and the apex flip taught the expensive version of that lesson: the HTTPRoute serving the source's own hostname lives in a downstream tier that cannot reconcile while the source is down, so srcReady=False / "unable to list remote" became a deadlock. If you are tempted to point the GitRepository at a public name, don't. This alert is currently INERT. count(gotk_reconcile_condition) and count(gotk_suspend_status) both returned 0 series on prod on 2026-09-13: there is no scrape job for the Flux controllers in Prometheus's 19 jobs, so every gotk_*-based rule (FluxReconciliationFailure, FluxResourceSuspended, ImageUpdateAutomationFailing, ImagePolicyStalled, UpdateHealthGateFailed) evaluates against nothing. Flux health is therefore unmonitored today and must be read from the CRs directly, with the commands below. See /runbooks/targets-down for the full list of inert rule groups. Adding the Flux controllers' ServiceMonitor is open work.

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two dev.sh invocations at once; they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git (git push origin main is the deploy). Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

The whole chain at a glance — this is the command to run first, every time:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \
     -o custom-columns=NAME:.metadata.name,PATH:.spec.path,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,REV:.status.lastAppliedRevision'

Is the SOURCE healthy? If not, nothing downstream can be:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n flux-system get gitrepositories.source.toolkit.fluxcd.io \
     -o custom-columns=NAME:.metadata.name,URL:.spec.url,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message'

HelmReleases (the usual culprit inside platform):

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get helmreleases.helm.toolkit.fluxcd.io -A \
     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message'

The controller's own view, with the real error text:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n flux-system logs deploy/kustomize-controller --tail=80; echo ====; \
   kubectl --context k3s-prod -n flux-system logs deploy/helm-controller --tail=80'

Likely causes, most likely first

1. "no matches for kind X" — a CR applied before its CRD

The recurring structural failure here. Flux server-side-apply fails the whole apply, so a tier that ships both a CRD-installing HelmRelease and CRs of that CRD can never converge — the CRD never gets created because the apply that would create it also contains the CR. Symptoms: no matches for kind "ClusterIssuer" / "PrometheusRule". The standing rule: any component shipping CRs of a CRD it installs must split the CRs into platform-config (dependsOn: platform), and second-order CRs whose CRD only appears after a controller reconciles go in platform-config-cdi. Remediation is a git change moving the resource; bootstrap/standup.sh crds is the belt-and-braces repair for a cluster already wedged.

2. A HelmRelease is genuinely failing its install/upgrade

Read the message. Known instances: the headlamp-0.40.1 -session-ttl chart/image desync; Grafana 13 OOM-looping after kps 65→88. Note platform's timeout: 5m versus Longhorn's HelmRelease timeout: 15m — the tier can time out while a legitimate Longhorn upgrade is still running. Give it another interval before concluding it is broken.

3. A ${TOKEN} problem in post-build substitution

Substitution fails open: an undefined variable becomes the empty string and the Kustomization still reports Ready — which is how k8sServiceHost: "" broke Cilium on 2 of 3 dev nodes on 2026-09-01. A parse error is the loud version and does fail the whole Kustomization (unable to parse variable name). If the message mentions envsubst or a variable, run the L0 check: conformance/tests/l0-static/flux-postbuild-substitution/check.sh. Note strict substitution became default-on in kustomize-controller v1.9.0, so a Flux upgrade flips this behaviour whether or not you opt in.

4. A remote kustomize base timed out

The kustomize-controller's git fetch has a 27s timeout and remote GitHub bases flake against it. external-snapshotter was vendored into the repo for this reason; platform/kubevirt still uses remote bases, so a BuildFail there is probably this. Remediation: vendor it the same way.

5. SOPS decryption

A tier that carries an encrypted Secret needs its own decryption stanza patched in by the overlay, by name. A newly split-out tier that inherited a secret but not the patch fails here.

Remediation notes

• Everything durable is a git change. flux reconcile only re-triggers; it does not fix a bad manifest, and it is blocked for agents here.
• Flux is level-triggered: an interrupted reconcile is retried, so a restart of a controller is safe and often the fastest way past a wedged in-flight apply (operator action).
• dependsOn is revision-gated, not merely Ready-gated: a dependent cannot reconcile revision N until its dependency has applied N. That is what makes the chain safe and also what makes one stuck tier so expensive.

When to escalate or stop

• If flux-system, cilium, platform or platform-config is the failing tier, treat it as a platform outage in progress — everything downstream is frozen, including the fix. Get the operator.
• Do not flux suspend a tier to make the alert stop. That trades a visible failure for /runbooks/flux-suspended and silent drift, which is how prod's whole platform tier ended up suspended and left that way after the 2026-09-04 external-dns incident.
• Do not enable driftDetection.mode: enabled on a CNI. During the 2026-09-01 Cilium incident the Helm storage manifest was the bad render, so drift correction would have actively re-asserted the broken value.
• If the GitRepository itself is not Ready, check Gitea before Flux — and remember prod deliberately reads from the in-cluster Service so that public DNS can never be in this path.

---

Source of truth: runbooks/flux-reconcile.md. Index: /runbooks/ Records: DECISIONS-LOG.md "2026-09-01 — INCIDENT: merging PR #4 briefly broke Cilium" (substitution fails open), the Headlamp gating stall, the oauth2-proxy cold-start deadlock, and the 2026-09-13 GitOps source deadlock.