jakman85andClaude Opus 5
8e017fad91
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
167 lines
10 KiB
Markdown
167 lines
10 KiB
Markdown
# flux-reconcile — A Flux resource has failed to reconcile for 15 minutes
|
|
|
|
**Alert:** `FluxReconciliationFailure` (warning, 15m)
|
|
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.flux`
|
|
**Expression:** `max by (namespace, name, kind) (gotk_reconcile_condition{type="Ready", status="False"}) == 1`
|
|
|
|
## What this means here
|
|
|
|
Something in the GitOps chain is stuck at Ready=False. On this cluster that is more consequential than
|
|
it sounds, because **merging a PR into `main` IS the deploy** — there is no other apply step. A stuck
|
|
Kustomization means git and the cluster have diverged and nothing will close the gap.
|
|
|
|
The chain has **14 tiers**: `flux-system` → `cilium` → `platform` → `platform-config` →
|
|
`platform-config-cdi` → `apps-wave-0/1/2`, plus the non-gating leaves `apps-vms`, `oauth2-proxy`,
|
|
`headlamp`, `homepage`, `crafty`, `runbooks`, `backup`. Which tier is failing decides the blast radius:
|
|
|
|
- **`flux-system`, `cilium`, `platform`, `platform-config` are gating (`wait: true`).** One Failed
|
|
HelmRelease in the `platform` tier stalls **everything downstream** on "dependency not ready". That
|
|
is not theoretical: a crashlooping Headlamp HelmRelease once stalled `platform-config`,
|
|
`oauth2-proxy` and every apps wave (the "Headlamp gating stall"), and oauth2-proxy twice deadlocked
|
|
by **blocking its own fix** — it sat in the gating tier while needing Keycloak + trusted TLS +
|
|
split-horizon DNS, so `platform-config`, which applies the fix, could never reconcile. Both were
|
|
resolved by moving the leaf into its own `wait: false` tier.
|
|
- **The leaf tiers are `wait: false`** precisely so their unhealth can never gate anything.
|
|
|
|
Prod's source is `http://gitea.gitea.svc.cluster.local:3000` — the **in-cluster Service**, on purpose.
|
|
It used to be a public hostname, and the apex flip taught the expensive version of that lesson: the
|
|
HTTPRoute serving the source's own hostname lives in a downstream tier that cannot reconcile while the
|
|
source is down, so `srcReady=False` / *"unable to list remote"* became a **deadlock**. If you are
|
|
tempted to point the GitRepository at a public name, don't.
|
|
**This alert is currently INERT.** `count(gotk_reconcile_condition)` and `count(gotk_suspend_status)`
|
|
both returned **0 series** on prod on 2026-09-13: there is no scrape job for the Flux controllers in
|
|
Prometheus's 19 jobs, so every `gotk_*`-based rule (`FluxReconciliationFailure`,
|
|
`FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`,
|
|
`UpdateHealthGateFailed`) evaluates against nothing. **Flux health is therefore unmonitored today** and
|
|
must be read from the CRs directly, with the commands below. See `/runbooks/targets-down` for the full
|
|
list of inert rule groups. Adding the Flux controllers' ServiceMonitor is open work.
|
|
## Running the commands in this runbook
|
|
|
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
|
that hang is not the cluster being down. Every command below runs as:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
|
|
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
|
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
|
|
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
|
|
the same thing next month as it does today; one that relies on the default does not, and the failure is
|
|
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
|
|
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
|
|
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
|
|
interfere.
|
|
|
|
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
|
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
|
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
|
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
|
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
|
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
|
|
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
|
and it says so.
|
|
|
|
## First commands
|
|
|
|
The whole chain at a glance — this is the command to run first, every time:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \
|
|
-o custom-columns=NAME:.metadata.name,PATH:.spec.path,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,REV:.status.lastAppliedRevision'
|
|
```
|
|
|
|
Is the SOURCE healthy? If not, nothing downstream can be:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n flux-system get gitrepositories.source.toolkit.fluxcd.io \
|
|
-o custom-columns=NAME:.metadata.name,URL:.spec.url,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message'
|
|
```
|
|
|
|
HelmReleases (the usual culprit inside `platform`):
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get helmreleases.helm.toolkit.fluxcd.io -A \
|
|
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message'
|
|
```
|
|
|
|
The controller's own view, with the real error text:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n flux-system logs deploy/kustomize-controller --tail=80; echo ====; \
|
|
kubectl --context k3s-prod -n flux-system logs deploy/helm-controller --tail=80'
|
|
```
|
|
|
|
## Likely causes, most likely first
|
|
|
|
### 1. "no matches for kind X" — a CR applied before its CRD
|
|
The recurring structural failure here. Flux server-side-apply fails the **whole** apply, so a tier that
|
|
ships both a CRD-installing HelmRelease and CRs of that CRD can never converge — the CRD never gets
|
|
created because the apply that would create it also contains the CR. Symptoms:
|
|
`no matches for kind "ClusterIssuer"` / `"PrometheusRule"`. The standing rule: **any component shipping
|
|
CRs of a CRD it installs must split the CRs into `platform-config`** (`dependsOn: platform`), and
|
|
second-order CRs whose CRD only appears after a controller reconciles go in `platform-config-cdi`.
|
|
Remediation is a git change moving the resource; `bootstrap/standup.sh crds` is the belt-and-braces
|
|
repair for a cluster already wedged.
|
|
|
|
### 2. A HelmRelease is genuinely failing its install/upgrade
|
|
Read the message. Known instances: the `headlamp-0.40.1` `-session-ttl` chart/image desync; Grafana 13
|
|
OOM-looping after kps 65→88. Note `platform`'s `timeout: 5m` versus Longhorn's HelmRelease
|
|
`timeout: 15m` — **the tier can time out while a legitimate Longhorn upgrade is still running**. Give
|
|
it another interval before concluding it is broken.
|
|
|
|
### 3. A `${TOKEN}` problem in post-build substitution
|
|
Substitution **fails open**: an undefined variable becomes the empty string and the Kustomization still
|
|
reports Ready — which is how `k8sServiceHost: ""` broke Cilium on 2 of 3 dev nodes on 2026-09-01. A
|
|
*parse* error is the loud version and does fail the whole Kustomization
|
|
(`unable to parse variable name`). If the message mentions envsubst or a variable, run the L0 check:
|
|
`conformance/tests/l0-static/flux-postbuild-substitution/check.sh`. Note **strict substitution became
|
|
default-on in kustomize-controller v1.9.0**, so a Flux upgrade flips this behaviour whether or not you
|
|
opt in.
|
|
|
|
### 4. A remote kustomize base timed out
|
|
The kustomize-controller's git fetch has a 27s timeout and remote GitHub bases flake against it.
|
|
`external-snapshotter` was vendored into the repo for this reason; **`platform/kubevirt` still uses
|
|
remote bases**, so a `BuildFail` there is probably this. Remediation: vendor it the same way.
|
|
|
|
### 5. SOPS decryption
|
|
A tier that carries an encrypted Secret needs its own `decryption` stanza patched in by the overlay, by
|
|
name. A newly split-out tier that inherited a secret but not the patch fails here.
|
|
|
|
## Remediation notes
|
|
|
|
- Everything durable is a **git change**. `flux reconcile` only re-triggers; it does not fix a bad
|
|
manifest, and it is blocked for agents here.
|
|
- Flux is **level-triggered**: an interrupted reconcile is retried, so a restart of a controller is
|
|
safe and often the fastest way past a wedged in-flight apply (operator action).
|
|
- `dependsOn` is **revision-gated**, not merely Ready-gated: a dependent cannot reconcile revision N
|
|
until its dependency has applied N. That is what makes the chain safe and also what makes one stuck
|
|
tier so expensive.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **If `flux-system`, `cilium`, `platform` or `platform-config` is the failing tier, treat it as a
|
|
platform outage in progress** — everything downstream is frozen, including the fix. Get the operator.
|
|
- **Do not `flux suspend` a tier to make the alert stop.** That trades a visible failure for
|
|
`/runbooks/flux-suspended` and silent drift, which is how prod's whole `platform` tier ended up
|
|
suspended and left that way after the 2026-09-04 external-dns incident.
|
|
- **Do not enable `driftDetection.mode: enabled` on a CNI.** During the 2026-09-01 Cilium incident the
|
|
Helm storage manifest *was* the bad render, so drift correction would have actively re-asserted the
|
|
broken value.
|
|
- If the GitRepository itself is not Ready, check Gitea before Flux — and remember prod deliberately
|
|
reads from the in-cluster Service so that public DNS can never be in this path.
|
|
|
|
---
|
|
Source of truth: `runbooks/flux-reconcile.md`. Index: `/runbooks/`
|
|
Records: `DECISIONS-LOG.md` "2026-09-01 — INCIDENT: merging PR #4 briefly broke Cilium" (substitution
|
|
fails open), the Headlamp gating stall, the oauth2-proxy cold-start deadlock, and the 2026-09-13
|
|
GitOps source deadlock.
|