jakman85andClaude Opus 5
44983095e1
Every platform-health alert rule carries `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations over 34 distinct names, across platform/monitoring/alerts.yaml, platform/monitoring/pvc-capacity-alerts.yaml, platform/cnpg/alerts.yaml and platform/kubevirt/alerts.yaml. There was no runbooks/ directory and no HTTPRoute serving that path: an in-cluster probe of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404. Every alert that reached a human handed them a dead link at the exact moment the documentation is read, and the annotation made it look solved — the same green-surface-over-absent-mechanism class as the import Job that reported success while storing placeholder secrets. No alerts.yaml is touched and no runbook_url value changes. The existing URL scheme is made to resolve instead. WHAT LANDS - runbooks/<34>.md — one per name, each grounded in this cluster's real topology and, where one exists, its real incident history: the 2026-09-07 k3s-prod-1 etcd I/O starvation (measured PSI 11.36 vs 0.75/1.29, the exact journal lines, the four mechanisms proven inert), the Longhorn reboot tax and the freeze/thaw of concurrent-replica-rebuild-per-node-limit, Cilium's once-resolved egress interface cache (Ifindex 3 against a live 1780) and the absence of OSS failover, the CNPG 38-hour ContinuousArchiving=False with its measured 6.8 MiB/h disk-full clock and the 2026-09-12 aggregation-scope fix, the 2026-09-06 Keycloak lockout in which all 12 clients held literal placeholder secrets behind an entirely green cluster, the Velero R2 x-amz-tagging NotImplemented incompatibility and its v1.14.3-rc.2 pin, and swap re-enabling itself across reboots via systemd-gpt-auto-generator. Each page states what the alert means HERE, the first diagnostic commands, likely causes ordered by probability with a remediation each, and when to escalate or stop. - Every page uses FULLY-QUALIFIED resource names and says why: three CRDs register the kind `Backup` here, the bare short name resolves to Longhorn's (zero objects), so `kubectl get backup -A` prints "No resources found" on a cluster holding 6 Velero and 27 CNPG backups. Acting on that false negative is how a CLI quirk becomes data loss. - Every page states that kubectl only works via `infra/devcontainer/dev.sh run bash -c '...'` with an explicit `--context k3s-prod`, because dev.sh defaults its current-context to k3s-dev. - Pages also record which alert groups are INERT today (measured 2026-09-13: no Longhorn, cert-manager, Flux-controller or Hubble scrape job exists, so 13 rules evaluate against nothing) so nobody reads their silence as health. SERVING — runbooks/serving/ plus clusters/base/runbooks.yaml A two-replica nginx-unprivileged serving a ConfigMap generated from runbooks/*.md, at the apex `/runbooks` prefix, in its own non-gating post-platform Flux tier (dependsOn platform-config, wait:false) — the platform/homepage pattern, file for file. - The service catalog cannot express this route: ingressRoutes[].host is pattern-bound to a single DNS label plus the apex, so the bare apex is unrepresentable, and there is no path field. - Rejected: redirecting to the Gitea web UI. A runbook is read when the platform is broken, and Gitea is an app-tier workload whose own dependencies several of these runbooks document; Gateway API's RequestRedirect also has no "append .md" modifier, so it would need 34 hand-maintained ReplaceFullPath rules. - configMapGenerator maps each file to an EXTENSIONLESS key (node-down=node-down.md), so /runbooks/node-down is a plain static lookup with no rewrite rule. Served as text/plain: the namespace has DNS-only egress, so a CDN markdown renderer could not load anyway. - The tier declares NO postBuild.substituteFrom, deliberately: envsubst would run over the prose and silently empty every ${...} shell example (it fails open — the 2026-09-01 emptied Cilium k8sServiceHost). The HTTPRoute omits `hostnames:` and inherits the https-apex listener's, so nothing needs substituting. AUTH — PUBLIC, deliberately auth-mode=none. Gating these pages behind oauth2-proxy/Keycloak would make runbooks/keycloak-down unreadable when Keycloak is down and runbooks/keycloak-client-secrets unreadable during the exact 2026-09-06 failure mode. An operator who cannot log in is this route's primary reader. Content is operational procedure only, no secrets. The rationale is recorded in an annotation on the HTTPRoute so the route is left OUT of the Gateway ext-authz attachment when forward-auth enforcement lands. PROVEN — in-cluster probe through the Gateway on prod /runbooks/node-down HTTP 200 8613 bytes text/plain; charset=utf-8 /runbooks/keycloak-down HTTP 200 9378 bytes text/plain; charset=utf-8 /runbooks/cnpg-archiving HTTP 200 11729 bytes text/plain; charset=utf-8 /runbooks/longhorn-faulted HTTP 200 9135 bytes text/plain; charset=utf-8 /runbooks/vpn-down HTTP 200 11092 bytes text/plain; charset=utf-8 /runbooks/backup-failure HTTP 200 9604 bytes text/plain; charset=utf-8 HTTPRoute Accepted by io.cilium/gateway-controller on sectionName https-apex; the `/runbooks` prefix wins over homepage's apex `/` by Gateway API longest-path precedence. GUARD — conformance/tests/l0-static/alert-runbooks-resolvable (R-OBS-04) 363 offline assertions. It checks BOTH halves, because the regression is silent in both directions: an alert with a new runbook_url and no runbook links to a 404, AND a runbook committed without its configMapGenerator entry is in git, reviewed, merged, and still 404s in the cluster. Also asserts a quality floor so a stub cannot make the link look fixed, and that route/listener/nginx prefix agree with the annotations. Proven to fail, then reverted: a bogus runbook_url (2 FAIL), a runbook removed from the generator (1 FAIL), and a 31-byte stub (7 FAIL). Co-Authored-By: Claude Opus 5 <[email protected]>
161 lines
9.9 KiB
Markdown
161 lines
9.9 KiB
Markdown
# flux-reconcile — A Flux resource has failed to reconcile for 15 minutes
|
|
|
|
**Alert:** `FluxReconciliationFailure` (warning, 15m)
|
|
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.flux`
|
|
**Expression:** `max by (namespace, name, kind) (gotk_reconcile_condition{type="Ready", status="False"}) == 1`
|
|
|
|
## What this means here
|
|
|
|
Something in the GitOps chain is stuck at Ready=False. On this cluster that is more consequential than
|
|
it sounds, because **`git push origin main` IS the deploy** — there is no other apply step. A stuck
|
|
Kustomization means git and the cluster have diverged and nothing will close the gap.
|
|
|
|
The chain has **14 tiers**: `flux-system` → `cilium` → `platform` → `platform-config` →
|
|
`platform-config-cdi` → `apps-wave-0/1/2`, plus the non-gating leaves `apps-vms`, `oauth2-proxy`,
|
|
`headlamp`, `homepage`, `crafty`, `runbooks`, `backup`. Which tier is failing decides the blast radius:
|
|
|
|
- **`flux-system`, `cilium`, `platform`, `platform-config` are gating (`wait: true`).** One Failed
|
|
HelmRelease in the `platform` tier stalls **everything downstream** on "dependency not ready". That
|
|
is not theoretical: a crashlooping Headlamp HelmRelease once stalled `platform-config`,
|
|
`oauth2-proxy` and every apps wave (the "Headlamp gating stall"), and oauth2-proxy twice deadlocked
|
|
by **blocking its own fix** — it sat in the gating tier while needing Keycloak + trusted TLS +
|
|
split-horizon DNS, so `platform-config`, which applies the fix, could never reconcile. Both were
|
|
resolved by moving the leaf into its own `wait: false` tier.
|
|
- **The leaf tiers are `wait: false`** precisely so their unhealth can never gate anything.
|
|
|
|
Prod's source is `http://gitea.gitea.svc.cluster.local:3000` — the **in-cluster Service**, on purpose.
|
|
It used to be a public hostname, and the apex flip taught the expensive version of that lesson: the
|
|
HTTPRoute serving the source's own hostname lives in a downstream tier that cannot reconcile while the
|
|
source is down, so `srcReady=False` / *"unable to list remote"* became a **deadlock**. If you are
|
|
tempted to point the GitRepository at a public name, don't.
|
|
**This alert is currently INERT.** `count(gotk_reconcile_condition)` and `count(gotk_suspend_status)`
|
|
both returned **0 series** on prod on 2026-09-13: there is no scrape job for the Flux controllers in
|
|
Prometheus's 19 jobs, so every `gotk_*`-based rule (`FluxReconciliationFailure`,
|
|
`FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`,
|
|
`UpdateHealthGateFailed`) evaluates against nothing. **Flux health is therefore unmonitored today** and
|
|
must be read from the CRs directly, with the commands below. See `/runbooks/targets-down` for the full
|
|
list of inert rule groups. Adding the Flux controllers' ServiceMonitor is open work.
|
|
## Running the commands in this runbook
|
|
|
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
|
that hang is not the cluster being down. Every command below runs as:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
|
|
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
|
|
defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
|
|
about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
|
|
Do not run two `dev.sh` invocations at once — they share one container and interfere.
|
|
|
|
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
|
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
|
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
|
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
|
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
|
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
|
|
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
|
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
|
Where this runbook gives a mutating command it is for an operator, and it says so.
|
|
|
|
## First commands
|
|
|
|
The whole chain at a glance — this is the command to run first, every time:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \
|
|
-o custom-columns=NAME:.metadata.name,PATH:.spec.path,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,REV:.status.lastAppliedRevision'
|
|
```
|
|
|
|
Is the SOURCE healthy? If not, nothing downstream can be:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n flux-system get gitrepositories.source.toolkit.fluxcd.io \
|
|
-o custom-columns=NAME:.metadata.name,URL:.spec.url,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message'
|
|
```
|
|
|
|
HelmReleases (the usual culprit inside `platform`):
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get helmreleases.helm.toolkit.fluxcd.io -A \
|
|
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message'
|
|
```
|
|
|
|
The controller's own view, with the real error text:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n flux-system logs deploy/kustomize-controller --tail=80; echo ====; \
|
|
kubectl --context k3s-prod -n flux-system logs deploy/helm-controller --tail=80'
|
|
```
|
|
|
|
## Likely causes, most likely first
|
|
|
|
### 1. "no matches for kind X" — a CR applied before its CRD
|
|
The recurring structural failure here. Flux server-side-apply fails the **whole** apply, so a tier that
|
|
ships both a CRD-installing HelmRelease and CRs of that CRD can never converge — the CRD never gets
|
|
created because the apply that would create it also contains the CR. Symptoms:
|
|
`no matches for kind "ClusterIssuer"` / `"PrometheusRule"`. The standing rule: **any component shipping
|
|
CRs of a CRD it installs must split the CRs into `platform-config`** (`dependsOn: platform`), and
|
|
second-order CRs whose CRD only appears after a controller reconciles go in `platform-config-cdi`.
|
|
Remediation is a git change moving the resource; `bootstrap/standup.sh crds` is the belt-and-braces
|
|
repair for a cluster already wedged.
|
|
|
|
### 2. A HelmRelease is genuinely failing its install/upgrade
|
|
Read the message. Known instances: the `headlamp-0.40.1` `-session-ttl` chart/image desync; Grafana 13
|
|
OOM-looping after kps 65→88. Note `platform`'s `timeout: 5m` versus Longhorn's HelmRelease
|
|
`timeout: 15m` — **the tier can time out while a legitimate Longhorn upgrade is still running**. Give
|
|
it another interval before concluding it is broken.
|
|
|
|
### 3. A `${TOKEN}` problem in post-build substitution
|
|
Substitution **fails open**: an undefined variable becomes the empty string and the Kustomization still
|
|
reports Ready — which is how `k8sServiceHost: ""` broke Cilium on 2 of 3 dev nodes on 2026-09-01. A
|
|
*parse* error is the loud version and does fail the whole Kustomization
|
|
(`unable to parse variable name`). If the message mentions envsubst or a variable, run the L0 check:
|
|
`conformance/tests/l0-static/flux-postbuild-substitution/check.sh`. Note **strict substitution became
|
|
default-on in kustomize-controller v1.9.0**, so a Flux upgrade flips this behaviour whether or not you
|
|
opt in.
|
|
|
|
### 4. A remote kustomize base timed out
|
|
The kustomize-controller's git fetch has a 27s timeout and remote GitHub bases flake against it.
|
|
`external-snapshotter` was vendored into the repo for this reason; **`platform/kubevirt` still uses
|
|
remote bases**, so a `BuildFail` there is probably this. Remediation: vendor it the same way.
|
|
|
|
### 5. SOPS decryption
|
|
A tier that carries an encrypted Secret needs its own `decryption` stanza patched in by the overlay, by
|
|
name. A newly split-out tier that inherited a secret but not the patch fails here.
|
|
|
|
## Remediation notes
|
|
|
|
- Everything durable is a **git change**. `flux reconcile` only re-triggers; it does not fix a bad
|
|
manifest, and it is blocked for agents here.
|
|
- Flux is **level-triggered**: an interrupted reconcile is retried, so a restart of a controller is
|
|
safe and often the fastest way past a wedged in-flight apply (operator action).
|
|
- `dependsOn` is **revision-gated**, not merely Ready-gated: a dependent cannot reconcile revision N
|
|
until its dependency has applied N. That is what makes the chain safe and also what makes one stuck
|
|
tier so expensive.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **If `flux-system`, `cilium`, `platform` or `platform-config` is the failing tier, treat it as a
|
|
platform outage in progress** — everything downstream is frozen, including the fix. Get the operator.
|
|
- **Do not `flux suspend` a tier to make the alert stop.** That trades a visible failure for
|
|
`/runbooks/flux-suspended` and silent drift, which is how prod's whole `platform` tier ended up
|
|
suspended and left that way after the 2026-09-04 external-dns incident.
|
|
- **Do not enable `driftDetection.mode: enabled` on a CNI.** During the 2026-09-01 Cilium incident the
|
|
Helm storage manifest *was* the bad render, so drift correction would have actively re-asserted the
|
|
broken value.
|
|
- If the GitRepository itself is not Ready, check Gitea before Flux — and remember prod deliberately
|
|
reads from the in-cluster Service so that public DNS can never be in this path.
|
|
|
|
---
|
|
Source of truth: `runbooks/flux-reconcile.md`. Index: `/runbooks/`
|
|
Records: `DECISIONS-LOG.md` "2026-09-01 — INCIDENT: merging PR #4 briefly broke Cilium" (substitution
|
|
fails open), the Headlamp gating stall, the oauth2-proxy cold-start deadlock, and the 2026-09-13
|
|
GitOps source deadlock.
|