jakman85andClaude Opus 5
8e017fad91
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
132 lines
7.6 KiB
Markdown
132 lines
7.6 KiB
Markdown
# image-automation — Flux's image-update-automation cannot write back to git
|
|
|
|
**Alert:** `ImageUpdateAutomationFailing` (warning, 30m)
|
|
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.automated-updates`
|
|
**Expression:** `max by (namespace, name) (gotk_reconcile_condition{kind="ImageUpdateAutomation", type="Ready", status="False"}) == 1`
|
|
|
|
## What this means here
|
|
|
|
`ImageUpdateAutomation` is the write-back half of hands-off updates (ADR-0009): it commits new image
|
|
tags into git so Flux then deploys them. When it is Ready=False, **stable image advances silently
|
|
stop** — nothing breaks, nothing degrades, the platform just quietly stops receiving updates. That is
|
|
the R-UPD-02 failure mode this alert exists for.
|
|
|
|
**Read this first: on this cluster, image automation is authored and DELIBERATELY DISABLED.** It has
|
|
been that way since 2026-08-24, pending (a) a Gitea write credential for the automation to push with
|
|
and (b) `flux install --components-extra=image-reflector-controller,image-automation-controller`. The
|
|
line is still commented out in `platform/kustomization.yaml`. So:
|
|
|
|
- If this alert fires, the most likely explanation is that **the feature was partially enabled** —
|
|
someone uncommented the component or a rendered app started emitting `ImageRepository`/`ImagePolicy`
|
|
CRs whose controllers are not installed.
|
|
- Rendered apps emit those CRs by default; `catalog/render.py --no-image-automation` is the documented
|
|
workaround for exactly this.
|
|
- There is consequently **no real incident history** for this alert. Everything below is mechanism, not
|
|
post-mortem, and it says so honestly rather than inventing precedent.
|
|
**This alert is currently INERT.** `count(gotk_reconcile_condition)` returned **0 series** on prod on
|
|
2026-09-13 — there is no Flux-controller scrape job among Prometheus's 19 jobs, so every `gotk_*` rule
|
|
evaluates against nothing. Read the CRs directly with the commands below, and see
|
|
`/runbooks/targets-down` for the full list of inert rule groups.
|
|
## Running the commands in this runbook
|
|
|
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
|
that hang is not the cluster being down. Every command below runs as:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
|
|
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
|
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
|
|
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
|
|
the same thing next month as it does today; one that relies on the default does not, and the failure is
|
|
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
|
|
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
|
|
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
|
|
interfere.
|
|
|
|
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
|
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
|
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
|
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
|
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
|
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
|
|
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
|
and it says so.
|
|
|
|
## First commands
|
|
|
|
Are the controllers even installed?
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n flux-system get deploy | sort; echo; \
|
|
kubectl --context k3s-prod api-resources --api-group=image.toolkit.fluxcd.io'
|
|
```
|
|
|
|
The automation objects and their messages:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get imageupdateautomations.image.toolkit.fluxcd.io -A \
|
|
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,LASTRUN:.status.lastAutomationRunTime'
|
|
```
|
|
|
|
The controller log — the git error is always in here:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n flux-system logs deploy/image-automation-controller --tail=100 2>&1 | tail -40'
|
|
```
|
|
|
|
Is the repo configured with a write path at all?
|
|
|
|
```
|
|
grep -rn "image-automation\|ImageUpdateAutomation\|components-extra" platform/kustomization.yaml bootstrap/ clusters/ 2>/dev/null | head -20
|
|
```
|
|
|
|
## Likely causes, most likely first
|
|
|
|
### 1. The controllers are not installed (expected state here)
|
|
`api-resources --api-group=image.toolkit.fluxcd.io` returns nothing, or the deployments are absent.
|
|
Then the CRs are inert objects and the alert reflects a half-enabled feature. Remediation: either
|
|
finish enabling it (an operator runs `flux install --components-extra=...` — note that Flux's own
|
|
controllers were installed **imperatively** and carry no Flux ownership labels, so **nothing under
|
|
`clusters/**` can install a controller or change a controller flag**), or stop rendering the CRs with
|
|
`catalog/render.py --no-image-automation`.
|
|
|
|
### 2. No git write credential
|
|
The automation must push. Prod's source is `http://gitea.gitea.svc.cluster.local:3000`, and the
|
|
read-only source credential is not a write credential. The controller log will say authentication
|
|
failed or permission denied. Remediation: a Gitea token in a SOPS-encrypted Secret referenced by the
|
|
`GitRepository`/automation, added in git — and note the standing rule from the OIDC incident:
|
|
**non-secret environment-specific values must not live inside an encrypted Secret**, because
|
|
`${BASE_DOMAIN}` substitution can never reach inside one.
|
|
|
|
### 3. It cannot push because the branch moved
|
|
Automation commits to a branch; a force-push or a protected branch rejects it. The log names the ref.
|
|
|
|
### 4. The commit is being made but the Kustomization is not picking it up
|
|
Then this is not the right runbook — go to `/runbooks/flux-reconcile`.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **This is a warning with a 30m `for` for a reason: nothing is broken.** Do not treat it as an
|
|
outage. The consequence is "no new image versions", which is measured in days.
|
|
- **Do not create a Gitea write credential ad hoc.** It is a new secret in the DR path and belongs to
|
|
the operator, SOPS-encrypted, in git.
|
|
- **Do not enable the extra Flux controllers to make the alert stop.** Enabling automation on a
|
|
cluster that has never run it will start proposing image bumps across every workload at once; that is
|
|
a planned change, with `/runbooks/update-rollback` as its companion.
|
|
- If the alert is firing because the feature is half-enabled, the honest fix is to finish or to revert
|
|
— both are git changes and both want the operator's decision.
|
|
|
|
---
|
|
Source of truth: `runbooks/image-automation.md`. Index: `/runbooks/`
|
|
Records: `DECISIONS-LOG.md` (image automation authored-and-disabled since 2026-08-24; the Flux
|
|
controllers are outside Flux's own inventory).
|