home-network-k3s-impl/runbooks/image-automation.md

# image-automation — Flux's image-update-automation cannot write back to git

**Alert:** `ImageUpdateAutomationFailing` (warning, 30m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.automated-updates`
**Expression:** `max by (namespace, name) (gotk_reconcile_condition{kind="ImageUpdateAutomation", type="Ready", status="False"}) == 1`

## What this means here

`ImageUpdateAutomation` is the write-back half of hands-off updates (ADR-0009): it commits new image
tags into git so Flux then deploys them. When it is Ready=False, **stable image advances silently
stop** — nothing breaks, nothing degrades, the platform just quietly stops receiving updates. That is
the R-UPD-02 failure mode this alert exists for.

**Read this first: on this cluster, image automation is authored and DELIBERATELY DISABLED.** It has
been that way since 2026-08-24, pending (a) a Gitea write credential for the automation to push with
and (b) `flux install --components-extra=image-reflector-controller,image-automation-controller`. The
line is still commented out in `platform/kustomization.yaml`. So:

- If this alert fires, the most likely explanation is that **the feature was partially enabled** —
  someone uncommented the component or a rendered app started emitting `ImageRepository`/`ImagePolicy`
  CRs whose controllers are not installed.
- Rendered apps emit those CRs by default; `catalog/render.py --no-image-automation` is the documented
  workaround for exactly this.
- There is consequently **no real incident history** for this alert. Everything below is mechanism, not
  post-mortem, and it says so honestly rather than inventing precedent.
**This alert is currently INERT.** `count(gotk_reconcile_condition)` returned **0 series** on prod on
2026-09-13 — there is no Flux-controller scrape job among Prometheus's 19 jobs, so every `gotk_*` rule
evaluates against nothing. Read the CRs directly with the commands below, and see
`/runbooks/targets-down` for the full list of inert rule groups.
## Running the commands in this runbook

`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.

## First commands

Are the controllers even installed?

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n flux-system get deploy | sort; echo; \
   kubectl --context k3s-prod api-resources --api-group=image.toolkit.fluxcd.io'
```

The automation objects and their messages:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get imageupdateautomations.image.toolkit.fluxcd.io -A \
     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,LASTRUN:.status.lastAutomationRunTime'
```

The controller log — the git error is always in here:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n flux-system logs deploy/image-automation-controller --tail=100 2>&1 | tail -40'
```

Is the repo configured with a write path at all?

```
grep -rn "image-automation\|ImageUpdateAutomation\|components-extra" platform/kustomization.yaml bootstrap/ clusters/ 2>/dev/null | head -20
```

## Likely causes, most likely first

### 1. The controllers are not installed (expected state here)
`api-resources --api-group=image.toolkit.fluxcd.io` returns nothing, or the deployments are absent.
Then the CRs are inert objects and the alert reflects a half-enabled feature. Remediation: either
finish enabling it (an operator runs `flux install --components-extra=...` — note that Flux's own
controllers were installed **imperatively** and carry no Flux ownership labels, so **nothing under
`clusters/**` can install a controller or change a controller flag**), or stop rendering the CRs with
`catalog/render.py --no-image-automation`.

### 2. No git write credential
The automation must push. Prod's source is `http://gitea.gitea.svc.cluster.local:3000`, and the
read-only source credential is not a write credential. The controller log will say authentication
failed or permission denied. Remediation: a Gitea token in a SOPS-encrypted Secret referenced by the
`GitRepository`/automation, added in git — and note the standing rule from the OIDC incident:
**non-secret environment-specific values must not live inside an encrypted Secret**, because
`${BASE_DOMAIN}` substitution can never reach inside one.

### 3. It cannot push because the branch moved
Automation commits to a branch; a force-push or a protected branch rejects it. The log names the ref.

### 4. The commit is being made but the Kustomization is not picking it up
Then this is not the right runbook — go to `/runbooks/flux-reconcile`.

## When to escalate or stop

- **This is a warning with a 30m `for` for a reason: nothing is broken.** Do not treat it as an
  outage. The consequence is "no new image versions", which is measured in days.
- **Do not create a Gitea write credential ad hoc.** It is a new secret in the DR path and belongs to
  the operator, SOPS-encrypted, in git.
- **Do not enable the extra Flux controllers to make the alert stop.** Enabling automation on a
  cluster that has never run it will start proposing image bumps across every workload at once; that is
  a planned change, with `/runbooks/update-rollback` as its companion.
- If the alert is firing because the feature is half-enabled, the honest fix is to finish or to revert
  — both are git changes and both want the operator's decision.

---
Source of truth: `runbooks/image-automation.md`. Index: `/runbooks/`
Records: `DECISIONS-LOG.md` (image automation authored-and-disabled since 2026-08-24; the Flux
controllers are outside Flux's own inventory).