# image-policy-stalled — An ImagePolicy or ImageRepository cannot resolve its tag **Alert:** `ImagePolicyStalled` (warning, 1h) **File:** `platform/monitoring/alerts.yaml`, group `platform-health.automated-updates` **Expression:** `max by (namespace, name, kind) (gotk_reconcile_condition{kind=~"ImagePolicy|ImageRepository", type="Ready", status="False"}) == 1` ## What this means here An `ImageRepository` cannot scan a registry, or an `ImagePolicy` cannot pick a tag out of what was scanned. The effect is narrower than `/runbooks/image-automation`: automation itself is healthy, but **this one service stops receiving updates** (R-UPD-01), while everything else continues. It is therefore easy to miss, which is what the 1h `for` and the per-object labels are for. The same framing applies as for image-automation: **this feature is authored and deliberately disabled on this cluster** and has been since 2026-08-24 (no Gitea write credential, and the `image-reflector-controller` / `image-automation-controller` components are not installed — the line is still commented out in `platform/kustomization.yaml`). Rendered apps emit `ImageRepository` and `ImagePolicy` CRs anyway, so the objects can exist with no controller to reconcile them; `catalog/render.py --no-image-automation` is the documented workaround. **There is no incident history for this alert** — the section below is mechanism, honestly labelled. One live constraint worth knowing while reading registry errors: the images in this platform come from `docker.io`, `ghcr.io`, `lscr.io`, `quay.io`, `registry.k8s.io` and `registry.gitlab.com`. Docker Hub's anonymous rate limit is the most likely real-world cause of a scan failure once automation is switched on. **This alert is currently INERT.** `count(gotk_reconcile_condition)` returned **0 series** on prod on 2026-09-13 — there is no Flux-controller scrape job among Prometheus's 19 jobs, so every `gotk_*` rule evaluates against nothing. Read the CRs directly with the commands below, and see `/runbooks/targets-down` for the full list of inert rule groups. ## Running the commands in this runbook `kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write `backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for agents here: every durable change goes through git. **`main` is a protected branch — a direct push is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, and it says so. ## First commands Which object, and the message (the message is nearly always the whole answer): ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get imagerepositories.image.toolkit.fluxcd.io -A \ -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,IMAGE:.spec.image,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,TAGS:.status.lastScanResult.tagCount; echo; \ kubectl --context k3s-prod get imagepolicies.image.toolkit.fluxcd.io -A \ -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,LATEST:.status.latestImage,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message' ``` Are the controllers installed at all? ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n flux-system get deploy | grep -i image || echo "image controllers NOT installed (expected on this cluster)"' ``` The reflector's log: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n flux-system logs deploy/image-reflector-controller --tail=80 2>&1 | tail -30' ``` Can the cluster reach the registry at all? (Default-deny is the baseline; a new namespace may simply not be allowed out.) ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get pods -A --field-selector status.phase!=Running | head; echo; \ kubectl --context k3s-prod get events -A --field-selector reason=Failed --sort-by=.lastTimestamp | grep -i "pull\|registry" | tail -10' ``` ## Likely causes, most likely first ### 1. The image controllers are not installed (expected state here) The CRs exist because `catalog/render.py` emits them; nothing reconciles them. Remediation: render with `--no-image-automation`, or have the operator finish enabling automation. Note that the Flux controllers were installed **imperatively** and are outside Flux's own inventory, so no git change can install them. ### 2. Registry rate limit Docker Hub anonymous pulls/scans are limited per IP, and all three nodes share one WAN address. The message says `toomanyrequests`. Remediation: a registry credential (SOPS-encrypted, in git), or scan less often via `spec.interval`. ### 3. Registry auth A private repository with no `secretRef`, or an expired token. The message names it. ### 4. The policy's range matches nothing `ImagePolicy` semver ranges come from each ServiceDefinition's `update.range` (for example `">=2024.0.0"` for pihole, `">=24.0.0"` for keycloak). A range that no published tag satisfies — or a repository that publishes only `latest` — leaves the policy with nothing to select. Remediation: fix the range in `catalog/.yaml` and re-render; a `latest`-only image cannot participate in automation at all and should not carry a policy. (CORRECTED 2026-10-30: this page used to name `lscr.io/linuxserver/jackett` and `plexinc/pms-docker` as `latest`-only. Neither is — both publish immutable version tags (`v0.24.2865-ls7`, `1.43.4.10903-e5521bd8c`) — they were merely *deployed* as `:latest`. Both are now pinned to those tags and maintained by Renovate, not by an ImagePolicy.) ### 5. Egress is blocked for that namespace Default-deny everywhere (R-NET-01) means outbound registry access is an explicit allow. Cross-read `/runbooks/denied-flows`, and in particular the post-Cilium-upgrade finding that `toEntities: world` no longer covers a call whose destination resolves to an in-cluster backend. ## When to escalate or stop - **Warning, 1h, single service: this does not page.** The consequence is one workload pinned at its current version. - **Do not widen an `update.range` to make the alert clear.** The range is the service's update policy (R-UPD-01); loosening it to satisfy a scanner is how an unintended major version arrives. - **Do not add registry credentials ad hoc** — new secret, DR path, operator's call. - If the same object has been stalled for days, the honest outcome is to remove its automation CRs (`--no-image-automation`) rather than leave a permanently-failing object training everyone to ignore the alert. Alert fatigue is a recorded, expensive failure mode on this cluster. --- Source of truth: `runbooks/image-policy-stalled.md`. Index: `/runbooks/` Records: `DECISIONS-LOG.md` (image automation authored-and-disabled; `--no-image-automation`); `catalog/*.yaml` (`update.channel` / `update.range` per service).