home-network-k3s-impl/runbooks/image-policy-stalled.md

image-policy-stalled — An ImagePolicy or ImageRepository cannot resolve its tag

Alert: ImagePolicyStalled (warning, 1h) File: platform/monitoring/alerts.yaml, group platform-health.automated-updates Expression: max by (namespace, name, kind) (gotk_reconcile_condition{kind=~"ImagePolicy|ImageRepository", type="Ready", status="False"}) == 1

What this means here

An ImageRepository cannot scan a registry, or an ImagePolicy cannot pick a tag out of what was scanned. The effect is narrower than /runbooks/image-automation: automation itself is healthy, but this one service stops receiving updates (R-UPD-01), while everything else continues. It is therefore easy to miss, which is what the 1h for and the per-object labels are for.

The same framing applies as for image-automation: this feature is authored and deliberately disabled on this cluster and has been since 2026-08-24 (no Gitea write credential, and the image-reflector-controller / image-automation-controller components are not installed — the line is still commented out in platform/kustomization.yaml). Rendered apps emit ImageRepository and ImagePolicy CRs anyway, so the objects can exist with no controller to reconcile them; catalog/render.py --no-image-automation is the documented workaround. There is no incident history for this alert — the section below is mechanism, honestly labelled.

One live constraint worth knowing while reading registry errors: the images in this platform come from docker.io, ghcr.io, lscr.io, quay.io, registry.k8s.io and registry.gitlab.com. Docker Hub's anonymous rate limit is the most likely real-world cause of a scan failure once automation is switched on. This alert is currently INERT. count(gotk_reconcile_condition) returned 0 series on prod on 2026-09-13 — there is no Flux-controller scrape job among Prometheus's 19 jobs, so every gotk_* rule evaluates against nothing. Read the CRs directly with the commands below, and see /runbooks/targets-down for the full list of inert rule groups.

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two dev.sh invocations at once; they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git. main is a protected branch — a direct push is rejected. A change lands by pushing a branch, opening a PR, and merging it once the gate is green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

Which object, and the message (the message is nearly always the whole answer):

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get imagerepositories.image.toolkit.fluxcd.io -A \
     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,IMAGE:.spec.image,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,TAGS:.status.lastScanResult.tagCount; echo; \
   kubectl --context k3s-prod get imagepolicies.image.toolkit.fluxcd.io -A \
     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,LATEST:.status.latestImage,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message'

Are the controllers installed at all?

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n flux-system get deploy | grep -i image || echo "image controllers NOT installed (expected on this cluster)"'

The reflector's log:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n flux-system logs deploy/image-reflector-controller --tail=80 2>&1 | tail -30'

Can the cluster reach the registry at all? (Default-deny is the baseline; a new namespace may simply not be allowed out.)

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get pods -A --field-selector status.phase!=Running | head; echo; \
   kubectl --context k3s-prod get events -A --field-selector reason=Failed --sort-by=.lastTimestamp | grep -i "pull\|registry" | tail -10'

Likely causes, most likely first

1. The image controllers are not installed (expected state here)

The CRs exist because catalog/render.py emits them; nothing reconciles them. Remediation: render with --no-image-automation, or have the operator finish enabling automation. Note that the Flux controllers were installed imperatively and are outside Flux's own inventory, so no git change can install them.

2. Registry rate limit

Docker Hub anonymous pulls/scans are limited per IP, and all three nodes share one WAN address. The message says toomanyrequests. Remediation: a registry credential (SOPS-encrypted, in git), or scan less often via spec.interval.

3. Registry auth

A private repository with no secretRef, or an expired token. The message names it.

4. The policy's range matches nothing

ImagePolicy semver ranges come from each ServiceDefinition's update.range (for example ">=2024.0.0" for pihole, ">=24.0.0" for keycloak). A range that no published tag satisfies — or a repository that publishes only latest — leaves the policy with nothing to select. Remediation: fix the range in catalog/<service>.yaml and re-render; a latest-only image cannot participate in automation at all and should not carry a policy. (CORRECTED 2026-10-30: this page used to name lscr.io/linuxserver/jackett and plexinc/pms-docker as latest-only. Neither is — both publish immutable version tags (v0.24.2865-ls7, 1.43.4.10903-e5521bd8c) — they were merely deployed as :latest. Both are now pinned to those tags and maintained by Renovate, not by an ImagePolicy.)

5. Egress is blocked for that namespace

Default-deny everywhere (R-NET-01) means outbound registry access is an explicit allow. Cross-read /runbooks/denied-flows, and in particular the post-Cilium-upgrade finding that toEntities: world no longer covers a call whose destination resolves to an in-cluster backend.

When to escalate or stop

• Warning, 1h, single service: this does not page. The consequence is one workload pinned at its current version.
• Do not widen an update.range to make the alert clear. The range is the service's update policy (R-UPD-01); loosening it to satisfy a scanner is how an unintended major version arrives.
• Do not add registry credentials ad hoc — new secret, DR path, operator's call.
• If the same object has been stalled for days, the honest outcome is to remove its automation CRs (--no-image-automation) rather than leave a permanently-failing object training everyone to ignore the alert. Alert fatigue is a recorded, expensive failure mode on this cluster.

---

Source of truth: runbooks/image-policy-stalled.md. Index: /runbooks/ Records: DECISIONS-LOG.md (image automation authored-and-disabled; --no-image-automation); catalog/*.yaml (update.channel / update.range per service).