jakman85andClaude Opus 5
8e017fad91
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
8.3 KiB
image-policy-stalled — An ImagePolicy or ImageRepository cannot resolve its tag
Alert: ImagePolicyStalled (warning, 1h)
File: platform/monitoring/alerts.yaml, group platform-health.automated-updates
Expression: max by (namespace, name, kind) (gotk_reconcile_condition{kind=~"ImagePolicy|ImageRepository", type="Ready", status="False"}) == 1
What this means here
An ImageRepository cannot scan a registry, or an ImagePolicy cannot pick a tag out of what was
scanned. The effect is narrower than /runbooks/image-automation: automation itself is healthy, but
this one service stops receiving updates (R-UPD-01), while everything else continues. It is
therefore easy to miss, which is what the 1h for and the per-object labels are for.
The same framing applies as for image-automation: this feature is authored and deliberately disabled
on this cluster and has been since 2026-08-24 (no Gitea write credential, and the
image-reflector-controller / image-automation-controller components are not installed — the line is
still commented out in platform/kustomization.yaml). Rendered apps emit ImageRepository and
ImagePolicy CRs anyway, so the objects can exist with no controller to reconcile them;
catalog/render.py --no-image-automation is the documented workaround. There is no incident history
for this alert — the section below is mechanism, honestly labelled.
One live constraint worth knowing while reading registry errors: the images in this platform come from
docker.io, ghcr.io, lscr.io, quay.io, registry.k8s.io and registry.gitlab.com. Docker
Hub's anonymous rate limit is the most likely real-world cause of a scan failure once automation is
switched on.
This alert is currently INERT. count(gotk_reconcile_condition) returned 0 series on prod on
2026-09-13 — there is no Flux-controller scrape job among Prometheus's 19 jobs, so every gotk_* rule
evaluates against nothing. Read the CRs directly with the commands below, and see
/runbooks/targets-down for the full list of inert rule groups.
Running the commands in this runbook
kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's k3s-oidc context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and that default has already changed once — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two dev.sh invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup:
longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to
Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io,
settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.
Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for
agents here: every durable change goes through git. main is a protected branch — a direct push
is rejected. A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
First commands
Which object, and the message (the message is nearly always the whole answer):
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get imagerepositories.image.toolkit.fluxcd.io -A \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,IMAGE:.spec.image,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,TAGS:.status.lastScanResult.tagCount; echo; \
kubectl --context k3s-prod get imagepolicies.image.toolkit.fluxcd.io -A \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,LATEST:.status.latestImage,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message'
Are the controllers installed at all?
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n flux-system get deploy | grep -i image || echo "image controllers NOT installed (expected on this cluster)"'
The reflector's log:
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n flux-system logs deploy/image-reflector-controller --tail=80 2>&1 | tail -30'
Can the cluster reach the registry at all? (Default-deny is the baseline; a new namespace may simply not be allowed out.)
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get pods -A --field-selector status.phase!=Running | head; echo; \
kubectl --context k3s-prod get events -A --field-selector reason=Failed --sort-by=.lastTimestamp | grep -i "pull\|registry" | tail -10'
Likely causes, most likely first
1. The image controllers are not installed (expected state here)
The CRs exist because catalog/render.py emits them; nothing reconciles them. Remediation: render with
--no-image-automation, or have the operator finish enabling automation. Note that the Flux
controllers were installed imperatively and are outside Flux's own inventory, so no git change can
install them.
2. Registry rate limit
Docker Hub anonymous pulls/scans are limited per IP, and all three nodes share one WAN address. The
message says toomanyrequests. Remediation: a registry credential (SOPS-encrypted, in git), or scan
less often via spec.interval.
3. Registry auth
A private repository with no secretRef, or an expired token. The message names it.
4. The policy's range matches nothing
ImagePolicy semver ranges come from each ServiceDefinition's update.range (for example
">=2024.0.0" for pihole, ">=24.0.0" for keycloak). A range that no published tag satisfies — or a
repository that publishes only latest (this platform has lscr.io/linuxserver/jackett:latest and
plexinc/pms-docker:latest) — leaves the policy with nothing to select. Remediation: fix the range in
catalog/<service>.yaml and re-render; a latest-only image cannot participate in automation at all
and should not carry a policy.
5. Egress is blocked for that namespace
Default-deny everywhere (R-NET-01) means outbound registry access is an explicit allow. Cross-read
/runbooks/denied-flows, and in particular the post-Cilium-upgrade finding that toEntities: world no
longer covers a call whose destination resolves to an in-cluster backend.
When to escalate or stop
- Warning, 1h, single service: this does not page. The consequence is one workload pinned at its current version.
- Do not widen an
update.rangeto make the alert clear. The range is the service's update policy (R-UPD-01); loosening it to satisfy a scanner is how an unintended major version arrives. - Do not add registry credentials ad hoc — new secret, DR path, operator's call.
- If the same object has been stalled for days, the honest outcome is to remove its automation CRs
(
--no-image-automation) rather than leave a permanently-failing object training everyone to ignore the alert. Alert fatigue is a recorded, expensive failure mode on this cluster.
Source of truth: runbooks/image-policy-stalled.md. Index: /runbooks/
Records: DECISIONS-LOG.md (image automation authored-and-disabled; --no-image-automation);
catalog/*.yaml (update.channel / update.range per service).