jakman85andClaude Opus 5
b7341b329b
Three corrections after rebasing onto main. 1. THE GUARD CAUGHT A REAL ONE ON ITS FIRST CONTACT WITH main. main had gained `CNPGPrimariesCoLocated` (platform/cnpg/alerts.yaml) pointing at /runbooks/cnpg-primary-spread, with no runbook. The check failed the rebase exactly as designed — 46 annotations, 35 names, 34 present — naming both the missing file AND the missing configMapGenerator entry. runbooks/cnpg-primary-spread.md now exists: what co-located primaries actually cost here (blast radius, not an outage, because instances: 3 + required anti-affinity), why the drift is uncorrectable-by-declaration (CNPG has no primary-placement field), the readiness gate that keeps the rule quiet during a real node outage, and the ~9s-per-database switchover rebalance, one cluster at a time. The 2026-09-07 outage in which all three primaries sat on one node is the reason the alert exists at all. 2. WHOLE-REPO SCOPE, NOT A HAND-WRITTEN STAGE. main's refactor replaced ci/l0.sh's per-check "UNCONDITIONAL" stages with a declaration: `conformance.home.k3s/scope: "whole-repo"`, unioned into every gate run by Stage 3b's `--include-scope`. My +38-line Stage 3f is removed (ci/l0.sh is now byte-identical to main) and the annotation is declared instead. Assertion (7) of the check was rewritten to match: it no longer asserts its own path appears in ci/l0.sh — it asserts the test DECLARES the scope and that ci/l0.sh still passes --include-scope, since neither half alone makes the check load-bearing. That matters here because ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02", not R-OBS-04, and maps platform/kubevirt/* and runbooks/* to nothing at all. 3. A FACTUAL CORRECTION IN ALL 35 RUNBOOKS. Every page claimed `dev.sh` defaults its current-context to k3s-dev. main's `9b4744d fix(devcontainer): dev.sh defaults to prod` made that false, and a runbook that misstates which cluster a command talks to is worse than one that says nothing. The pages now say what is durably true: the default has already changed once, so naming `--context k3s-prod` is what makes a command mean the same thing next month, and the per-invocation banner is there to be read. The check's own failure message was corrected the same way. Re-verified after all three: check green at 374 assertions (46 annotations, 35/35 runbooks present, 35/35 projected), re-applied to prod, and re-probed in-cluster through the Gateway — node-down, keycloak-down, keycloak-client-secrets, cnpg-archiving, longhorn-faulted, vpn-down, node-io and cnpg-primary-spread all HTTP 200 with text/plain; charset=utf-8 and 8.9-12.1 KB of real body; /runbooks/ index 200; an unknown name 404s with a pointer rather than a blank page; and homepage still answers the apex `/`, confirming Gateway API longest-path precedence. Full merge gate: PASS (spec-immutability, lint, l0, render-golden, l1 all green; prom-rules SKIPPED for want of promtool in the image; L2/L3 SKIPPED, no cluster). Co-Authored-By: Claude Opus 5 <[email protected]>
8.1 KiB
image-policy-stalled — An ImagePolicy or ImageRepository cannot resolve its tag
Alert: ImagePolicyStalled (warning, 1h)
File: platform/monitoring/alerts.yaml, group platform-health.automated-updates
Expression: max by (namespace, name, kind) (gotk_reconcile_condition{kind=~"ImagePolicy|ImageRepository", type="Ready", status="False"}) == 1
What this means here
An ImageRepository cannot scan a registry, or an ImagePolicy cannot pick a tag out of what was
scanned. The effect is narrower than /runbooks/image-automation: automation itself is healthy, but
this one service stops receiving updates (R-UPD-01), while everything else continues. It is
therefore easy to miss, which is what the 1h for and the per-object labels are for.
The same framing applies as for image-automation: this feature is authored and deliberately disabled
on this cluster and has been since 2026-08-24 (no Gitea write credential, and the
image-reflector-controller / image-automation-controller components are not installed — the line is
still commented out in platform/kustomization.yaml). Rendered apps emit ImageRepository and
ImagePolicy CRs anyway, so the objects can exist with no controller to reconcile them;
catalog/render.py --no-image-automation is the documented workaround. There is no incident history
for this alert — the section below is mechanism, honestly labelled.
One live constraint worth knowing while reading registry errors: the images in this platform come from
docker.io, ghcr.io, lscr.io, quay.io, registry.k8s.io and registry.gitlab.com. Docker
Hub's anonymous rate limit is the most likely real-world cause of a scan failure once automation is
switched on.
This alert is currently INERT. count(gotk_reconcile_condition) returned 0 series on prod on
2026-09-13 — there is no Flux-controller scrape job among Prometheus's 19 jobs, so every gotk_* rule
evaluates against nothing. Read the CRs directly with the commands below, and see
/runbooks/targets-down for the full list of inert rule groups.
Running the commands in this runbook
kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's k3s-oidc context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and that default has already changed once — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two dev.sh invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup:
longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to
Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io,
settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.
Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for
agents here: every durable change goes through git (git push origin main is the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
First commands
Which object, and the message (the message is nearly always the whole answer):
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get imagerepositories.image.toolkit.fluxcd.io -A \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,IMAGE:.spec.image,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,TAGS:.status.lastScanResult.tagCount; echo; \
kubectl --context k3s-prod get imagepolicies.image.toolkit.fluxcd.io -A \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,LATEST:.status.latestImage,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message'
Are the controllers installed at all?
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n flux-system get deploy | grep -i image || echo "image controllers NOT installed (expected on this cluster)"'
The reflector's log:
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n flux-system logs deploy/image-reflector-controller --tail=80 2>&1 | tail -30'
Can the cluster reach the registry at all? (Default-deny is the baseline; a new namespace may simply not be allowed out.)
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get pods -A --field-selector status.phase!=Running | head; echo; \
kubectl --context k3s-prod get events -A --field-selector reason=Failed --sort-by=.lastTimestamp | grep -i "pull\|registry" | tail -10'
Likely causes, most likely first
1. The image controllers are not installed (expected state here)
The CRs exist because catalog/render.py emits them; nothing reconciles them. Remediation: render with
--no-image-automation, or have the operator finish enabling automation. Note that the Flux
controllers were installed imperatively and are outside Flux's own inventory, so no git change can
install them.
2. Registry rate limit
Docker Hub anonymous pulls/scans are limited per IP, and all three nodes share one WAN address. The
message says toomanyrequests. Remediation: a registry credential (SOPS-encrypted, in git), or scan
less often via spec.interval.
3. Registry auth
A private repository with no secretRef, or an expired token. The message names it.
4. The policy's range matches nothing
ImagePolicy semver ranges come from each ServiceDefinition's update.range (for example
">=2024.0.0" for pihole, ">=24.0.0" for keycloak). A range that no published tag satisfies — or a
repository that publishes only latest (this platform has lscr.io/linuxserver/jackett:latest and
plexinc/pms-docker:latest) — leaves the policy with nothing to select. Remediation: fix the range in
catalog/<service>.yaml and re-render; a latest-only image cannot participate in automation at all
and should not carry a policy.
5. Egress is blocked for that namespace
Default-deny everywhere (R-NET-01) means outbound registry access is an explicit allow. Cross-read
/runbooks/denied-flows, and in particular the post-Cilium-upgrade finding that toEntities: world no
longer covers a call whose destination resolves to an in-cluster backend.
When to escalate or stop
- Warning, 1h, single service: this does not page. The consequence is one workload pinned at its current version.
- Do not widen an
update.rangeto make the alert clear. The range is the service's update policy (R-UPD-01); loosening it to satisfy a scanner is how an unintended major version arrives. - Do not add registry credentials ad hoc — new secret, DR path, operator's call.
- If the same object has been stalled for days, the honest outcome is to remove its automation CRs
(
--no-image-automation) rather than leave a permanently-failing object training everyone to ignore the alert. Alert fatigue is a recorded, expensive failure mode on this cluster.
Source of truth: runbooks/image-policy-stalled.md. Index: /runbooks/
Records: DECISIONS-LOG.md (image automation authored-and-disabled; --no-image-automation);
catalog/*.yaml (update.channel / update.range per service).