Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 3 Actions Packages Projects Releases Wiki Activity
Files
a79693d3bd2cb83e48bf6fa3ae1dae0d07dbc0e9
home-network-k3s-impl/runbooks/image-policy-stalled.md
T
jakman85andClaude Opus 5 b7341b329b fix(runbooks): declare whole-repo scope, drop the bespoke l0 stage, +cnpg-primary-spread 
Three corrections after rebasing onto main.

1. THE GUARD CAUGHT A REAL ONE ON ITS FIRST CONTACT WITH main. main had gained
   `CNPGPrimariesCoLocated` (platform/cnpg/alerts.yaml) pointing at
   /runbooks/cnpg-primary-spread, with no runbook. The check failed the rebase exactly as
   designed — 46 annotations, 35 names, 34 present — naming both the missing file AND the
   missing configMapGenerator entry. runbooks/cnpg-primary-spread.md now exists: what
   co-located primaries actually cost here (blast radius, not an outage, because
   instances: 3 + required anti-affinity), why the drift is uncorrectable-by-declaration
   (CNPG has no primary-placement field), the readiness gate that keeps the rule quiet
   during a real node outage, and the ~9s-per-database switchover rebalance, one cluster
   at a time. The 2026-09-07 outage in which all three primaries sat on one node is the
   reason the alert exists at all.

2. WHOLE-REPO SCOPE, NOT A HAND-WRITTEN STAGE. main's refactor replaced ci/l0.sh's
   per-check "UNCONDITIONAL" stages with a declaration:
   `conformance.home.k3s/scope: "whole-repo"`, unioned into every gate run by Stage 3b's
   `--include-scope`. My +38-line Stage 3f is removed (ci/l0.sh is now byte-identical to
   main) and the annotation is declared instead. Assertion (7) of the check was rewritten
   to match: it no longer asserts its own path appears in ci/l0.sh — it asserts the test
   DECLARES the scope and that ci/l0.sh still passes --include-scope, since neither half
   alone makes the check load-bearing. That matters here because
   ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02", not
   R-OBS-04, and maps platform/kubevirt/* and runbooks/* to nothing at all.

3. A FACTUAL CORRECTION IN ALL 35 RUNBOOKS. Every page claimed `dev.sh` defaults its
   current-context to k3s-dev. main's `9b4744d fix(devcontainer): dev.sh defaults to
   prod` made that false, and a runbook that misstates which cluster a command talks to is
   worse than one that says nothing. The pages now say what is durably true: the default
   has already changed once, so naming `--context k3s-prod` is what makes a command mean
   the same thing next month, and the per-invocation banner is there to be read. The
   check's own failure message was corrected the same way.

Re-verified after all three: check green at 374 assertions (46 annotations, 35/35 runbooks
present, 35/35 projected), re-applied to prod, and re-probed in-cluster through the
Gateway — node-down, keycloak-down, keycloak-client-secrets, cnpg-archiving,
longhorn-faulted, vpn-down, node-io and cnpg-primary-spread all HTTP 200 with
text/plain; charset=utf-8 and 8.9-12.1 KB of real body; /runbooks/ index 200; an unknown
name 404s with a pointer rather than a blank page; and homepage still answers the apex `/`,
confirming Gateway API longest-path precedence.

Full merge gate: PASS (spec-immutability, lint, l0, render-golden, l1 all green;
prom-rules SKIPPED for want of promtool in the image; L2/L3 SKIPPED, no cluster).

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 00:00:24 -05:00

138 lines
8.1 KiB
Markdown
Raw Blame History

# image-policy-stalled — An ImagePolicy or ImageRepository cannot resolve its tag
**Alert:** `ImagePolicyStalled` (warning, 1h)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.automated-updates`
**Expression:** `max by (namespace, name, kind) (gotk_reconcile_condition{kind=~"ImagePolicy|ImageRepository", type="Ready", status="False"}) == 1`
## What this means here
An `ImageRepository` cannot scan a registry, or an `ImagePolicy` cannot pick a tag out of what was
scanned. The effect is narrower than `/runbooks/image-automation`: automation itself is healthy, but
**this one service stops receiving updates** (R-UPD-01), while everything else continues. It is
therefore easy to miss, which is what the 1h `for` and the per-object labels are for.
The same framing applies as for image-automation: **this feature is authored and deliberately disabled
on this cluster** and has been since 2026-08-24 (no Gitea write credential, and the
`image-reflector-controller` / `image-automation-controller` components are not installed — the line is
still commented out in `platform/kustomization.yaml`). Rendered apps emit `ImageRepository` and
`ImagePolicy` CRs anyway, so the objects can exist with no controller to reconcile them;
`catalog/render.py --no-image-automation` is the documented workaround. **There is no incident history
for this alert** — the section below is mechanism, honestly labelled.
One live constraint worth knowing while reading registry errors: the images in this platform come from
`docker.io`, `ghcr.io`, `lscr.io`, `quay.io`, `registry.k8s.io` and `registry.gitlab.com`. Docker
Hub's anonymous rate limit is the most likely real-world cause of a scan failure once automation is
switched on.
**This alert is currently INERT.** `count(gotk_reconcile_condition)` returned **0 series** on prod on
2026-09-13 — there is no Flux-controller scrape job among Prometheus's 19 jobs, so every `gotk_*` rule
evaluates against nothing. Read the CRs directly with the commands below, and see
`/runbooks/targets-down` for the full list of inert rule groups.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
## First commands
Which object, and the message (the message is nearly always the whole answer):
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get imagerepositories.image.toolkit.fluxcd.io -A \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,IMAGE:.spec.image,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,TAGS:.status.lastScanResult.tagCount; echo; \
kubectl --context k3s-prod get imagepolicies.image.toolkit.fluxcd.io -A \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,LATEST:.status.latestImage,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message'
```
Are the controllers installed at all?
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n flux-system get deploy | grep -i image || echo "image controllers NOT installed (expected on this cluster)"'
```
The reflector's log:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n flux-system logs deploy/image-reflector-controller --tail=80 2>&1 | tail -30'
```
Can the cluster reach the registry at all? (Default-deny is the baseline; a new namespace may simply
not be allowed out.)
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get pods -A --field-selector status.phase!=Running | head; echo; \
kubectl --context k3s-prod get events -A --field-selector reason=Failed --sort-by=.lastTimestamp | grep -i "pull\|registry" | tail -10'
```
## Likely causes, most likely first
### 1. The image controllers are not installed (expected state here)
The CRs exist because `catalog/render.py` emits them; nothing reconciles them. Remediation: render with
`--no-image-automation`, or have the operator finish enabling automation. Note that the Flux
controllers were installed **imperatively** and are outside Flux's own inventory, so no git change can
install them.
### 2. Registry rate limit
Docker Hub anonymous pulls/scans are limited per IP, and all three nodes share one WAN address. The
message says `toomanyrequests`. Remediation: a registry credential (SOPS-encrypted, in git), or scan
less often via `spec.interval`.
### 3. Registry auth
A private repository with no `secretRef`, or an expired token. The message names it.
### 4. The policy's range matches nothing
`ImagePolicy` semver ranges come from each ServiceDefinition's `update.range` (for example
`">=2024.0.0"` for pihole, `">=24.0.0"` for keycloak). A range that no published tag satisfies — or a
repository that publishes only `latest` (this platform has `lscr.io/linuxserver/jackett:latest` and
`plexinc/pms-docker:latest`) — leaves the policy with nothing to select. Remediation: fix the range in
`catalog/<service>.yaml` and re-render; a `latest`-only image cannot participate in automation at all
and should not carry a policy.
### 5. Egress is blocked for that namespace
Default-deny everywhere (R-NET-01) means outbound registry access is an explicit allow. Cross-read
`/runbooks/denied-flows`, and in particular the post-Cilium-upgrade finding that `toEntities: world` no
longer covers a call whose destination resolves to an in-cluster backend.
## When to escalate or stop
- **Warning, 1h, single service: this does not page.** The consequence is one workload pinned at its
current version.
- **Do not widen an `update.range` to make the alert clear.** The range is the service's update policy
(R-UPD-01); loosening it to satisfy a scanner is how an unintended major version arrives.
- **Do not add registry credentials ad hoc** — new secret, DR path, operator's call.
- If the same object has been stalled for days, the honest outcome is to remove its automation CRs
(`--no-image-automation`) rather than leave a permanently-failing object training everyone to ignore
the alert. Alert fatigue is a recorded, expensive failure mode on this cluster.
---
Source of truth: `runbooks/image-policy-stalled.md`. Index: `/runbooks/`
Records: `DECISIONS-LOG.md` (image automation authored-and-disabled; `--no-image-automation`);
`catalog/*.yaml` (`update.channel` / `update.range` per service).
Reference in new issue View Git Blame Copy Permalink