jakman85andClaude Opus 5.5
ea30ba876c
PR #121 bumped the three keycloak-k8s-resources URLs to 26.8.2, but 26.8.2 ships two more CRDs (keycloakoidcclients, keycloaksamlclients). The new operator fails to start its keycloaksamlclients informer (Not Found) and CrashLoops, holding Flux `platform` in health checks. - add both CRD URLs with the same renovate marker/depName - patch operator memory 450Mi -> limit 768Mi / request 512Mi (26.0.6 was OOMKilled at 421/450Mi, 154 restarts) - README/renovate comments: Renovate cannot add CRDs a release introduces Co-Authored-By: Claude Opus 5.5 <[email protected]>
75 lines
4.7 KiB
Markdown
75 lines
4.7 KiB
Markdown
---
|
|
date: 2026-11-08
|
|
branch: fix/keycloak-operator-crds
|
|
title: "Keycloak 26.8.2 operator CrashLoop: install its two new CRDs, raise its memory"
|
|
---
|
|
|
|
## What changed
|
|
|
|
**Incident.** PR #121 (Renovate) bumped `platform/keycloak/kustomization.yaml`'s three
|
|
`keycloak-k8s-resources` URLs 26.0.6 -> 26.8.2. Upstream `kubernetes/` at 26.8.2 holds **four**
|
|
CRDs, not two (GitHub contents API, `?ref=26.8.2`):
|
|
|
|
| file | at 26.0.6 | at 26.8.2 |
|
|
|---|---|---|
|
|
| `keycloaks.k8s.keycloak.org-v1.yml` | yes | yes |
|
|
| `keycloakrealmimports.k8s.keycloak.org-v1.yml` | yes | yes |
|
|
| `keycloakoidcclients.k8s.keycloak.org-v1.yml` | — | **new** (v2alpha1) |
|
|
| `keycloaksamlclients.k8s.keycloak.org-v1.yml` | — | **new** (v2alpha1) |
|
|
| `kubernetes.yml` | yes | yes (now carries RBAC for both new kinds) |
|
|
|
|
(`cluster-wide/` is a separate variant; upstream's own `kubernetes/kustomization.yml` lists exactly
|
|
the five files above.) The 26.8.2 operator starts an informer for every controller it has and dies
|
|
on the first missing kind: `Couldn't start informer for keycloaksamlclients.k8s.keycloak.org/v2alpha1
|
|
... GET .../namespaces/keycloak/keycloaksamlclients ... Not Found`, exit 1, CrashLoopBackOff. Flux
|
|
`platform` (`wait: true`, 5m) sat in health checks and every dependent Kustomization reported
|
|
`dependency ... is not ready`. The old 26.0.6 ReplicaSet kept serving (RollingUpdate 25%/25%), so SSO
|
|
stayed up.
|
|
|
|
**Fix.**
|
|
- `platform/keycloak/kustomization.yaml`: added the two CRD URLs, each with the same
|
|
`# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources` marker (the repo's
|
|
custom.regex manager matches all five — checked by running its regex over the file). Header
|
|
comment updated to v26.8.2, plus a warning that Renovate cannot add a CRD a release introduces.
|
|
- Same file: a `patches:` JSON6902 patch on Deployment `keycloak-operator`: memory limit
|
|
450Mi -> **768Mi**, request 450Mi -> **512Mi**. 450Mi is upstream's default (unchanged at 26.8.2,
|
|
so not something the bump changed). Evidence it is too small here: the 26.0.6 pod had **154
|
|
restarts**, `lastState.terminated.reason: OOMKilled` (exit 137, 2026-11-07T23:09:54Z), and
|
|
`kubectl top` showed **421Mi / 450Mi**. 26.8.2 runs four controllers instead of two on Quarkus
|
|
3.40.1.1 (was 3.15.1). No variable in the operator to disable watching unused kinds was used —
|
|
installing the CRDs is upstream's supported shape.
|
|
- `platform/keycloak/README.md` "Upgrading Keycloak": the diff-the-CRD-list step; "three URLs" ->
|
|
"the release URLs". `renovate.json5`: same wording fix in a comment only.
|
|
|
|
## What I verified
|
|
|
|
- Upstream file lists at 26.0.6 and 26.8.2 (GitHub API) and `diff` of `kubernetes.yml` between the
|
|
two: the new kinds' ClusterRoles + RoleBindings ship in it, and the Deployment's resources are
|
|
still 450Mi/450Mi.
|
|
- Live, read-only (`kubectl --context k3s-prod`, 2026-11-08 ~05:57Z): only `keycloaks` and
|
|
`keycloakrealmimports` CRDs installed; Deployment strategy RollingUpdate maxSurge/maxUnavailable
|
|
25%; RS `keycloak-operator-58dff6c4` (26.8.2) 0/1 CrashLoopBackOff, RS `78dbb75f64` (26.0.6) 1/1;
|
|
operator log is the informer error above; Flux `platform` `Healthy=Unknown` "Running health
|
|
checks for revision main@sha1:e8d2666"; no NetworkPolicy and no CiliumNetworkPolicy in ns
|
|
`keycloak`.
|
|
- No sensitive path touched (`infra/gitea/approval-policy.json`), so this should merge on green.
|
|
- Gate: see the PR body for the exact rc and stages of the run against the final commit.
|
|
|
|
## What I did NOT verify
|
|
|
|
- **That the 26.8.2 operator becomes Ready after this merges.** Only the two missing kinds were
|
|
observed failing; another start-up failure behind them is possible.
|
|
- **That 768Mi is enough** — no 26.8.2 memory measurement exists yet; re-check `kubectl top` after
|
|
a week.
|
|
- **The Keycloak server upgrade 26.0.6 -> 26.8.2 itself.** Once the operator runs it will roll
|
|
keycloak-0/1 and Keycloak migrates the DB schema. The CR's `spec.update.strategy` defaults to
|
|
`RecreateOnImageChange` — expect a **full SSO outage** while both pods recreate, not a rolling
|
|
one. Back up / confirm the latest CNPG backup first if that matters.
|
|
- **The 26.8.2 Keycloak CRD's `spec.networkPolicy.enabled` defaults to `true`.** The operator will
|
|
create a NetworkPolicy selecting the Keycloak pods (first policy in that namespace). With no
|
|
`from` lists set it allows all sources on the HTTP/management ports, so Gateway, ServiceMonitor
|
|
and the reconciler Jobs should still reach it — but this was read from the CRD text, not observed.
|
|
- **The passkey reconciler's 26.0.6 Admin API assumptions against 26.8.2** (README "Upgrading
|
|
Keycloak" list) — not re-checked here; PR #121 merged without it.
|
|
- `catalog/keycloak.yaml`'s documentation-only image tag was not updated.
|