From ea30ba876ccfabd53c5f038a04ca4710e9db4bcd Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 8 Nov 2026 00:00:01 -0600 Subject: [PATCH] fix(keycloak): install the 26.8.2 operator's new CRDs (operator CrashLoop) PR #121 bumped the three keycloak-k8s-resources URLs to 26.8.2, but 26.8.2 ships two more CRDs (keycloakoidcclients, keycloaksamlclients). The new operator fails to start its keycloaksamlclients informer (Not Found) and CrashLoops, holding Flux `platform` in health checks. - add both CRD URLs with the same renovate marker/depName - patch operator memory 450Mi -> limit 768Mi / request 512Mi (26.0.6 was OOMKilled at 421/450Mi, 154 restarts) - README/renovate comments: Renovate cannot add CRDs a release introduces Co-Authored-By: Claude Opus 5.5 --- .../2026-11-08-keycloak-operator-crds.md | 74 +++++++++++++++++++ platform/keycloak/README.md | 10 ++- platform/keycloak/kustomization.yaml | 40 +++++++++- renovate.json5 | 2 +- 4 files changed, 121 insertions(+), 5 deletions(-) create mode 100644 decisions/2026-11-08-keycloak-operator-crds.md diff --git a/decisions/2026-11-08-keycloak-operator-crds.md b/decisions/2026-11-08-keycloak-operator-crds.md new file mode 100644 index 0000000..66c50c8 --- /dev/null +++ b/decisions/2026-11-08-keycloak-operator-crds.md @@ -0,0 +1,74 @@ +--- +date: 2026-11-08 +branch: fix/keycloak-operator-crds +title: "Keycloak 26.8.2 operator CrashLoop: install its two new CRDs, raise its memory" +--- + +## What changed + +**Incident.** PR #121 (Renovate) bumped `platform/keycloak/kustomization.yaml`'s three +`keycloak-k8s-resources` URLs 26.0.6 -> 26.8.2. Upstream `kubernetes/` at 26.8.2 holds **four** +CRDs, not two (GitHub contents API, `?ref=26.8.2`): + +| file | at 26.0.6 | at 26.8.2 | +|---|---|---| +| `keycloaks.k8s.keycloak.org-v1.yml` | yes | yes | +| `keycloakrealmimports.k8s.keycloak.org-v1.yml` | yes | yes | +| `keycloakoidcclients.k8s.keycloak.org-v1.yml` | — | **new** (v2alpha1) | +| `keycloaksamlclients.k8s.keycloak.org-v1.yml` | — | **new** (v2alpha1) | +| `kubernetes.yml` | yes | yes (now carries RBAC for both new kinds) | + +(`cluster-wide/` is a separate variant; upstream's own `kubernetes/kustomization.yml` lists exactly +the five files above.) The 26.8.2 operator starts an informer for every controller it has and dies +on the first missing kind: `Couldn't start informer for keycloaksamlclients.k8s.keycloak.org/v2alpha1 +... GET .../namespaces/keycloak/keycloaksamlclients ... Not Found`, exit 1, CrashLoopBackOff. Flux +`platform` (`wait: true`, 5m) sat in health checks and every dependent Kustomization reported +`dependency ... is not ready`. The old 26.0.6 ReplicaSet kept serving (RollingUpdate 25%/25%), so SSO +stayed up. + +**Fix.** +- `platform/keycloak/kustomization.yaml`: added the two CRD URLs, each with the same + `# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources` marker (the repo's + custom.regex manager matches all five — checked by running its regex over the file). Header + comment updated to v26.8.2, plus a warning that Renovate cannot add a CRD a release introduces. +- Same file: a `patches:` JSON6902 patch on Deployment `keycloak-operator`: memory limit + 450Mi -> **768Mi**, request 450Mi -> **512Mi**. 450Mi is upstream's default (unchanged at 26.8.2, + so not something the bump changed). Evidence it is too small here: the 26.0.6 pod had **154 + restarts**, `lastState.terminated.reason: OOMKilled` (exit 137, 2026-11-07T23:09:54Z), and + `kubectl top` showed **421Mi / 450Mi**. 26.8.2 runs four controllers instead of two on Quarkus + 3.40.1.1 (was 3.15.1). No variable in the operator to disable watching unused kinds was used — + installing the CRDs is upstream's supported shape. +- `platform/keycloak/README.md` "Upgrading Keycloak": the diff-the-CRD-list step; "three URLs" -> + "the release URLs". `renovate.json5`: same wording fix in a comment only. + +## What I verified + +- Upstream file lists at 26.0.6 and 26.8.2 (GitHub API) and `diff` of `kubernetes.yml` between the + two: the new kinds' ClusterRoles + RoleBindings ship in it, and the Deployment's resources are + still 450Mi/450Mi. +- Live, read-only (`kubectl --context k3s-prod`, 2026-11-08 ~05:57Z): only `keycloaks` and + `keycloakrealmimports` CRDs installed; Deployment strategy RollingUpdate maxSurge/maxUnavailable + 25%; RS `keycloak-operator-58dff6c4` (26.8.2) 0/1 CrashLoopBackOff, RS `78dbb75f64` (26.0.6) 1/1; + operator log is the informer error above; Flux `platform` `Healthy=Unknown` "Running health + checks for revision main@sha1:e8d2666"; no NetworkPolicy and no CiliumNetworkPolicy in ns + `keycloak`. +- No sensitive path touched (`infra/gitea/approval-policy.json`), so this should merge on green. +- Gate: see the PR body for the exact rc and stages of the run against the final commit. + +## What I did NOT verify + +- **That the 26.8.2 operator becomes Ready after this merges.** Only the two missing kinds were + observed failing; another start-up failure behind them is possible. +- **That 768Mi is enough** — no 26.8.2 memory measurement exists yet; re-check `kubectl top` after + a week. +- **The Keycloak server upgrade 26.0.6 -> 26.8.2 itself.** Once the operator runs it will roll + keycloak-0/1 and Keycloak migrates the DB schema. The CR's `spec.update.strategy` defaults to + `RecreateOnImageChange` — expect a **full SSO outage** while both pods recreate, not a rolling + one. Back up / confirm the latest CNPG backup first if that matters. +- **The 26.8.2 Keycloak CRD's `spec.networkPolicy.enabled` defaults to `true`.** The operator will + create a NetworkPolicy selecting the Keycloak pods (first policy in that namespace). With no + `from` lists set it allows all sources on the HTTP/management ports, so Gateway, ServiceMonitor + and the reconciler Jobs should still reach it — but this was read from the CRD text, not observed. +- **The passkey reconciler's 26.0.6 Admin API assumptions against 26.8.2** (README "Upgrading + Keycloak" list) — not re-checked here; PR #121 merged without it. +- `catalog/keycloak.yaml`'s documentation-only image tag was not updated. diff --git a/platform/keycloak/README.md b/platform/keycloak/README.md index 032fa98..a6769f3 100644 --- a/platform/keycloak/README.md +++ b/platform/keycloak/README.md @@ -367,9 +367,17 @@ is involved — so step 2 failing with no log entry is the expected signature of The `Keycloak` CR has no `spec.image`, so the server image is whatever the operator's `RELATED_IMAGE_KEYCLOAK` names — the same tag as the operator (prod, 2026-10-30: operator `26.0.6`, `keycloak-0/1` on `quay.io/keycloak/keycloak:26.0.6`). Operator, CRDs and server are -released in lockstep from `keycloak/keycloak-k8s-resources`, and the three URLs in +released in lockstep from `keycloak/keycloak-k8s-resources`, and the release URLs in `kustomization.yaml` are the one knob. Renovate raises them as ONE PR (`keycloak-operator` group, labels `needs-review` + `blast-radius`), **never auto-merged**; majors wait for a dashboard tick. + +**Renovate bumps the URLs that exist; it cannot add a CRD a release introduced.** Before merging a +bump, diff `kubernetes/` (and its `kustomization.yml`) at the new tag against the URL list. 26.0.6 +shipped 2 CRDs, 26.8.2 ships 4 (`keycloakoidcclients`, `keycloaksamlclients` added); PR #121 bumped +only the existing URLs and the 26.8.2 operator CrashLooped on the missing `keycloaksamlclients` +informer, stalling Flux `platform` and every tier below it (2026-11-08, +`decisions/2026-11-08-keycloak-operator-crds.md`). The operator's memory is also patched in +`kustomization.yaml` (upstream's 450Mi OOMKilled 26.0.6 here); keep the patch across bumps. `catalog/keycloak.yaml`'s image tag is documentation only (its render deploys nothing) and Renovate is disabled for it — update it by hand to match after an operator bump merges. diff --git a/platform/keycloak/kustomization.yaml b/platform/keycloak/kustomization.yaml index 9c65401..0be4188 100644 --- a/platform/keycloak/kustomization.yaml +++ b/platform/keycloak/kustomization.yaml @@ -20,18 +20,32 @@ kind: Kustomization namespace: keycloak resources: - namespace.yaml - # --- Keycloak Operator v26.0.6 (pinned upstream release manifests) --- + # --- Keycloak Operator v26.8.2 (pinned upstream release manifests) --- # The operator, its CRDs and the SERVER IMAGE move in lockstep: the Keycloak CR has no # spec.image, so the operator's RELATED_IMAGE_KEYCLOAK (quay.io/keycloak/keycloak:) - # is what runs. These three URLs are therefore THE Keycloak version. Each carries a Renovate - # marker (custom.regex manager, renovate.json5) under ONE depName, so all three bump in ONE + # is what runs. These URLs are therefore THE Keycloak version. Each carries a Renovate + # marker (custom.regex manager, renovate.json5) under ONE depName, so all of them bump in ONE # branch, never auto-merged (packageRule "keycloak-operator"). Before merging a bump, re-check # passkey-reconciler.yaml's 26.0.6 Admin API assumptions — see README.md "Upgrading Keycloak". + # + # THE URL LIST MUST MATCH UPSTREAM'S kubernetes/kustomization.yml AT THE NEW TAG — Renovate only + # rewrites the version in the URLs that are here; it cannot add a CRD the new release introduced. + # 26.0.6 shipped 2 CRDs; 26.8.2 ships 4 (+ keycloakoidcclients, keycloaksamlclients, both + # v2alpha1). PR #121 bumped only the existing three URLs, and the 26.8.2 operator — which starts an + # informer for EVERY kind it controls, used or not — CrashLooped on + # `Couldn't start informer for keycloaksamlclients.k8s.keycloak.org/v2alpha1 ... Not Found`, + # stalling Flux `platform` and everything downstream (2026-11-08). On every bump, diff + # https://github.com/keycloak/keycloak-k8s-resources/tree//kubernetes + # against this list before merging. # renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/keycloaks.k8s.keycloak.org-v1.yml # renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/keycloakrealmimports.k8s.keycloak.org-v1.yml # renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources + - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/keycloakoidcclients.k8s.keycloak.org-v1.yml + # renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources + - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/keycloaksamlclients.k8s.keycloak.org-v1.yml + # renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/kubernetes.yml # --- Admin (break-glass) credential — SOPS-encrypted, referenced by name (ADR-0007, R-IDP-04) --- - keycloak-admin-credentials.sops.yaml @@ -51,3 +65,23 @@ resources: - keycloak-db-backup-credentials-prod.sops.yaml # --- Grafana dashboard for Keycloak health (plain ConfigMap; R-OBS-03) --- - grafana-dashboard.yaml +# --- Operator memory: upstream ships requests=limits=450Mi, which this cluster outgrew ON 26.0.6 --- +# Measured on prod 2026-11-08: the 26.0.6 operator pod had 154 restarts in 64d, lastState +# `OOMKilled` (exit 137, 2026-11-07T23:09:54Z), and sat at 421Mi of its 450Mi (94%) in `kubectl top`. +# 26.8.2 keeps the same 450Mi default but runs FOUR controllers/informers instead of two (adds +# keycloakoidcclient + keycloaksamlclient) on a newer Quarkus (3.15.1 -> 3.40.1.1), so it would +# plausibly start above a ceiling 26.0.6 already crossed. Limit 768Mi = ~1.8x the observed 421Mi +# working set; request 512Mi = the observed working set plus headroom, so the scheduler reserves what +# it actually uses. Revisit with `kubectl top` once 26.8.2 has run for a week. Patched here (not in +# the remote kubernetes.yml) so a Renovate bump keeps it. +patches: + - target: + kind: Deployment + name: keycloak-operator + patch: |- + - op: replace + path: /spec/template/spec/containers/0/resources/limits/memory + value: 768Mi + - op: replace + path: /spec/template/spec/containers/0/resources/requests/memory + value: 512Mi diff --git a/renovate.json5 b/renovate.json5 index 21214e4..2080217 100644 --- a/renovate.json5 +++ b/renovate.json5 @@ -264,7 +264,7 @@ // and the server image are released in lockstep from keycloak/keycloak-k8s-resources, and the // Keycloak CR carries no spec.image, so the operator's RELATED_IMAGE_KEYCLOAK picks the server // (measured on prod 2026-10-30: operator 26.0.6 -> keycloak-0/1 run quay.io/keycloak/keycloak: - // 26.0.6). The three release URLs in platform/keycloak/kustomization.yaml carry `# renovate:` + // 26.0.6). The release URLs in platform/keycloak/kustomization.yaml carry `# renovate:` // markers under this ONE depName, so they move together in one branch (groupName makes that // explicit). NEVER AUTO-MERGED, not even a patch: Keycloak is the IdP that gates SSO for every // app AND kubectl OIDC, and platform/keycloak/passkey-reconciler.yaml encodes Keycloak 26.0.6