home-network-k3s-impl/decisions/2026-11-08-keycloak-operator-crds.md

date, branch, title

date	branch	title
2026-11-08	fix/keycloak-operator-crds	Keycloak 26.8.2 operator CrashLoop: install its two new CRDs, raise its memory

What changed

Incident. PR #121 (Renovate) bumped platform/keycloak/kustomization.yaml's three keycloak-k8s-resources URLs 26.0.6 -> 26.8.2. Upstream kubernetes/ at 26.8.2 holds four CRDs, not two (GitHub contents API, ?ref=26.8.2):

file	at 26.0.6	at 26.8.2
keycloaks.k8s.keycloak.org-v1.yml	yes	yes
keycloakrealmimports.k8s.keycloak.org-v1.yml	yes	yes
keycloakoidcclients.k8s.keycloak.org-v1.yml	—	new (v2alpha1)
keycloaksamlclients.k8s.keycloak.org-v1.yml	—	new (v2alpha1)
kubernetes.yml	yes	yes (now carries RBAC for both new kinds)

(cluster-wide/ is a separate variant; upstream's own kubernetes/kustomization.yml lists exactly the five files above.) The 26.8.2 operator starts an informer for every controller it has and dies on the first missing kind: Couldn't start informer for keycloaksamlclients.k8s.keycloak.org/v2alpha1 ... GET .../namespaces/keycloak/keycloaksamlclients ... Not Found, exit 1, CrashLoopBackOff. Flux platform (wait: true, 5m) sat in health checks and every dependent Kustomization reported dependency ... is not ready. The old 26.0.6 ReplicaSet kept serving (RollingUpdate 25%/25%), so SSO stayed up.

Fix.

• platform/keycloak/kustomization.yaml: added the two CRD URLs, each with the same # renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources marker (the repo's custom.regex manager matches all five — checked by running its regex over the file). Header comment updated to v26.8.2, plus a warning that Renovate cannot add a CRD a release introduces.
• Same file: a patches: JSON6902 patch on Deployment keycloak-operator: memory limit 450Mi -> 768Mi, request 450Mi -> 512Mi. 450Mi is upstream's default (unchanged at 26.8.2, so not something the bump changed). Evidence it is too small here: the 26.0.6 pod had 154 restarts, lastState.terminated.reason: OOMKilled (exit 137, 2026-11-07T23:09:54Z), and kubectl top showed 421Mi / 450Mi. 26.8.2 runs four controllers instead of two on Quarkus 3.40.1.1 (was 3.15.1). No variable in the operator to disable watching unused kinds was used — installing the CRDs is upstream's supported shape.
• platform/keycloak/README.md "Upgrading Keycloak": the diff-the-CRD-list step; "three URLs" -> "the release URLs". renovate.json5: same wording fix in a comment only.

What I verified

• Upstream file lists at 26.0.6 and 26.8.2 (GitHub API) and diff of kubernetes.yml between the two: the new kinds' ClusterRoles + RoleBindings ship in it, and the Deployment's resources are still 450Mi/450Mi.
• Live, read-only (kubectl --context k3s-prod, 2026-11-08 ~05:57Z): only keycloaks and keycloakrealmimports CRDs installed; Deployment strategy RollingUpdate maxSurge/maxUnavailable 25%; RS keycloak-operator-58dff6c4 (26.8.2) 0/1 CrashLoopBackOff, RS 78dbb75f64 (26.0.6) 1/1; operator log is the informer error above; Flux platform Healthy=Unknown "Running health checks for revision main@sha1:e8d2666"; no NetworkPolicy and no CiliumNetworkPolicy in ns keycloak.
• No sensitive path touched (infra/gitea/approval-policy.json), so this should merge on green.
• Gate: see the PR body for the exact rc and stages of the run against the final commit.

What I did NOT verify

• That the 26.8.2 operator becomes Ready after this merges. Only the two missing kinds were observed failing; another start-up failure behind them is possible.
• That 768Mi is enough — no 26.8.2 memory measurement exists yet; re-check kubectl top after a week.
• The Keycloak server upgrade 26.0.6 -> 26.8.2 itself. Once the operator runs it will roll keycloak-0/1 and Keycloak migrates the DB schema. The CR's spec.update.strategy defaults to RecreateOnImageChange — expect a full SSO outage while both pods recreate, not a rolling one. Back up / confirm the latest CNPG backup first if that matters.
• The 26.8.2 Keycloak CRD's spec.networkPolicy.enabled defaults to true. The operator will create a NetworkPolicy selecting the Keycloak pods (first policy in that namespace). With no from lists set it allows all sources on the HTTP/management ports, so Gateway, ServiceMonitor and the reconciler Jobs should still reach it — but this was read from the CRD text, not observed.
• The passkey reconciler's 26.0.6 Admin API assumptions against 26.8.2 (README "Upgrading Keycloak" list) — not re-checked here; PR #121 merged without it.
• catalog/keycloak.yaml's documentation-only image tag was not updated.