Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

fix(keycloak): install the 26.8.2 operator's new CRDs (operator CrashLoop) #144

Merged
jakman85 merged 1 commits from fix/keycloak-operator-crds into main 2026-11-08 06:16:56 +00:00
Conversation 0 Commits 1 Files Changed 4
+121 -5
4 changed files with 121 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

decisions/2026-11-08-keycloak-operator-crds.md
+74
View File
@@ -0,0 +1,74 @@
---
date: 2026-11-08
branch: fix/keycloak-operator-crds
title: "Keycloak 26.8.2 operator CrashLoop: install its two new CRDs, raise its memory"
---
## What changed
**Incident.** PR #121 (Renovate) bumped `platform/keycloak/kustomization.yaml`'s three
`keycloak-k8s-resources` URLs 26.0.6 -> 26.8.2. Upstream `kubernetes/` at 26.8.2 holds **four**
CRDs, not two (GitHub contents API, `?ref=26.8.2`):
| file | at 26.0.6 | at 26.8.2 |
|---|---|---|
| `keycloaks.k8s.keycloak.org-v1.yml` | yes | yes |
| `keycloakrealmimports.k8s.keycloak.org-v1.yml` | yes | yes |
| `keycloakoidcclients.k8s.keycloak.org-v1.yml` | — | **new** (v2alpha1) |
| `keycloaksamlclients.k8s.keycloak.org-v1.yml` | — | **new** (v2alpha1) |
| `kubernetes.yml` | yes | yes (now carries RBAC for both new kinds) |
(`cluster-wide/` is a separate variant; upstream's own `kubernetes/kustomization.yml` lists exactly
the five files above.) The 26.8.2 operator starts an informer for every controller it has and dies
on the first missing kind: `Couldn't start informer for keycloaksamlclients.k8s.keycloak.org/v2alpha1
... GET .../namespaces/keycloak/keycloaksamlclients ... Not Found`, exit 1, CrashLoopBackOff. Flux
`platform` (`wait: true`, 5m) sat in health checks and every dependent Kustomization reported
`dependency ... is not ready`. The old 26.0.6 ReplicaSet kept serving (RollingUpdate 25%/25%), so SSO
stayed up.
**Fix.**
- `platform/keycloak/kustomization.yaml`: added the two CRD URLs, each with the same
`# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources` marker (the repo's
custom.regex manager matches all five — checked by running its regex over the file). Header
comment updated to v26.8.2, plus a warning that Renovate cannot add a CRD a release introduces.
- Same file: a `patches:` JSON6902 patch on Deployment `keycloak-operator`: memory limit
450Mi -> **768Mi**, request 450Mi -> **512Mi**. 450Mi is upstream's default (unchanged at 26.8.2,
so not something the bump changed). Evidence it is too small here: the 26.0.6 pod had **154
restarts**, `lastState.terminated.reason: OOMKilled` (exit 137, 2026-11-07T23:09:54Z), and
`kubectl top` showed **421Mi / 450Mi**. 26.8.2 runs four controllers instead of two on Quarkus
3.40.1.1 (was 3.15.1). No variable in the operator to disable watching unused kinds was used —
installing the CRDs is upstream's supported shape.
- `platform/keycloak/README.md` "Upgrading Keycloak": the diff-the-CRD-list step; "three URLs" ->
"the release URLs". `renovate.json5`: same wording fix in a comment only.
## What I verified
- Upstream file lists at 26.0.6 and 26.8.2 (GitHub API) and `diff` of `kubernetes.yml` between the
two: the new kinds' ClusterRoles + RoleBindings ship in it, and the Deployment's resources are
still 450Mi/450Mi.
- Live, read-only (`kubectl --context k3s-prod`, 2026-11-08 ~05:57Z): only `keycloaks` and
`keycloakrealmimports` CRDs installed; Deployment strategy RollingUpdate maxSurge/maxUnavailable
25%; RS `keycloak-operator-58dff6c4` (26.8.2) 0/1 CrashLoopBackOff, RS `78dbb75f64` (26.0.6) 1/1;
operator log is the informer error above; Flux `platform` `Healthy=Unknown` "Running health
checks for revision main@sha1:e8d2666"; no NetworkPolicy and no CiliumNetworkPolicy in ns
`keycloak`.
- No sensitive path touched (`infra/gitea/approval-policy.json`), so this should merge on green.
- Gate: see the PR body for the exact rc and stages of the run against the final commit.
## What I did NOT verify
- **That the 26.8.2 operator becomes Ready after this merges.** Only the two missing kinds were
observed failing; another start-up failure behind them is possible.
- **That 768Mi is enough** — no 26.8.2 memory measurement exists yet; re-check `kubectl top` after
a week.
- **The Keycloak server upgrade 26.0.6 -> 26.8.2 itself.** Once the operator runs it will roll
keycloak-0/1 and Keycloak migrates the DB schema. The CR's `spec.update.strategy` defaults to
`RecreateOnImageChange` — expect a **full SSO outage** while both pods recreate, not a rolling
one. Back up / confirm the latest CNPG backup first if that matters.
- **The 26.8.2 Keycloak CRD's `spec.networkPolicy.enabled` defaults to `true`.** The operator will
create a NetworkPolicy selecting the Keycloak pods (first policy in that namespace). With no
`from` lists set it allows all sources on the HTTP/management ports, so Gateway, ServiceMonitor
and the reconciler Jobs should still reach it — but this was read from the CRD text, not observed.
- **The passkey reconciler's 26.0.6 Admin API assumptions against 26.8.2** (README "Upgrading
Keycloak" list) — not re-checked here; PR #121 merged without it.
- `catalog/keycloak.yaml`'s documentation-only image tag was not updated.
platform/keycloak/README.md
+9 -1
View File
@@ -367,9 +367,17 @@ is involved — so step 2 failing with no log entry is the expected signature of
The `Keycloak` CR has no `spec.image`, so the server image is whatever the operator's
`RELATED_IMAGE_KEYCLOAK` names — the same tag as the operator (prod, 2026-10-30: operator
`26.0.6`, `keycloak-0/1` on `quay.io/keycloak/keycloak:26.0.6`). Operator, CRDs and server are
released in lockstep from `keycloak/keycloak-k8s-resources`, and the three URLs in
released in lockstep from `keycloak/keycloak-k8s-resources`, and the release URLs in
`kustomization.yaml` are the one knob. Renovate raises them as ONE PR (`keycloak-operator` group,
labels `needs-review` + `blast-radius`), **never auto-merged**; majors wait for a dashboard tick.
**Renovate bumps the URLs that exist; it cannot add a CRD a release introduced.** Before merging a
bump, diff `kubernetes/` (and its `kustomization.yml`) at the new tag against the URL list. 26.0.6
shipped 2 CRDs, 26.8.2 ships 4 (`keycloakoidcclients`, `keycloaksamlclients` added); PR #121 bumped
only the existing URLs and the 26.8.2 operator CrashLooped on the missing `keycloaksamlclients`
informer, stalling Flux `platform` and every tier below it (2026-11-08,
`decisions/2026-11-08-keycloak-operator-crds.md`). The operator's memory is also patched in
`kustomization.yaml` (upstream's 450Mi OOMKilled 26.0.6 here); keep the patch across bumps.
`catalog/keycloak.yaml`'s image tag is documentation only (its render deploys nothing) and
Renovate is disabled for it — update it by hand to match after an operator bump merges.
platform/keycloak/kustomization.yaml
+37 -3
View File
@@ -20,18 +20,32 @@ kind: Kustomization
namespace: keycloak
resources:
- namespace.yaml
# --- Keycloak Operator v26.0.6 (pinned upstream release manifests) ---
# --- Keycloak Operator v26.8.2 (pinned upstream release manifests) ---
# The operator, its CRDs and the SERVER IMAGE move in lockstep: the Keycloak CR has no
# spec.image, so the operator's RELATED_IMAGE_KEYCLOAK (quay.io/keycloak/keycloak:<same tag>)
# is what runs. These three URLs are therefore THE Keycloak version. Each carries a Renovate
# marker (custom.regex manager, renovate.json5) under ONE depName, so all three bump in ONE
# is what runs. These URLs are therefore THE Keycloak version. Each carries a Renovate
# marker (custom.regex manager, renovate.json5) under ONE depName, so all of them bump in ONE
# branch, never auto-merged (packageRule "keycloak-operator"). Before merging a bump, re-check
# passkey-reconciler.yaml's 26.0.6 Admin API assumptions — see README.md "Upgrading Keycloak".
#
# THE URL LIST MUST MATCH UPSTREAM'S kubernetes/kustomization.yml AT THE NEW TAG — Renovate only
# rewrites the version in the URLs that are here; it cannot add a CRD the new release introduced.
# 26.0.6 shipped 2 CRDs; 26.8.2 ships 4 (+ keycloakoidcclients, keycloaksamlclients, both
# v2alpha1). PR #121 bumped only the existing three URLs, and the 26.8.2 operator — which starts an
# informer for EVERY kind it controls, used or not — CrashLooped on
# `Couldn't start informer for keycloaksamlclients.k8s.keycloak.org/v2alpha1 ... Not Found`,
# stalling Flux `platform` and everything downstream (2026-11-08). On every bump, diff
# https://github.com/keycloak/keycloak-k8s-resources/tree/<tag>/kubernetes
# against this list before merging.
# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
- https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/keycloaks.k8s.keycloak.org-v1.yml
# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
- https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/keycloakrealmimports.k8s.keycloak.org-v1.yml
# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
- https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/keycloakoidcclients.k8s.keycloak.org-v1.yml
# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
- https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/keycloaksamlclients.k8s.keycloak.org-v1.yml
# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
- https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/kubernetes.yml
# --- Admin (break-glass) credential — SOPS-encrypted, referenced by name (ADR-0007, R-IDP-04) ---
- keycloak-admin-credentials.sops.yaml
@@ -51,3 +65,23 @@ resources:
- keycloak-db-backup-credentials-prod.sops.yaml
# --- Grafana dashboard for Keycloak health (plain ConfigMap; R-OBS-03) ---
- grafana-dashboard.yaml
# --- Operator memory: upstream ships requests=limits=450Mi, which this cluster outgrew ON 26.0.6 ---
# Measured on prod 2026-11-08: the 26.0.6 operator pod had 154 restarts in 64d, lastState
# `OOMKilled` (exit 137, 2026-11-07T23:09:54Z), and sat at 421Mi of its 450Mi (94%) in `kubectl top`.
# 26.8.2 keeps the same 450Mi default but runs FOUR controllers/informers instead of two (adds
# keycloakoidcclient + keycloaksamlclient) on a newer Quarkus (3.15.1 -> 3.40.1.1), so it would
# plausibly start above a ceiling 26.0.6 already crossed. Limit 768Mi = ~1.8x the observed 421Mi
# working set; request 512Mi = the observed working set plus headroom, so the scheduler reserves what
# it actually uses. Revisit with `kubectl top` once 26.8.2 has run for a week. Patched here (not in
# the remote kubernetes.yml) so a Renovate bump keeps it.
patches:
- target:
kind: Deployment
name: keycloak-operator
patch: |-
- op: replace
path: /spec/template/spec/containers/0/resources/limits/memory
value: 768Mi
- op: replace
path: /spec/template/spec/containers/0/resources/requests/memory
value: 512Mi
renovate.json5
+1 -1
View File
@@ -264,7 +264,7 @@
// and the server image are released in lockstep from keycloak/keycloak-k8s-resources, and the
// Keycloak CR carries no spec.image, so the operator's RELATED_IMAGE_KEYCLOAK picks the server
// (measured on prod 2026-10-30: operator 26.0.6 -> keycloak-0/1 run quay.io/keycloak/keycloak:
// 26.0.6). The three release URLs in platform/keycloak/kustomization.yaml carry `# renovate:`
// 26.0.6). The release URLs in platform/keycloak/kustomization.yaml carry `# renovate:`
// markers under this ONE depName, so they move together in one branch (groupName makes that
// explicit). NEVER AUTO-MERGED, not even a patch: Keycloak is the IdP that gates SSO for every
// app AND kubectl OIDC, and platform/keycloak/passkey-reconciler.yaml encodes Keycloak 26.0.6