Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

fix(keycloak): install the 26.8.2 operator's new CRDs (operator CrashLoop) #144

Merged
jakman85 merged 1 commits from fix/keycloak-operator-crds into main 2026-11-08 06:16:56 +00:00
Conversation 0 Commits 1 Files Changed 4
+121 -5
gitea-admin commented 2026-11-08 06:06:21 +00:00
Owner
Copy link
Copy source

Incident (measured 2026-11-08 ~05:57Z, read-only kubectl --context k3s-prod)

PR #121 bumped the three keycloak-k8s-resources URLs 26.0.6 -> 26.8.2. Upstream kubernetes/ at 26.8.2 ships four CRDs; we install two.

  • RS keycloak-operator-58dff6c4 (quay.io/keycloak/keycloak-operator:26.8.2): 0/1 CrashLoopBackOff, exit 1:
    Couldn't start informer for keycloaksamlclients.k8s.keycloak.org/v2alpha1 resources ... GET https://10.43.0.1:443/apis/k8s.keycloak.org/v2alpha1/namespaces/keycloak/keycloaksamlclients?resourceVersion=0. Message: Not Found.
  • kubectl get crd | grep keycloak: only keycloaks and keycloakrealmimports.
  • RS 78dbb75f64 (26.0.6) still 1/1 (RollingUpdate 25%/25%), keycloak-0/1 on 26.0.6, so SSO is up.
  • Flux platform stuck Healthy=Unknown "Running health checks for revision main@sha1:e8d2666"; platform-config, apps-wave-*, oauth2-proxy, velero, kured, ... all dependency ... is not ready.

Fix

upstream kubernetes/ @ 26.8.2 before after
keycloaks.k8s.keycloak.org-v1.yml yes yes
keycloakrealmimports.k8s.keycloak.org-v1.yml yes yes
keycloakoidcclients.k8s.keycloak.org-v1.yml no added
keycloaksamlclients.k8s.keycloak.org-v1.yml no added
kubernetes.yml (already ships RBAC for both new kinds) yes yes
  • Both new URLs carry the same # renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources marker; the custom.regex manager's pattern matches all 5 (run over the file). Renovate can bump URLs, not add them, so the README now says to diff upstream's list on every bump.
  • Operator memory. Patched (kustomize patches:) from upstream's 450Mi/450Mi to limit 768Mi / request 512Mi. 450Mi is the upstream default and is unchanged at 26.8.2, but on prod the 26.0.6 pod had 154 restarts, lastState: OOMKilled (exit 137, 2026-11-07T23:09:54Z), and kubectl top 421Mi/450Mi. 26.8.2 runs 4 controllers (was 2) on Quarkus 3.40.1.1 (was 3.15.1).
  • No sensitive path touched.

Local gate (pinned toolbox, worktree, both mounts, commit ea30ba8)

kubectl kustomize platform/keycloak: rc 0, 4 CRDs, operator memory 768Mi/512Mi.
ci/gate.sh rc 3 (UNDETERMINED). Did not run: gitea-settings (no token locally). conformance L2/L3 skipped (no cluster). Passed: spec-immutability, approval-policy (N/A offline), decisions-log (78 entries, this one included), lint, l0, render-golden, l1.

Post-merge checks (operator)

  1. kubectl -n keycloak get crd | grep keycloak: 4 CRDs.
  2. New operator pod 1/1 Ready, no restarts; old ReplicaSet 78dbb75f64 scaled to 0.
  3. keycloak-0/1 roll to quay.io/keycloak/keycloak:26.8.2. Expect a full SSO outage while they roll: the CR's spec.update.strategy defaults to RecreateOnImageChange, and the server migrates the DB schema. Check the last CNPG backup first.
  4. kubectl -n keycloak get keycloak keycloak: Ready=True; KeycloakRealmImport not failing.
  5. Flux platform Ready, then platform-config and apps-wave-* Ready.
  6. SSO login works (Grafana + an oauth2-proxy app + kubectl OIDC).
  7. The next keycloak-passkey-reconciler and keycloak-client-secret-reconciler Jobs succeed. The passkey reconciler encodes 26.0.6 Admin API shapes and was not re-checked against 26.8.2.
  8. 26.8.2's Keycloak CRD defaults spec.networkPolicy.enabled: true, so the operator will now create a NetworkPolicy on the Keycloak pods. With no from lists it should allow all sources on HTTP/management. Confirm Gateway traffic and the ServiceMonitor scrape still work.
  9. kubectl top the operator after a week; tune 768Mi.

Handoff: decisions/2026-11-08-keycloak-operator-crds.md.

🤖 Generated with Claude Code

## Incident (measured 2026-11-08 ~05:57Z, read-only `kubectl --context k3s-prod`) PR #121 bumped the three `keycloak-k8s-resources` URLs 26.0.6 -> 26.8.2. Upstream `kubernetes/` at 26.8.2 ships **four** CRDs; we install two. - RS `keycloak-operator-58dff6c4` (quay.io/keycloak/keycloak-operator:26.8.2): 0/1 `CrashLoopBackOff`, exit 1: `Couldn't start informer for keycloaksamlclients.k8s.keycloak.org/v2alpha1 resources ... GET https://10.43.0.1:443/apis/k8s.keycloak.org/v2alpha1/namespaces/keycloak/keycloaksamlclients?resourceVersion=0. Message: Not Found.` - `kubectl get crd | grep keycloak`: only `keycloaks` and `keycloakrealmimports`. - RS `78dbb75f64` (26.0.6) still 1/1 (RollingUpdate 25%/25%), keycloak-0/1 on 26.0.6, so SSO is up. - Flux `platform` stuck `Healthy=Unknown` "Running health checks for revision main@sha1:e8d2666"; platform-config, apps-wave-*, oauth2-proxy, velero, kured, ... all `dependency ... is not ready`. ## Fix | upstream `kubernetes/` @ 26.8.2 | before | after | |---|---|---| | keycloaks.k8s.keycloak.org-v1.yml | yes | yes | | keycloakrealmimports.k8s.keycloak.org-v1.yml | yes | yes | | **keycloakoidcclients.k8s.keycloak.org-v1.yml** | no | **added** | | **keycloaksamlclients.k8s.keycloak.org-v1.yml** | no | **added** | | kubernetes.yml (already ships RBAC for both new kinds) | yes | yes | - Both new URLs carry the same `# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources` marker; the custom.regex manager's pattern matches all 5 (run over the file). Renovate can bump URLs, not add them, so the README now says to diff upstream's list on every bump. - **Operator memory.** Patched (kustomize `patches:`) from upstream's 450Mi/450Mi to **limit 768Mi / request 512Mi**. 450Mi is the upstream default and is unchanged at 26.8.2, but on prod the 26.0.6 pod had **154 restarts**, `lastState: OOMKilled` (exit 137, 2026-11-07T23:09:54Z), and `kubectl top` **421Mi/450Mi**. 26.8.2 runs 4 controllers (was 2) on Quarkus 3.40.1.1 (was 3.15.1). - No sensitive path touched. ## Local gate (pinned toolbox, worktree, both mounts, commit `ea30ba8`) `kubectl kustomize platform/keycloak`: rc 0, 4 CRDs, operator memory 768Mi/512Mi. `ci/gate.sh` **rc 3 (UNDETERMINED)**. Did not run: `gitea-settings` (no token locally). `conformance` L2/L3 skipped (no cluster). Passed: spec-immutability, approval-policy (N/A offline), decisions-log (78 entries, this one included), lint, l0, render-golden, l1. ## Post-merge checks (operator) 1. `kubectl -n keycloak get crd | grep keycloak`: 4 CRDs. 2. New operator pod `1/1 Ready`, no restarts; old ReplicaSet `78dbb75f64` scaled to 0. 3. keycloak-0/1 roll to `quay.io/keycloak/keycloak:26.8.2`. **Expect a full SSO outage while they roll:** the CR's `spec.update.strategy` defaults to `RecreateOnImageChange`, and the server migrates the DB schema. Check the last CNPG backup first. 4. `kubectl -n keycloak get keycloak keycloak`: Ready=True; KeycloakRealmImport not failing. 5. Flux `platform` Ready, then platform-config and apps-wave-* Ready. 6. SSO login works (Grafana + an oauth2-proxy app + kubectl OIDC). 7. The next `keycloak-passkey-reconciler` and `keycloak-client-secret-reconciler` Jobs succeed. The passkey reconciler encodes 26.0.6 Admin API shapes and was **not re-checked** against 26.8.2. 8. 26.8.2's Keycloak CRD defaults `spec.networkPolicy.enabled: true`, so the operator will now create a NetworkPolicy on the Keycloak pods. With no `from` lists it should allow all sources on HTTP/management. Confirm Gateway traffic and the ServiceMonitor scrape still work. 9. `kubectl top` the operator after a week; tune 768Mi. Handoff: `decisions/2026-11-08-keycloak-operator-crds.md`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
gitea-admin added the needs-reviewblast-radius labels 2026-11-08 06:06:21 +00:00
gitea-admin added 1 commit 2026-11-08 06:06:21 +00:00
fix(keycloak): install the 26.8.2 operator's new CRDs (operator CrashLoop)
gate / gate (pull_request) Successful in 3m5s
Details
gate / conformance (pull_request) Skipped
Details
ea30ba876c 
PR #121 bumped the three keycloak-k8s-resources URLs to 26.8.2, but 26.8.2
ships two more CRDs (keycloakoidcclients, keycloaksamlclients). The new
operator fails to start its keycloaksamlclients informer (Not Found) and
CrashLoops, holding Flux `platform` in health checks.

- add both CRD URLs with the same renovate marker/depName
- patch operator memory 450Mi -> limit 768Mi / request 512Mi (26.0.6 was
  OOMKilled at 421/450Mi, 154 restarts)
- README/renovate comments: Renovate cannot add CRDs a release introduces

Co-Authored-By: Claude Opus 5.5 <[email protected]>
jakman85 merged commit 4a3f6dee59 into main 2026-11-08 06:16:56 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
agent
automated-update
blast-radius
breaking
ci-tooling
dependencies
needs-review
own-image
range-widen
No labels blast-radius needs-review
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
gitea-admin jakman85 (Keith Jackson) pm-agent renovate-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Home-Network/home-network-k3s-impl#144