fix(keycloak): one owner for the keycloak ServiceMonitor (26.8 operator 409 loop blocks Flux)

The 26.x Keycloak operator creates its own ServiceMonitor named after the
CR (keycloak/keycloak) when metrics are enabled; spec.serviceMonitor.enabled
defaults true. That collides with the Flux-owned platform/keycloak/
servicemonitor.yaml: the operator's PATCH 409s, it exhausts retries, the CR
stays Ready=False/HasErrors=True, and platform-config (wait: true) stalls
every downstream Kustomization.

Keep the repo's ServiceMonitor (conformance L0/L2 and the dashboard are
written against it; the operator's scrapes the same port 9000 /metrics) and
set spec.serviceMonitor.enabled: false.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

decisions/2026-11-08-keycloak-operator-crds.md

@@ -0,0 +1,74 @@
+---
+date: 2026-11-08
+branch: fix/keycloak-operator-crds
+title: "Keycloak 26.8.2 operator CrashLoop: install its two new CRDs, raise its memory"
+---
+
+## What changed
+
+**Incident.** PR #121 (Renovate) bumped `platform/keycloak/kustomization.yaml`'s three
+`keycloak-k8s-resources` URLs 26.0.6 -> 26.8.2. Upstream `kubernetes/` at 26.8.2 holds **four**
+CRDs, not two (GitHub contents API, `?ref=26.8.2`):
+
+| file | at 26.0.6 | at 26.8.2 |
+|---|---|---|
+| `keycloaks.k8s.keycloak.org-v1.yml` | yes | yes |
+| `keycloakrealmimports.k8s.keycloak.org-v1.yml` | yes | yes |
+| `keycloakoidcclients.k8s.keycloak.org-v1.yml` | — | **new** (v2alpha1) |
+| `keycloaksamlclients.k8s.keycloak.org-v1.yml` | — | **new** (v2alpha1) |
+| `kubernetes.yml` | yes | yes (now carries RBAC for both new kinds) |
+
+(`cluster-wide/` is a separate variant; upstream's own `kubernetes/kustomization.yml` lists exactly
+the five files above.) The 26.8.2 operator starts an informer for every controller it has and dies
+on the first missing kind: `Couldn't start informer for keycloaksamlclients.k8s.keycloak.org/v2alpha1
+... GET .../namespaces/keycloak/keycloaksamlclients ... Not Found`, exit 1, CrashLoopBackOff. Flux
+`platform` (`wait: true`, 5m) sat in health checks and every dependent Kustomization reported
+`dependency ... is not ready`. The old 26.0.6 ReplicaSet kept serving (RollingUpdate 25%/25%), so SSO
+stayed up.
+
+**Fix.**
+- `platform/keycloak/kustomization.yaml`: added the two CRD URLs, each with the same
+  `# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources` marker (the repo's
+  custom.regex manager matches all five — checked by running its regex over the file). Header
+  comment updated to v26.8.2, plus a warning that Renovate cannot add a CRD a release introduces.
+- Same file: a `patches:` JSON6902 patch on Deployment `keycloak-operator`: memory limit
+  450Mi -> **768Mi**, request 450Mi -> **512Mi**. 450Mi is upstream's default (unchanged at 26.8.2,
+  so not something the bump changed). Evidence it is too small here: the 26.0.6 pod had **154
+  restarts**, `lastState.terminated.reason: OOMKilled` (exit 137, 2026-11-07T23:09:54Z), and
+  `kubectl top` showed **421Mi / 450Mi**. 26.8.2 runs four controllers instead of two on Quarkus
+  3.40.1.1 (was 3.15.1). No variable in the operator to disable watching unused kinds was used —
+  installing the CRDs is upstream's supported shape.
+- `platform/keycloak/README.md` "Upgrading Keycloak": the diff-the-CRD-list step; "three URLs" ->
+  "the release URLs". `renovate.json5`: same wording fix in a comment only.
+
+## What I verified
+
+- Upstream file lists at 26.0.6 and 26.8.2 (GitHub API) and `diff` of `kubernetes.yml` between the
+  two: the new kinds' ClusterRoles + RoleBindings ship in it, and the Deployment's resources are
+  still 450Mi/450Mi.
+- Live, read-only (`kubectl --context k3s-prod`, 2026-11-08 ~05:57Z): only `keycloaks` and
+  `keycloakrealmimports` CRDs installed; Deployment strategy RollingUpdate maxSurge/maxUnavailable
+  25%; RS `keycloak-operator-58dff6c4` (26.8.2) 0/1 CrashLoopBackOff, RS `78dbb75f64` (26.0.6) 1/1;
+  operator log is the informer error above; Flux `platform` `Healthy=Unknown` "Running health
+  checks for revision main@sha1:e8d2666"; no NetworkPolicy and no CiliumNetworkPolicy in ns
+  `keycloak`.
+- No sensitive path touched (`infra/gitea/approval-policy.json`), so this should merge on green.
+- Gate: see the PR body for the exact rc and stages of the run against the final commit.
+
+## What I did NOT verify
+
+- **That the 26.8.2 operator becomes Ready after this merges.** Only the two missing kinds were
+  observed failing; another start-up failure behind them is possible.
+- **That 768Mi is enough** — no 26.8.2 memory measurement exists yet; re-check `kubectl top` after
+  a week.
+- **The Keycloak server upgrade 26.0.6 -> 26.8.2 itself.** Once the operator runs it will roll
+  keycloak-0/1 and Keycloak migrates the DB schema. The CR's `spec.update.strategy` defaults to
+  `RecreateOnImageChange` — expect a **full SSO outage** while both pods recreate, not a rolling
+  one. Back up / confirm the latest CNPG backup first if that matters.
+- **The 26.8.2 Keycloak CRD's `spec.networkPolicy.enabled` defaults to `true`.** The operator will
+  create a NetworkPolicy selecting the Keycloak pods (first policy in that namespace). With no
+  `from` lists set it allows all sources on the HTTP/management ports, so Gateway, ServiceMonitor
+  and the reconciler Jobs should still reach it — but this was read from the CRD text, not observed.
+- **The passkey reconciler's 26.0.6 Admin API assumptions against 26.8.2** (README "Upgrading
+  Keycloak" list) — not re-checked here; PR #121 merged without it.
+- `catalog/keycloak.yaml`'s documentation-only image tag was not updated.

decisions/2026-11-08-keycloak-servicemonitor-owner.md

@@ -0,0 +1,60 @@
+---
+date: 2026-11-08
+branch: fix/keycloak-servicemonitor-owner
+title: One owner for the keycloak ServiceMonitor — disable the 26.8 operator's (409 loop stalled platform-config)
+---
+
+## What changed
+
+- `platform/keycloak/keycloak-instance.yaml`: added `spec.serviceMonitor.enabled: false`.
+  The 26.x Keycloak operator creates its own ServiceMonitor, named after the CR (`keycloak/keycloak`),
+  whenever `metrics-enabled=true` and the ServiceMonitor CRD exists. `serviceMonitor.enabled` defaults
+  to `true` in the 26.8.2 CRD, both in v2alpha1 and in v2beta1 (v2beta1 is now the storage version).
+  That is the same name and namespace as the repo's Flux-owned `platform/keycloak/servicemonitor.yaml`.
+  After the 26.0.6 -> 26.8.2 bump (#121/#144) the operator's PATCH against it returned
+  `409 Conflict` and it ran out of retries. The CR then sat at `Ready=False`/`HasErrors=True`, and
+  `platform-config` (`wait: true`, which health-checks every object it applies, this CR included)
+  blocked everything that `dependsOn` it: velero, apps-vms, apps-wave-0 (and so waves 1/2 and
+  pm-agent), homepage, oauth2-proxy, headlamp, conformance, vpn-ingress, crafty, runbooks.
+- `platform/keycloak/servicemonitor.yaml`: a comment saying it is the sole owner.
+- **Why ours is kept:** these are written against it: the L0 `servicemonitor-ports-resolve`
+  allow-list entry `platform/keycloak/servicemonitor.yaml:keycloak`, the L2 `keycloak-monitored`
+  chainsaw test, the README's ownership table, and the dashboard's `up{namespace="keycloak"}`.
+  The operator's version (upstream `KeycloakServiceMonitorDependentResource` @26.8.2) scrapes the same
+  target: port name `management` (9000 on `keycloak-service`), path `/metrics`, interval 30s,
+  selector = the operator's instance labels. It offers nothing ours lacks, so having two owners
+  would add a fight and gain nothing.
+
+## What I verified
+
+- Live, read-only, `k3s-prod` ~06:25Z: the CRD schema field `spec.serviceMonitor.enabled`
+  (`default: true`) exists in both served versions. The live `keycloak` ServiceMonitor has exactly
+  **one** managedFields manager, `kustomize-controller` (Apply, 2026-09-05), **no ownerReferences**,
+  and labels `kustomize.toolkit.fluxcd.io/name=platform-config`, `release=kube-prometheus-stack`. So
+  the operator's apply never landed, and nothing on the object ties it to the Keycloak CR.
+- The ServiceMonitor will not be deleted when the operator is switched off: the operator's dependent
+  informer is filtered on `Constants.DEFAULT_LABELS_AS_STRING`
+  (`app.kubernetes.io/managed-by=keycloak-operator`). Our object does not carry that label and has
+  no ownerReference, so the operator cannot see it as its secondary and cannot delete or GC it.
+  This conclusion comes from reading the code. It was not observed.
+- The other operator dependents in ns `keycloak` do not collide with anything: `keycloak-service`,
+  `keycloak-discovery`, the StatefulSet `keycloak`, `keycloak-initial-admin`, and
+  `keycloak-network-policy` (owner Keycloak, no Flux label). The repo declares none of these names.
+  It only *references* `keycloak-service` by name (HTTPRoute, reconcilers). There are no PDBs in the
+  namespace, and the Ingress is already `enabled: false`. The only NetworkPolicy in the namespace is
+  the operator's. The repo has no Keycloak NetworkPolicy, and no Kyverno-generated default-deny is
+  present there. NetworkPolicies add up, so the operator's allow-policy cannot fight another owner.
+  Left at its default.
+- Local gate: see the PR body. It records the rc and stages.
+
+## What I did NOT verify
+
+- That the CR reaches `Ready=True`/`HasErrors=False` after merge. The status also says
+  "Performing Keycloak upgrade, scaling down the deployment". I am inferring that this half clears
+  once the workflow stops erroring. It is not proven.
+- That the operator stops issuing the PATCH once `enabled: false` is applied. Watch the operator
+  logs for 409.
+- That `platform-config` and its downstream Kustomizations go Ready on the new revision, and that
+  Prometheus keeps exactly one `serviceMonitor/keycloak/keycloak/0` pool with targets up.
+- The operator's actual label set for `DEFAULT_LABELS_AS_STRING`. I took it from the observed labels
+  on operator-created objects, not from `Constants.java`.

platform/keycloak/README.md

@@ -367,9 +367,17 @@ is involved — so step 2 failing with no log entry is the expected signature of
 The `Keycloak` CR has no `spec.image`, so the server image is whatever the operator's
 `RELATED_IMAGE_KEYCLOAK` names — the same tag as the operator (prod, 2026-10-30: operator
 `26.0.6`, `keycloak-0/1` on `quay.io/keycloak/keycloak:26.0.6`). Operator, CRDs and server are
-released in lockstep from `keycloak/keycloak-k8s-resources`, and the three URLs in
+released in lockstep from `keycloak/keycloak-k8s-resources`, and the release URLs in
 `kustomization.yaml` are the one knob. Renovate raises them as ONE PR (`keycloak-operator` group,
 labels `needs-review` + `blast-radius`), **never auto-merged**; majors wait for a dashboard tick.
+
+**Renovate bumps the URLs that exist; it cannot add a CRD a release introduced.** Before merging a
+bump, diff `kubernetes/` (and its `kustomization.yml`) at the new tag against the URL list. 26.0.6
+shipped 2 CRDs, 26.8.2 ships 4 (`keycloakoidcclients`, `keycloaksamlclients` added); PR #121 bumped
+only the existing URLs and the 26.8.2 operator CrashLooped on the missing `keycloaksamlclients`
+informer, stalling Flux `platform` and every tier below it (2026-11-08,
+`decisions/2026-11-08-keycloak-operator-crds.md`). The operator's memory is also patched in
+`kustomization.yaml` (upstream's 450Mi OOMKilled 26.0.6 here); keep the patch across bumps.
 `catalog/keycloak.yaml`'s image tag is documentation only (its render deploys nothing) and
 Renovate is disabled for it — update it by hand to match after an operator bump merges.
 

platform/keycloak/keycloak-instance.yaml

@@ -38,6 +38,20 @@ spec:
   ingress:
     enabled: false                         # exposed via Gateway API HTTPRoute (httproute.yaml), not the
                                            # operator's built-in Ingress
+  # --- ONE OWNER for the `keycloak` ServiceMonitor: this repo (servicemonitor.yaml), not the operator.
+  #     Since 26.x the operator creates its OWN ServiceMonitor, named after this CR (`keycloak`, i.e.
+  #     the SAME name/namespace as ours), whenever metrics-enabled=true and the CRD exists —
+  #     spec.serviceMonitor.enabled defaults TRUE. On 2026-11-08, right after the 26.0.6 -> 26.8.2
+  #     bump, its server-side apply onto our Flux-owned object failed `409 Conflict`, the operator
+  #     exhausted retries, the CR sat Ready=False/HasErrors=True, and `platform-config` (wait: true,
+  #     so it health-checks this CR) stalled every Kustomization downstream of it. Ours is kept
+  #     because the L0 servicemonitor-ports-resolve and L2 keycloak-monitored conformance tests and
+  #     the dashboard's up{namespace="keycloak"} are written against it; the operator's would scrape
+  #     the same thing (management port 9000, /metrics, 30s). Do NOT flip this to true without
+  #     deleting servicemonitor.yaml in the same change — two owners of one object is a fight
+  #     (cf. the default-deny NetworkPolicy drift loop).
+  serviceMonitor:
+    enabled: false
   # --- Hardening + observability options (docs/15). Brute-force detection is a REALM setting
   #     (bruteForceProtected in the realm import); these enable health + metrics endpoints. ---
   additionalOptions:

platform/keycloak/kustomization.yaml

@@ -20,18 +20,32 @@ kind: Kustomization
 namespace: keycloak
 resources:
   - namespace.yaml
-  # --- Keycloak Operator v26.0.6 (pinned upstream release manifests) ---
+  # --- Keycloak Operator v26.8.2 (pinned upstream release manifests) ---
   # The operator, its CRDs and the SERVER IMAGE move in lockstep: the Keycloak CR has no
   # spec.image, so the operator's RELATED_IMAGE_KEYCLOAK (quay.io/keycloak/keycloak:<same tag>)
-  # is what runs. These three URLs are therefore THE Keycloak version. Each carries a Renovate
-  # marker (custom.regex manager, renovate.json5) under ONE depName, so all three bump in ONE
+  # is what runs. These URLs are therefore THE Keycloak version. Each carries a Renovate
+  # marker (custom.regex manager, renovate.json5) under ONE depName, so all of them bump in ONE
   # branch, never auto-merged (packageRule "keycloak-operator"). Before merging a bump, re-check
   # passkey-reconciler.yaml's 26.0.6 Admin API assumptions — see README.md "Upgrading Keycloak".
+  #
+  # THE URL LIST MUST MATCH UPSTREAM'S kubernetes/kustomization.yml AT THE NEW TAG — Renovate only
+  # rewrites the version in the URLs that are here; it cannot add a CRD the new release introduced.
+  # 26.0.6 shipped 2 CRDs; 26.8.2 ships 4 (+ keycloakoidcclients, keycloaksamlclients, both
+  # v2alpha1). PR #121 bumped only the existing three URLs, and the 26.8.2 operator — which starts an
+  # informer for EVERY kind it controls, used or not — CrashLooped on
+  # `Couldn't start informer for keycloaksamlclients.k8s.keycloak.org/v2alpha1 ... Not Found`,
+  # stalling Flux `platform` and everything downstream (2026-11-08). On every bump, diff
+  #   https://github.com/keycloak/keycloak-k8s-resources/tree/<tag>/kubernetes
+  # against this list before merging.
   # renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
   - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/keycloaks.k8s.keycloak.org-v1.yml
   # renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
   - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/keycloakrealmimports.k8s.keycloak.org-v1.yml
   # renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
+  - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/keycloakoidcclients.k8s.keycloak.org-v1.yml
+  # renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
+  - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/keycloaksamlclients.k8s.keycloak.org-v1.yml
+  # renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
   - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/kubernetes.yml
   # --- Admin (break-glass) credential — SOPS-encrypted, referenced by name (ADR-0007, R-IDP-04) ---
   - keycloak-admin-credentials.sops.yaml
@@ -51,3 +65,23 @@ resources:
   - keycloak-db-backup-credentials-prod.sops.yaml
   # --- Grafana dashboard for Keycloak health (plain ConfigMap; R-OBS-03) ---
   - grafana-dashboard.yaml
+# --- Operator memory: upstream ships requests=limits=450Mi, which this cluster outgrew ON 26.0.6 ---
+# Measured on prod 2026-11-08: the 26.0.6 operator pod had 154 restarts in 64d, lastState
+# `OOMKilled` (exit 137, 2026-11-07T23:09:54Z), and sat at 421Mi of its 450Mi (94%) in `kubectl top`.
+# 26.8.2 keeps the same 450Mi default but runs FOUR controllers/informers instead of two (adds
+# keycloakoidcclient + keycloaksamlclient) on a newer Quarkus (3.15.1 -> 3.40.1.1), so it would
+# plausibly start above a ceiling 26.0.6 already crossed. Limit 768Mi = ~1.8x the observed 421Mi
+# working set; request 512Mi = the observed working set plus headroom, so the scheduler reserves what
+# it actually uses. Revisit with `kubectl top` once 26.8.2 has run for a week. Patched here (not in
+# the remote kubernetes.yml) so a Renovate bump keeps it.
+patches:
+  - target:
+      kind: Deployment
+      name: keycloak-operator
+    patch: |-
+      - op: replace
+        path: /spec/template/spec/containers/0/resources/limits/memory
+        value: 768Mi
+      - op: replace
+        path: /spec/template/spec/containers/0/resources/requests/memory
+        value: 512Mi

platform/keycloak/servicemonitor.yaml

@@ -8,6 +8,9 @@
 # /metrics) once metrics-enabled/health-enabled are set (see keycloak-instance.yaml additionalOptions).
 # We select the operator-created keycloak-service (which selects the Keycloak pods) and scrape the pods'
 # management port by number via targetPort.
+# SOLE OWNER: the operator's own ServiceMonitor (same name `keycloak`) is switched OFF in
+#   keycloak-instance.yaml (spec.serviceMonitor.enabled: false) — 26.x defaults it ON and its apply
+#   409-looped against this object on 2026-11-08. Keep exactly one of the two.
 # TODO(WS-SEC, bring-up): confirm the management port number/name the operator exposes (9000 default)
 #   and adjust targetPort if the pinned operator changes it. The KeycloakDown / KeycloakDatabaseDown
 #   alerts (platform/monitoring/alerts.yaml) key on kube-state-metrics and fire regardless of this SM.

renovate.json5

@@ -287,7 +287,7 @@
     // and the server image are released in lockstep from keycloak/keycloak-k8s-resources, and the
     // Keycloak CR carries no spec.image, so the operator's RELATED_IMAGE_KEYCLOAK picks the server
     // (measured on prod 2026-10-30: operator 26.0.6 -> keycloak-0/1 run quay.io/keycloak/keycloak:
-    // 26.0.6). The three release URLs in platform/keycloak/kustomization.yaml carry `# renovate:`
+    // 26.0.6). The release URLs in platform/keycloak/kustomization.yaml carry `# renovate:`
     // markers under this ONE depName, so they move together in one branch (groupName makes that
     // explicit). NEVER AUTO-MERGED, not even a patch: Keycloak is the IdP that gates SSO for every
     // app AND kubectl OIDC, and platform/keycloak/passkey-reconciler.yaml encodes Keycloak 26.0.6