From 35ba8753809b528df1b3101dfc5993df27d124b6 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 8 Nov 2026 00:25:45 -0600 Subject: [PATCH] fix(keycloak): one owner for the keycloak ServiceMonitor (26.8 operator 409 loop blocks Flux) The 26.x Keycloak operator creates its own ServiceMonitor named after the CR (keycloak/keycloak) when metrics are enabled; spec.serviceMonitor.enabled defaults true. That collides with the Flux-owned platform/keycloak/ servicemonitor.yaml: the operator's PATCH 409s, it exhausts retries, the CR stays Ready=False/HasErrors=True, and platform-config (wait: true) stalls every downstream Kustomization. Keep the repo's ServiceMonitor (conformance L0/L2 and the dashboard are written against it; the operator's scrapes the same port 9000 /metrics) and set spec.serviceMonitor.enabled: false. Co-Authored-By: Claude Opus 5.5 --- ...026-11-08-keycloak-servicemonitor-owner.md | 60 +++++++++++++++++++ platform/keycloak/keycloak-instance.yaml | 14 +++++ platform/keycloak/servicemonitor.yaml | 3 + 3 files changed, 77 insertions(+) create mode 100644 decisions/2026-11-08-keycloak-servicemonitor-owner.md diff --git a/decisions/2026-11-08-keycloak-servicemonitor-owner.md b/decisions/2026-11-08-keycloak-servicemonitor-owner.md new file mode 100644 index 0000000..6fb8dce --- /dev/null +++ b/decisions/2026-11-08-keycloak-servicemonitor-owner.md @@ -0,0 +1,60 @@ +--- +date: 2026-11-08 +branch: fix/keycloak-servicemonitor-owner +title: One owner for the keycloak ServiceMonitor — disable the 26.8 operator's (409 loop stalled platform-config) +--- + +## What changed + +- `platform/keycloak/keycloak-instance.yaml`: added `spec.serviceMonitor.enabled: false`. + The 26.x Keycloak operator creates its own ServiceMonitor, named after the CR (`keycloak/keycloak`), + whenever `metrics-enabled=true` and the ServiceMonitor CRD exists. `serviceMonitor.enabled` defaults + to `true` in the 26.8.2 CRD, both in v2alpha1 and in v2beta1 (v2beta1 is now the storage version). + That is the same name and namespace as the repo's Flux-owned `platform/keycloak/servicemonitor.yaml`. + After the 26.0.6 -> 26.8.2 bump (#121/#144) the operator's PATCH against it returned + `409 Conflict` and it ran out of retries. The CR then sat at `Ready=False`/`HasErrors=True`, and + `platform-config` (`wait: true`, which health-checks every object it applies, this CR included) + blocked everything that `dependsOn` it: velero, apps-vms, apps-wave-0 (and so waves 1/2 and + pm-agent), homepage, oauth2-proxy, headlamp, conformance, vpn-ingress, crafty, runbooks. +- `platform/keycloak/servicemonitor.yaml`: a comment saying it is the sole owner. +- **Why ours is kept:** these are written against it: the L0 `servicemonitor-ports-resolve` + allow-list entry `platform/keycloak/servicemonitor.yaml:keycloak`, the L2 `keycloak-monitored` + chainsaw test, the README's ownership table, and the dashboard's `up{namespace="keycloak"}`. + The operator's version (upstream `KeycloakServiceMonitorDependentResource` @26.8.2) scrapes the same + target: port name `management` (9000 on `keycloak-service`), path `/metrics`, interval 30s, + selector = the operator's instance labels. It offers nothing ours lacks, so having two owners + would add a fight and gain nothing. + +## What I verified + +- Live, read-only, `k3s-prod` ~06:25Z: the CRD schema field `spec.serviceMonitor.enabled` + (`default: true`) exists in both served versions. The live `keycloak` ServiceMonitor has exactly + **one** managedFields manager, `kustomize-controller` (Apply, 2026-09-05), **no ownerReferences**, + and labels `kustomize.toolkit.fluxcd.io/name=platform-config`, `release=kube-prometheus-stack`. So + the operator's apply never landed, and nothing on the object ties it to the Keycloak CR. +- The ServiceMonitor will not be deleted when the operator is switched off: the operator's dependent + informer is filtered on `Constants.DEFAULT_LABELS_AS_STRING` + (`app.kubernetes.io/managed-by=keycloak-operator`). Our object does not carry that label and has + no ownerReference, so the operator cannot see it as its secondary and cannot delete or GC it. + This conclusion comes from reading the code. It was not observed. +- The other operator dependents in ns `keycloak` do not collide with anything: `keycloak-service`, + `keycloak-discovery`, the StatefulSet `keycloak`, `keycloak-initial-admin`, and + `keycloak-network-policy` (owner Keycloak, no Flux label). The repo declares none of these names. + It only *references* `keycloak-service` by name (HTTPRoute, reconcilers). There are no PDBs in the + namespace, and the Ingress is already `enabled: false`. The only NetworkPolicy in the namespace is + the operator's. The repo has no Keycloak NetworkPolicy, and no Kyverno-generated default-deny is + present there. NetworkPolicies add up, so the operator's allow-policy cannot fight another owner. + Left at its default. +- Local gate: see the PR body. It records the rc and stages. + +## What I did NOT verify + +- That the CR reaches `Ready=True`/`HasErrors=False` after merge. The status also says + "Performing Keycloak upgrade, scaling down the deployment". I am inferring that this half clears + once the workflow stops erroring. It is not proven. +- That the operator stops issuing the PATCH once `enabled: false` is applied. Watch the operator + logs for 409. +- That `platform-config` and its downstream Kustomizations go Ready on the new revision, and that + Prometheus keeps exactly one `serviceMonitor/keycloak/keycloak/0` pool with targets up. +- The operator's actual label set for `DEFAULT_LABELS_AS_STRING`. I took it from the observed labels + on operator-created objects, not from `Constants.java`. diff --git a/platform/keycloak/keycloak-instance.yaml b/platform/keycloak/keycloak-instance.yaml index 4969995..a94e430 100644 --- a/platform/keycloak/keycloak-instance.yaml +++ b/platform/keycloak/keycloak-instance.yaml @@ -38,6 +38,20 @@ spec: ingress: enabled: false # exposed via Gateway API HTTPRoute (httproute.yaml), not the # operator's built-in Ingress + # --- ONE OWNER for the `keycloak` ServiceMonitor: this repo (servicemonitor.yaml), not the operator. + # Since 26.x the operator creates its OWN ServiceMonitor, named after this CR (`keycloak`, i.e. + # the SAME name/namespace as ours), whenever metrics-enabled=true and the CRD exists — + # spec.serviceMonitor.enabled defaults TRUE. On 2026-11-08, right after the 26.0.6 -> 26.8.2 + # bump, its server-side apply onto our Flux-owned object failed `409 Conflict`, the operator + # exhausted retries, the CR sat Ready=False/HasErrors=True, and `platform-config` (wait: true, + # so it health-checks this CR) stalled every Kustomization downstream of it. Ours is kept + # because the L0 servicemonitor-ports-resolve and L2 keycloak-monitored conformance tests and + # the dashboard's up{namespace="keycloak"} are written against it; the operator's would scrape + # the same thing (management port 9000, /metrics, 30s). Do NOT flip this to true without + # deleting servicemonitor.yaml in the same change — two owners of one object is a fight + # (cf. the default-deny NetworkPolicy drift loop). + serviceMonitor: + enabled: false # --- Hardening + observability options (docs/15). Brute-force detection is a REALM setting # (bruteForceProtected in the realm import); these enable health + metrics endpoints. --- additionalOptions: diff --git a/platform/keycloak/servicemonitor.yaml b/platform/keycloak/servicemonitor.yaml index b5d2e63..a09cc86 100644 --- a/platform/keycloak/servicemonitor.yaml +++ b/platform/keycloak/servicemonitor.yaml @@ -8,6 +8,9 @@ # /metrics) once metrics-enabled/health-enabled are set (see keycloak-instance.yaml additionalOptions). # We select the operator-created keycloak-service (which selects the Keycloak pods) and scrape the pods' # management port by number via targetPort. +# SOLE OWNER: the operator's own ServiceMonitor (same name `keycloak`) is switched OFF in +# keycloak-instance.yaml (spec.serviceMonitor.enabled: false) — 26.x defaults it ON and its apply +# 409-looped against this object on 2026-11-08. Keep exactly one of the two. # TODO(WS-SEC, bring-up): confirm the management port number/name the operator exposes (9000 default) # and adjust targetPort if the pinned operator changes it. The KeycloakDown / KeycloakDatabaseDown # alerts (platform/monitoring/alerts.yaml) key on kube-state-metrics and fire regardless of this SM.