--- date: 2026-11-08 branch: fix/keycloak-servicemonitor-owner title: One owner for the keycloak ServiceMonitor — disable the 26.8 operator's (409 loop stalled platform-config) --- ## What changed - `platform/keycloak/keycloak-instance.yaml`: added `spec.serviceMonitor.enabled: false`. The 26.x Keycloak operator creates its own ServiceMonitor, named after the CR (`keycloak/keycloak`), whenever `metrics-enabled=true` and the ServiceMonitor CRD exists. `serviceMonitor.enabled` defaults to `true` in the 26.8.2 CRD, both in v2alpha1 and in v2beta1 (v2beta1 is now the storage version). That is the same name and namespace as the repo's Flux-owned `platform/keycloak/servicemonitor.yaml`. After the 26.0.6 -> 26.8.2 bump (#121/#144) the operator's PATCH against it returned `409 Conflict` and it ran out of retries. The CR then sat at `Ready=False`/`HasErrors=True`, and `platform-config` (`wait: true`, which health-checks every object it applies, this CR included) blocked everything that `dependsOn` it: velero, apps-vms, apps-wave-0 (and so waves 1/2 and pm-agent), homepage, oauth2-proxy, headlamp, conformance, vpn-ingress, crafty, runbooks. - `platform/keycloak/servicemonitor.yaml`: a comment saying it is the sole owner. - **Why ours is kept:** these are written against it: the L0 `servicemonitor-ports-resolve` allow-list entry `platform/keycloak/servicemonitor.yaml:keycloak`, the L2 `keycloak-monitored` chainsaw test, the README's ownership table, and the dashboard's `up{namespace="keycloak"}`. The operator's version (upstream `KeycloakServiceMonitorDependentResource` @26.8.2) scrapes the same target: port name `management` (9000 on `keycloak-service`), path `/metrics`, interval 30s, selector = the operator's instance labels. It offers nothing ours lacks, so having two owners would add a fight and gain nothing. ## What I verified - Live, read-only, `k3s-prod` ~06:25Z: the CRD schema field `spec.serviceMonitor.enabled` (`default: true`) exists in both served versions. The live `keycloak` ServiceMonitor has exactly **one** managedFields manager, `kustomize-controller` (Apply, 2026-09-05), **no ownerReferences**, and labels `kustomize.toolkit.fluxcd.io/name=platform-config`, `release=kube-prometheus-stack`. So the operator's apply never landed, and nothing on the object ties it to the Keycloak CR. - The ServiceMonitor will not be deleted when the operator is switched off: the operator's dependent informer is filtered on `Constants.DEFAULT_LABELS_AS_STRING` (`app.kubernetes.io/managed-by=keycloak-operator`). Our object does not carry that label and has no ownerReference, so the operator cannot see it as its secondary and cannot delete or GC it. This conclusion comes from reading the code. It was not observed. - The other operator dependents in ns `keycloak` do not collide with anything: `keycloak-service`, `keycloak-discovery`, the StatefulSet `keycloak`, `keycloak-initial-admin`, and `keycloak-network-policy` (owner Keycloak, no Flux label). The repo declares none of these names. It only *references* `keycloak-service` by name (HTTPRoute, reconcilers). There are no PDBs in the namespace, and the Ingress is already `enabled: false`. The only NetworkPolicy in the namespace is the operator's. The repo has no Keycloak NetworkPolicy, and no Kyverno-generated default-deny is present there. NetworkPolicies add up, so the operator's allow-policy cannot fight another owner. Left at its default. - Local gate: see the PR body. It records the rc and stages. ## What I did NOT verify - That the CR reaches `Ready=True`/`HasErrors=False` after merge. The status also says "Performing Keycloak upgrade, scaling down the deployment". I am inferring that this half clears once the workflow stops erroring. It is not proven. - That the operator stops issuing the PATCH once `enabled: false` is applied. Watch the operator logs for 409. - That `platform-config` and its downstream Kustomizations go Ready on the new revision, and that Prometheus keeps exactly one `serviceMonitor/keycloak/keycloak/0` pool with targets up. - The operator's actual label set for `DEFAULT_LABELS_AS_STRING`. I took it from the observed labels on operator-created objects, not from `Constants.java`.