home-network-k3s-impl/decisions/2026-11-08-keycloak-servicemonitor-owner.md

date, branch, title

date	branch	title
2026-11-08	fix/keycloak-servicemonitor-owner	One owner for the keycloak ServiceMonitor — disable the 26.8 operator's (409 loop stalled platform-config)

What changed

• platform/keycloak/keycloak-instance.yaml: added spec.serviceMonitor.enabled: false. The 26.x Keycloak operator creates its own ServiceMonitor, named after the CR (keycloak/keycloak), whenever metrics-enabled=true and the ServiceMonitor CRD exists. serviceMonitor.enabled defaults to true in the 26.8.2 CRD, both in v2alpha1 and in v2beta1 (v2beta1 is now the storage version). That is the same name and namespace as the repo's Flux-owned platform/keycloak/servicemonitor.yaml. After the 26.0.6 -> 26.8.2 bump (#121/#144) the operator's PATCH against it returned 409 Conflict and it ran out of retries. The CR then sat at Ready=False/HasErrors=True, and platform-config (wait: true, which health-checks every object it applies, this CR included) blocked everything that dependsOn it: velero, apps-vms, apps-wave-0 (and so waves 1/2 and pm-agent), homepage, oauth2-proxy, headlamp, conformance, vpn-ingress, crafty, runbooks.
• platform/keycloak/servicemonitor.yaml: a comment saying it is the sole owner.
• Why ours is kept: these are written against it: the L0 servicemonitor-ports-resolve allow-list entry platform/keycloak/servicemonitor.yaml:keycloak, the L2 keycloak-monitored chainsaw test, the README's ownership table, and the dashboard's up{namespace="keycloak"}. The operator's version (upstream KeycloakServiceMonitorDependentResource @26.8.2) scrapes the same target: port name management (9000 on keycloak-service), path /metrics, interval 30s, selector = the operator's instance labels. It offers nothing ours lacks, so having two owners would add a fight and gain nothing.

What I verified

• Live, read-only, k3s-prod ~06:25Z: the CRD schema field spec.serviceMonitor.enabled (default: true) exists in both served versions. The live keycloak ServiceMonitor has exactly one managedFields manager, kustomize-controller (Apply, 2026-09-05), no ownerReferences, and labels kustomize.toolkit.fluxcd.io/name=platform-config, release=kube-prometheus-stack. So the operator's apply never landed, and nothing on the object ties it to the Keycloak CR.
• The ServiceMonitor will not be deleted when the operator is switched off: the operator's dependent informer is filtered on Constants.DEFAULT_LABELS_AS_STRING (app.kubernetes.io/managed-by=keycloak-operator). Our object does not carry that label and has no ownerReference, so the operator cannot see it as its secondary and cannot delete or GC it. This conclusion comes from reading the code. It was not observed.
• The other operator dependents in ns keycloak do not collide with anything: keycloak-service, keycloak-discovery, the StatefulSet keycloak, keycloak-initial-admin, and keycloak-network-policy (owner Keycloak, no Flux label). The repo declares none of these names. It only references keycloak-service by name (HTTPRoute, reconcilers). There are no PDBs in the namespace, and the Ingress is already enabled: false. The only NetworkPolicy in the namespace is the operator's. The repo has no Keycloak NetworkPolicy, and no Kyverno-generated default-deny is present there. NetworkPolicies add up, so the operator's allow-policy cannot fight another owner. Left at its default.
• Local gate: see the PR body. It records the rc and stages.

What I did NOT verify

• That the CR reaches Ready=True/HasErrors=False after merge. The status also says "Performing Keycloak upgrade, scaling down the deployment". I am inferring that this half clears once the workflow stops erroring. It is not proven.
• That the operator stops issuing the PATCH once enabled: false is applied. Watch the operator logs for 409.
• That platform-config and its downstream Kustomizations go Ready on the new revision, and that Prometheus keeps exactly one serviceMonitor/keycloak/keycloak/0 pool with targets up.
• The operator's actual label set for DEFAULT_LABELS_AS_STRING. I took it from the observed labels on operator-created objects, not from Constants.java.