fix(keycloak): one owner for the keycloak ServiceMonitor (26.8 operator 409 loop blocks Flux) #148

2026-11-08T06:30:57Z

gitea-admin commented

2026-11-08 06:30:57 +00:00

Incident

Keycloak 26.0.6 -> 26.8.2 (#121, #144). The operator and keycloak-0/1 are on 26.8.2 and Ready. But the Keycloak CR keycloak/keycloak reports Ready=False ("Waiting for more replicas / Performing Keycloak upgrade, scaling down the deployment") and HasErrors=True:

KeycloakServiceMonitorDependentResource -> PATCH .../monitoring.coreos.com/v1/namespaces/keycloak/servicemonitors/keycloak?fieldManager=keycloakcontroller&force=true ... 409 Conflict, then Exhausted retries.

platform-config (clusters/base/platform-config.yaml, wait: true, so it health-checks every object it applies, this CR included) is stuck in health checks. That blocks every Kustomization that dependsOn: platform-config: velero, apps-vms, apps-wave-0 (and so waves 1/2 and pm-agent), homepage, oauth2-proxy, headlamp, conformance, vpn-ingress, crafty, runbooks.

Cause

The 26.x operator creates its own ServiceMonitor, named after the CR (keycloak), whenever metrics-enabled=true and the CRD exists. The 26.8.2 CRD sets spec.serviceMonitor.enabled to default: true, in both v2alpha1 and v2beta1 (measured live). That is the same name and namespace as our Flux-owned platform/keycloak/servicemonitor.yaml, so the object has two owners.

Fix: one owner, ours

platform/keycloak/keycloak-instance.yaml gets spec.serviceMonitor.enabled: false. servicemonitor.yaml is unchanged except for a comment saying it is the sole owner.

Why ours and not the operator's:

These are written against ours: L0 servicemonitor-ports-resolve (allow-list entry platform/keycloak/servicemonitor.yaml:keycloak), the L2 keycloak-monitored chainsaw test, the README ownership table, and the dashboard's up{namespace="keycloak"}.
The operator's version (upstream KeycloakServiceMonitorDependentResource @26.8.2) scrapes the same target: port name management (9000 on keycloak-service), /metrics, 30s. It gives us nothing we lack.

Evidence (live, read-only, k3s-prod, 2026-11-08 ~06:25Z)

servicemonitor keycloak: managedFields has only kustomize-controller (Apply, 2026-09-05). No ownerReferences. Labels are kustomize.toolkit.fluxcd.io/name=platform-config and release=kube-prometheus-stack. The operator's apply never landed. It is the only ServiceMonitor in the namespace.
GC risk when disabled: none expected. The operator's dependent informer is filtered on Constants.DEFAULT_LABELS_AS_STRING (app.kubernetes.io/managed-by=keycloak-operator). Our object carries neither that label nor an ownerReference, so the operator cannot see it as a secondary and has nothing to delete. Flux keeps reconciling it as before. This comes from reading the code, not from observation; post-merge check 3 confirms it.

Other operator dependents (same collision class?): none

Operator object (owner `Keycloak`)	Repo / Kyverno also manages it?
`NetworkPolicy keycloak-network-policy` (`networkPolicy.enabled` default true)	No. It is the only NetworkPolicy in ns `keycloak`. The repo has no Keycloak NetworkPolicy and there is no generated default-deny there. NetworkPolicies add up, so there is nothing to fight over. Left at its default.
`Service keycloak-service`, `keycloak-discovery`	No. The repo only references `keycloak-service` (HTTPRoute, reconcilers).
`StatefulSet keycloak`, `Secret keycloak-initial-admin`	No
Ingress	already `ingress.enabled: false`
PDB	none exist in the namespace

Local gate (pinned toolbox, from the worktree, with the .git and spec mounts), commit `35ba875`

rc 3 (UNDETERMINED), not green. The only stage that did not run is gitea-settings (no token locally). Green: spec-immutability, approval-policy, decisions-log (80 entries, which includes this PR's new file; origin/main has 79), lint, l0, render-golden, l1. L2/L3 were not run (no cluster), so R-IDP-01..06 are not certified.

Post-merge checks (operator)

kubectl --context k3s-prod -n keycloak get keycloak keycloak -o jsonpath='{.status.conditions}' shows Ready=True, HasErrors=False. The "Performing Keycloak upgrade, scaling down" message also clears. Inferred, not proven: that half should clear once the workflow stops erroring.
flux get kustomizations: platform-config and every downstream Kustomization (velero, apps-vms, apps-wave-0/1/2, pm-agent, homepage, oauth2-proxy, headlamp, conformance, vpn-ingress, crafty, runbooks) are Ready, with lastAppliedRevision = the merge commit.
kubectl -n keycloak get servicemonitor lists exactly one keycloak. Its managedFields still have only kustomize-controller, and it has no ownerReference.
Prometheus target pool serviceMonitor/keycloak/keycloak/0 has 2 targets, both up.
kubectl -n keycloak logs deploy/keycloak-operator --since=10m | grep -c 409 returns 0.

If the operator does delete our ServiceMonitor (check 3), Flux recreates it on the next platform-config interval (10m). Report it, because that would contradict the code reading above.

🤖 Generated with Claude Code

## Incident Keycloak 26.0.6 -> 26.8.2 (#121, #144). The operator and keycloak-0/1 are on 26.8.2 and Ready. But the `Keycloak` CR `keycloak/keycloak` reports `Ready=False` ("Waiting for more replicas / Performing Keycloak upgrade, scaling down the deployment") and `HasErrors=True`: `KeycloakServiceMonitorDependentResource -> PATCH .../monitoring.coreos.com/v1/namespaces/keycloak/servicemonitors/keycloak?fieldManager=keycloakcontroller&force=true ... 409 Conflict`, then `Exhausted retries`. `platform-config` (`clusters/base/platform-config.yaml`, **`wait: true`**, so it health-checks every object it applies, this CR included) is stuck in health checks. That blocks every Kustomization that `dependsOn: platform-config`: **velero, apps-vms, apps-wave-0 (and so waves 1/2 and pm-agent), homepage, oauth2-proxy, headlamp, conformance, vpn-ingress, crafty, runbooks**. ## Cause The 26.x operator creates its own ServiceMonitor, named after the CR (`keycloak`), whenever `metrics-enabled=true` and the CRD exists. The 26.8.2 CRD sets `spec.serviceMonitor.enabled` to **`default: true`**, in both v2alpha1 and v2beta1 (measured live). That is the same name and namespace as our Flux-owned `platform/keycloak/servicemonitor.yaml`, so the object has two owners. ## Fix: one owner, ours `platform/keycloak/keycloak-instance.yaml` gets `spec.serviceMonitor.enabled: false`. `servicemonitor.yaml` is unchanged except for a comment saying it is the sole owner. Why ours and not the operator's: - These are written against ours: L0 `servicemonitor-ports-resolve` (allow-list entry `platform/keycloak/servicemonitor.yaml:keycloak`), the L2 `keycloak-monitored` chainsaw test, the README ownership table, and the dashboard's `up{namespace="keycloak"}`. - The operator's version (upstream `KeycloakServiceMonitorDependentResource` @26.8.2) scrapes the **same target**: port name `management` (9000 on `keycloak-service`), `/metrics`, 30s. It gives us nothing we lack. ## Evidence (live, read-only, k3s-prod, 2026-11-08 ~06:25Z) - `servicemonitor keycloak`: managedFields has only **`kustomize-controller`** (Apply, 2026-09-05). **No ownerReferences.** Labels are `kustomize.toolkit.fluxcd.io/name=platform-config` and `release=kube-prometheus-stack`. The operator's apply never landed. It is the only ServiceMonitor in the namespace. - **GC risk when disabled: none expected.** The operator's dependent informer is filtered on `Constants.DEFAULT_LABELS_AS_STRING` (`app.kubernetes.io/managed-by=keycloak-operator`). Our object carries neither that label nor an ownerReference, so the operator cannot see it as a secondary and has nothing to delete. Flux keeps reconciling it as before. *This comes from reading the code, not from observation; post-merge check 3 confirms it.* ## Other operator dependents (same collision class?): none | Operator object (owner `Keycloak`) | Repo / Kyverno also manages it? | |---|---| | `NetworkPolicy keycloak-network-policy` (`networkPolicy.enabled` default true) | No. It is the only NetworkPolicy in ns `keycloak`. The repo has no Keycloak NetworkPolicy and there is no generated default-deny there. NetworkPolicies add up, so there is nothing to fight over. Left at its default. | | `Service keycloak-service`, `keycloak-discovery` | No. The repo only *references* `keycloak-service` (HTTPRoute, reconcilers). | | `StatefulSet keycloak`, `Secret keycloak-initial-admin` | No | | Ingress | already `ingress.enabled: false` | | PDB | none exist in the namespace | ## Local gate (pinned toolbox, from the worktree, with the .git and spec mounts), commit 35ba875 **rc 3 (UNDETERMINED), not green.** The only stage that did not run is `gitea-settings` (no token locally). Green: spec-immutability, approval-policy, decisions-log (80 entries, which includes this PR's new file; origin/main has 79), lint, l0, render-golden, l1. L2/L3 were not run (no cluster), so R-IDP-01..06 are not certified. ## Post-merge checks (operator) 1. `kubectl --context k3s-prod -n keycloak get keycloak keycloak -o jsonpath='{.status.conditions}'` shows **Ready=True, HasErrors=False**. The "Performing Keycloak upgrade, scaling down" message also clears. *Inferred, not proven: that half should clear once the workflow stops erroring.* 2. `flux get kustomizations`: `platform-config` and every downstream Kustomization (velero, apps-vms, apps-wave-0/1/2, pm-agent, homepage, oauth2-proxy, headlamp, conformance, vpn-ingress, crafty, runbooks) are Ready, with `lastAppliedRevision` = the merge commit. 3. `kubectl -n keycloak get servicemonitor` lists **exactly one** `keycloak`. Its managedFields still have only `kustomize-controller`, and it has no ownerReference. 4. Prometheus target pool `serviceMonitor/keycloak/keycloak/0` has 2 targets, both up. 5. `kubectl -n keycloak logs deploy/keycloak-operator --since=10m | grep -c 409` returns 0. If the operator *does* delete our ServiceMonitor (check 3), Flux recreates it on the next `platform-config` interval (10m). Report it, because that would contradict the code reading above. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea-admin added the needs-review blast-radius labels 2026-11-08 06:30:57 +00:00

gitea-admin added 1 commit 2026-11-08 06:30:57 +00:00

fix(keycloak): one owner for the keycloak ServiceMonitor (26.8 operator 409 loop blocks Flux)

gate / gate (pull_request) Successful in 3m2s

Details

gate / conformance (pull_request) Skipped

Details

35ba875380

The 26.x Keycloak operator creates its own ServiceMonitor named after the
CR (keycloak/keycloak) when metrics are enabled; spec.serviceMonitor.enabled
defaults true. That collides with the Flux-owned platform/keycloak/
servicemonitor.yaml: the operator's PATCH 409s, it exhausts retries, the CR
stays Ready=False/HasErrors=True, and platform-config (wait: true) stalls
every downstream Kustomization.

Keep the repo's ServiceMonitor (conformance L0/L2 and the dashboard are
written against it; the operator's scrapes the same port 9000 /metrics) and
set spec.serviceMonitor.enabled: false.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

jakman85 merged commit 9e9a3ea8cc into main

2026-11-08 06:37:13 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Home-Network/home-network-k3s-impl#148