diff --git a/apps/wave-1/keycloak.yaml b/apps/wave-1/keycloak.yaml index a21e8ff..29a38d7 100644 --- a/apps/wave-1/keycloak.yaml +++ b/apps/wave-1/keycloak.yaml @@ -75,7 +75,7 @@ spec: spec: containers: - name: keycloak - image: quay.io/keycloak/keycloak + image: quay.io/keycloak/keycloak:26.0.6 ports: - containerPort: 8080 volumeMounts: diff --git a/catalog/keycloak.yaml b/catalog/keycloak.yaml index 30b5dd5..06dcc08 100644 --- a/catalog/keycloak.yaml +++ b/catalog/keycloak.yaml @@ -5,8 +5,15 @@ # Validates against schemas/servicedefinition.schema.json. name: keycloak kind: container +# DOCUMENTATION-ONLY IMAGE. The running server is NOT chosen here: the Keycloak operator +# (platform/keycloak/kustomization.yaml, three pinned release URLs) picks it via its +# RELATED_IMAGE_KEYCLOAK, and the Keycloak CR has no spec.image. This SD's render, +# apps/wave-1/keycloak.yaml, is not listed in apps/wave-1/kustomization.yaml and deploys nothing +# (ci/render-golden.sh RENDER_GOLDEN_NOT_ONBOARDED). The tag records the operator's version for +# readers; renovate.json5 disables Renovate for this image so it cannot raise PRs against a file +# nothing deploys. The Keycloak version moves with the operator — update this tag by hand then. source: - image: quay.io/keycloak/keycloak + image: quay.io/keycloak/keycloak:26.0.6 exposure: public # login/redirect flows for internet-exposed services; hardened (docs/15; R-DNS/R-NET) egress: internet # normal WAN egress (docs/03; R-NET-05) storage: diff --git a/decisions/2026-10-30-renovate-track-keycloak-operator.md b/decisions/2026-10-30-renovate-track-keycloak-operator.md new file mode 100644 index 0000000..822ea18 --- /dev/null +++ b/decisions/2026-10-30-renovate-track-keycloak-operator.md @@ -0,0 +1,113 @@ +--- +date: 2026-10-30 +branch: feat/renovate-track-keycloak-operator +title: Renovate tracks the Keycloak operator (the real Keycloak version), never auto-merged; the catalog's phantom image is pinned as documentation and disabled +--- + +## What changed + +- **`platform/keycloak/kustomization.yaml`**: each of the three `keycloak-k8s-resources/26.0.6` + release URLs (two CRDs and `kubernetes.yml`) now has a + `# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources` marker on the line + above it. That is the format the existing `custom.regex` marker manager parses (the kubevirt + pattern). A comment explains that these URLs are the Keycloak version. The `Keycloak` CR has no + `spec.image`, so the operator's `RELATED_IMAGE_KEYCLOAK` picks the server image. +- **`renovate.json5`** has two new packageRules: + - **`keycloak/keycloak-k8s-resources`**: `groupName: keycloak-operator`, `automerge: false`, + `addLabels: [needs-review, blast-radius]`. A comment explains why. The operator, the CRDs and + the server are released in lockstep. The IdP gates SSO for every app and for kubectl. And + `passkey-reconciler.yaml` encodes Keycloak 26.0.6 Admin API shapes that the static gate never + exercises. Majors still go through the existing dashboard-approval rule. + - **`quay.io/keycloak/keycloak`**: `enabled: false`. Its only occurrences are in files that + deploy nothing. + + The "STILL CANNOT FIX" comment also gains a one-line correction for keycloak. +- **`catalog/keycloak.yaml`**: `source.image` changed from `quay.io/keycloak/keycloak` (no tag) to + `quay.io/keycloak/keycloak:26.0.6`. A comment says the image is DOCUMENTATION-ONLY and that the + operator chooses the real one. `apps/wave-1/keycloak.yaml` was re-rendered, which changes one + line. +- **`platform/keycloak/README.md`**: + - Corrects the false claim "Renovate maintains the version". Until now Renovate could not see + the version. + - Adds a new section, **"Upgrading Keycloak: the operator IS the version"**. It lists the 26.0.6 + assumptions in the passkey reconciler to diff against a new tag, and the post-merge checks. + +**The phantom: pin and disable, not delete.** `apps/wave-1/keycloak.yaml` is not listed in +`apps/wave-1/kustomization.yaml`, and render-golden excludes it (`RENDER_GOLDEN_NOT_ONBOARDED`). +Deleting it would still not be a no-op: + +- It is the fixture for the keycloak cases in `apps/wave-1/tests/app-slis.test.yaml` (the + StatefulSet-based `keycloak-down` rule). Those cases are discovered by the + every-PrometheusRule-in-apps/wave-1 convention. +- Removing it means moving that rule and its tests to `platform/keycloak`. That is a separate + observability change, not a Renovate one. + +Pinning plus `enabled: false` changes one line in a file nothing applies. It cannot affect the +cluster, and render-golden stays green. + +## What I verified + +- **Live state, prod, read-only kubectl, 2026-10-30.** + - Keycloak CR `keycloak` has an empty `spec.image`. + - The operator Deployment runs `quay.io/keycloak/keycloak-operator:26.0.6`, imageID + `sha256:8e3a1a56346f…`, with `RELATED_IMAGE_KEYCLOAK=quay.io/keycloak/keycloak:26.0.6`. + - `keycloak-0` and `keycloak-1` both run `quay.io/keycloak/keycloak:26.0.6`, imageID + `sha256:a93d22e13b86…`. + - There is no PrometheusRule containing "keycloak" in any namespace. This confirms the phantom + deploys nothing, including its `keycloak-down` rule (see the next section). +- **`renovate-config-validator`** on the pinned `ghcr.io/renovatebot/renovate:44.167.0`: + `Config validated successfully`. The only migration note is the pre-existing `docker` block. +- **Local dry run** (`RENOVATE_PLATFORM=local`, `RENOVATE_DRY_RUN=full`, debug, the same pinned + image, this worktree, with a GitHub token passed by environment variable; the token string + occurs 0 times in the log): + - The regex manager went to 21 files / 24 deps. + - `keycloak/keycloak-k8s-resources` was extracted three times from + `platform/keycloak/kustomization.yaml`, each at `currentValue 26.0.6`, datasource + `github-tags`, versioning `semver`. + - Each extraction has one update, `26.8.1`, updateType `minor`, and all three share **one** + `branchName: renovate/keycloak-operator`. + - `quay.io/keycloak/keycloak` was extracted from `catalog/keycloak.yaml` (regex) and from + `apps/wave-1/keycloak.yaml` (kubernetes) at 26.0.6, with `skipReason: disabled` and no + updates. +- **Rule effects.** `--platform=local` does not print each branch's merged flags. So I ran + Renovate's own `applyPackageRules` (from `dist/util/package-rules` in the same 44.167.0 image) + over this `renovate.json5` for each case: + + | Update | Result | + |---|---| + | operator minor 26.0.6 → 26.8.1 | `automerge:false`, `groupName:keycloak-operator`, labels `dependencies, automated-update, needs-review, blast-radius` | + | operator patch | same as the minor | + | operator major | `automerge:false`, `dependencyDashboardApproval:true`, labels add `breaking` | + | `quay.io/keycloak/keycloak`, catalog and rendered | `enabled:false` | + | control: sonarr patch | `automerge:true`, so the rule did not disable auto-merge globally | +- **render-golden, host:** `render.py --selftest` passed, and `all 10 onboarded service golden(s) + reproduce`. Before the pin, the committed phantom was byte-identical to the canonical render. I + re-rendered it after the pin. +- **Toolbox gate** (commit 24db269, worktree with the ENGINE shim mounting `.git` and the spec + repo, `DEV_KUBECONFIG=/dev/null`): **rc 3**. + - The only stage that did not run is **`gitea-settings`**. + - spec-immutability, decisions-log, lint, l0, render-golden, l1 and approval-policy all passed. + - L2/L3 `conformance` scaffold-skipped, as expected. + - **Not green.** +- **Sensitive paths:** none touched (checked against `infra/gitea/approval-policy.json`), so no + `/approve-override` is needed. + +## What I did NOT verify + +- **A real Renovate run.** No PR was opened by Renovate. The labels and `automerge:false` are + proven by Renovate's own matcher, not by an opened PR. +- **That 26.8.1 is safe to adopt.** I did not diff 26.0.6 against 26.8.1. That includes the + reconciler's assumptions: `recurseExecutions` displayName, `updateExecutions`, the + `RpEntityName` guard, `requiredActions` REPLACE, and config aliases. The README section lists + them for whoever reviews the first bump PR. It also says to take or confirm a fresh CNPG backup + first, because a server minor may migrate the schema. +- **FINDING, not fixed: Keycloak has no deployed "IdP down" alert.** The `keycloak-down` SLI rule + exists only inside the undeployed `apps/wave-1/keycloak.yaml`. promtool tests it, but it is not + on the cluster: there is no keycloak PrometheusRule live. Separately, because that file exists, + `render.py --emit-waves` would treat keycloak as onboarded. It would then add a Deployment named + `keycloak` beside the operator's StatefulSet (the known `--emit-waves` hazard). The proper fix is + a separate PR: move the rule and its tests into `platform/keycloak`, then delete the phantom. +- **The catalog tag can drift.** Nothing ties `catalog/keycloak.yaml`'s documentation tag to the + operator URLs. It must be updated by hand after an operator bump, and the README says so. +- **Concurrent branches** `fix/renovate-hold-auth-and-self` and `feat/pin-pihole-jackett-plex` + also edit `renovate.json5`. Neither had merged when I checked. A rebase may be needed. diff --git a/platform/keycloak/README.md b/platform/keycloak/README.md index 8b6c6c9..032fa98 100644 --- a/platform/keycloak/README.md +++ b/platform/keycloak/README.md @@ -19,7 +19,9 @@ Docker Hub images were purged (DECISIONS-LOG **F-KY1**) → that path ImagePullB 3-replica **`longhorn`** StorageClass (DR-critical, R-IDP-05), with declarative backup hooks. quay.io + ghcr.io are unaffected by the Bitnami purge. Operator manifests are pinned upstream release -files pulled as remote kustomize bases (the kubevirt-stub pattern); Renovate maintains the version. +files pulled as remote kustomize bases (the kubevirt-stub pattern). Renovate tracks the version via +`# renovate:` markers on those URLs (since 2026-10-30 — before that it could not see them) and never +auto-merges it; see [Upgrading Keycloak](#upgrading-keycloak-the-operator-is-the-version). ## Layout (controllers / configs split — F-MON1) @@ -360,6 +362,38 @@ it on the BOUND flow, after go-live, in order: A failed passkey ceremony is usually silent in the server log — the browser refuses before Keycloak is involved — so step 2 failing with no log entry is the expected signature of a wrong RP ID. +## Upgrading Keycloak: the operator IS the version + +The `Keycloak` CR has no `spec.image`, so the server image is whatever the operator's +`RELATED_IMAGE_KEYCLOAK` names — the same tag as the operator (prod, 2026-10-30: operator +`26.0.6`, `keycloak-0/1` on `quay.io/keycloak/keycloak:26.0.6`). Operator, CRDs and server are +released in lockstep from `keycloak/keycloak-k8s-resources`, and the three URLs in +`kustomization.yaml` are the one knob. Renovate raises them as ONE PR (`keycloak-operator` group, +labels `needs-review` + `blast-radius`), **never auto-merged**; majors wait for a dashboard tick. +`catalog/keycloak.yaml`'s image tag is documentation only (its render deploys nothing) and +Renovate is disabled for it — update it by hand to match after an operator bump merges. + +**An operator bump requires re-checking the passkey reconciler's assumptions.** +`passkey-reconciler.yaml` (and `ci/render-keycloak-passkey-policy.py`, which generates its desired +state) is written against **Keycloak 26.0.6 Admin API behaviour read from that tag's source** — +grep it for `26.0.6`. The gate cannot catch a break: it is static and never runs the reconciler +against a server. Before merging a bump, diff the new tag against 26.0.6 for at least: + +- `AuthenticationManagementResource.recurseExecutions` — does a sub-flow row still carry its name + in `displayName` (not `alias`)? The reconciler's flow matching depends on it. +- `updateExecutions` / execution-requirement updates — same request shape and id resolution? +- the WebAuthn Passwordless policy — is it still gated all-or-nothing on a non-empty + `webAuthnPolicyPasswordlessRpEntityName`, and does a realm `PUT` still REPLACE rather than merge + `requiredActions`? (The strict read-modify-write exists because of both.) +- authenticator-config alias handling for copied flows. + +Then, after the merge reconciles (operator actions): watch the `Keycloak` CR reach Ready and both +pods roll; run the reconciler once **with `--audit`** (see "Passkeys" above for the one-off Job) +and confirm it reports convergence with no write; confirm the client-secret and client-URL +reconcilers' next runs succeed; log in to one SSO app and run one `kubectl --context k3s-prod` +OIDC command. Roll back by reverting the PR — but a server minor may migrate the database +schema on start, so take/confirm a fresh CNPG backup before merging. + ## Exceptions that keep native auth (do not force SSO) — ADR-0014 Plex/Seerr (Plex auth), Minecraft (Mojang/Microsoft), UniFi (native). diff --git a/platform/keycloak/kustomization.yaml b/platform/keycloak/kustomization.yaml index de861be..8b58b52 100644 --- a/platform/keycloak/kustomization.yaml +++ b/platform/keycloak/kustomization.yaml @@ -21,8 +21,17 @@ namespace: keycloak resources: - namespace.yaml # --- Keycloak Operator v26.0.6 (pinned upstream release manifests) --- + # The operator, its CRDs and the SERVER IMAGE move in lockstep: the Keycloak CR has no + # spec.image, so the operator's RELATED_IMAGE_KEYCLOAK (quay.io/keycloak/keycloak:) + # is what runs. These three URLs are therefore THE Keycloak version. Each carries a Renovate + # marker (custom.regex manager, renovate.json5) under ONE depName, so all three bump in ONE + # branch, never auto-merged (packageRule "keycloak-operator"). Before merging a bump, re-check + # passkey-reconciler.yaml's 26.0.6 Admin API assumptions — see README.md "Upgrading Keycloak". + # renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.0.6/kubernetes/keycloaks.k8s.keycloak.org-v1.yml + # renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.0.6/kubernetes/keycloakrealmimports.k8s.keycloak.org-v1.yml + # renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.0.6/kubernetes/kubernetes.yml # --- Admin (break-glass) credential — SOPS-encrypted, referenced by name (ADR-0007, R-IDP-04) --- - keycloak-admin-credentials.sops.yaml diff --git a/renovate.json5 b/renovate.json5 index 08c9d2d..712d6d0 100644 --- a/renovate.json5 +++ b/renovate.json5 @@ -260,6 +260,35 @@ "blast-radius" ] }, + // ADDED 2026-10-30 — THE KEYCLOAK OPERATOR IS THE KEYCLOAK VERSION. The operator, its CRDs + // and the server image are released in lockstep from keycloak/keycloak-k8s-resources, and the + // Keycloak CR carries no spec.image, so the operator's RELATED_IMAGE_KEYCLOAK picks the server + // (measured on prod 2026-10-30: operator 26.0.6 -> keycloak-0/1 run quay.io/keycloak/keycloak: + // 26.0.6). The three release URLs in platform/keycloak/kustomization.yaml carry `# renovate:` + // markers under this ONE depName, so they move together in one branch (groupName makes that + // explicit). NEVER AUTO-MERGED, not even a patch: Keycloak is the IdP that gates SSO for every + // app AND kubectl OIDC, and platform/keycloak/passkey-reconciler.yaml encodes Keycloak 26.0.6 + // Admin API shapes read from that tag's source (sub-flow rows named by displayName, the + // RpEntityName all-or-nothing guard, requiredActions REPLACE-not-merge, ...). The gate is + // static and does not run that reconciler against a server, so a green check says nothing + // about whether a bump breaks it. Majors still wait on the dashboard (rule at the bottom). + { + "description": "Keycloak operator + CRDs + server (lockstep, one branch). IdP blast radius; passkey-reconciler.yaml encodes 26.0.6 Admin API shapes — re-check it against the new tag before merging. Never auto-merge.", + "matchDepNames": ["keycloak/keycloak-k8s-resources"], + "groupName": "keycloak-operator", + "automerge": false, + "addLabels": ["needs-review", "blast-radius"] + }, + // ADDED 2026-10-30 — quay.io/keycloak/keycloak IS NOT A DEPENDENCY RENOVATE MAY MOVE. Its only + // occurrences are catalog/keycloak.yaml `source.image` and the apps/wave-1/keycloak.yaml it + // renders, which is NOT listed in apps/wave-1/kustomization.yaml and deploys nothing (the + // operator above runs the server). A PR bumping it would change no running pod while looking + // like a Keycloak upgrade, and could drift from the operator's real version. + { + "description": "quay.io/keycloak/keycloak appears only in the documentation-only catalog/keycloak.yaml and its undeployed render; the operator (keycloak/keycloak-k8s-resources) chooses the real server image.", + "matchPackageNames": ["quay.io/keycloak/keycloak"], + "enabled": false + }, { "description": "ADDED 2026-10-29. This repo's own images are followed ONLY through the `# renovate-digest:` markers above. The `kubernetes` manager also sees platform/pm-agent/deployment.yaml and infra/agent/worker-job-template.yaml, where the image has a digest and no tag; left on, it would look those up as `latest` (a tag CI never pushes) and, for the template, re-open the self-referential loop the custom manager deliberately avoids.", "matchManagers": ["kubernetes"], @@ -334,6 +363,8 @@ // version before any automation can maintain them, and until then they drift on every pull with // nothing recording what is running. CORRECTED 2026-10-30: nextcloud:31-apache and docker:27-dind // ARE maintained (the first dry run proposed 35-apache and 29-dind; the suffix is kept). + // CORRECTED 2026-10-30: quay.io/keycloak/keycloak was never a running reference — the operator + // picks the server image — and the Keycloak version is now tracked via the operator URLs. // Separately, the HelmRelease minor-locking ranges (9 of 12, e.g. `>=88.0.0 <89.0.0` on // kube-prometheus-stack): THIS FILE USED TO SAY they cap Renovate at patches. They do not — // Renovate WIDENS the range (runs 353/354). The `range-widen` rule above keeps that a reviewed