jakman85andClaude Opus 5
51d8676f90
The realm import is a one-shot: once Done=True it never re-runs, so a redirect URI
changed in git never reaches the realm. Healing that needs a DESIRED STATE the
reconciler can read -- and the reconciler pod has no Kubernetes API access at all
(automountServiceAccountToken: false), so it cannot read the KeycloakRealmImport CR.
Its desired state must be mounted, hence a ConfigMap.
A hand-authored ConfigMap would be a second source of truth for every public
hostname in the realm, free to disagree with the import -- the same shape as the bug
being closed. So it is GENERATED from the import and the render is gated, the same
render==golden idiom catalog/render.py already uses for apps/.
URL strings are copied VERBATIM: a ${BASE_DOMAIN} token in the source appears
unchanged here, and both files are rendered by the SAME Flux Kustomization
(./platform-config), so both are substituted in one pass with one value. The
renderer never parses a hostname and never reads cluster-vars, so there is no
second place the domain can be decided.
Co-Authored-By: Claude Opus 5 <[email protected]>
121 lines
8.7 KiB
YAML
121 lines
8.7 KiB
YAML
---
|
|
# platform-config — the CONFIGS half of the controllers/configs split (flux2 idiomatic).
|
|
#
|
|
# WHY: the `platform` Flux Kustomization installs the CONTROLLERS (HelmReleases) that register CRDs
|
|
# (cert-manager.io, kyverno.io, …). The custom resources BELOW use those CRDs, so applying them in
|
|
# the SAME pass fails atomically ("no matches for kind ClusterIssuer") and blocks the whole platform
|
|
# apply. This Kustomization carries the CRs and is gated `dependsOn: platform` (wait) so the CRDs
|
|
# exist first. See clusters/base/platform-config.yaml + DECISIONS-LOG.
|
|
#
|
|
# Files are referenced IN PLACE via ../platform/... (Flux builds with LoadRestrictionsNone, and the
|
|
# Kyverno policies must stay under platform/kyverno/ for the CI L0 `kyverno test` gate). Build/verify
|
|
# locally with: kubectl kustomize --load-restrictor LoadRestrictionsNone platform-config
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
resources:
|
|
# cert-manager: DNS-01 issuers (need cert-manager.io CRDs from the cert-manager HelmRelease)
|
|
- ../platform/cert-manager/clusterissuers.yaml
|
|
# gateway: the Gateway + wildcard Certificate (Certificate needs cert-manager.io; Gateway needs the
|
|
# GatewayClass, which stays in platform/gateway as a controller resource)
|
|
- ../platform/gateway/gateway.yaml
|
|
- ../platform/gateway/wildcard-certificate.yaml
|
|
# kyverno: ClusterPolicies (need kyverno.io CRDs from the Kyverno HelmRelease)
|
|
- ../platform/kyverno/policies/require-default-deny-baseline.yaml
|
|
- ../platform/kyverno/policies/validate-default-deny-shape.yaml
|
|
- ../platform/kyverno/policies/restrict-host-namespaces.yaml
|
|
- ../platform/kyverno/policies/require-egress-class-label.yaml
|
|
- ../platform/kyverno/policies/require-auth-mode-label.yaml
|
|
# COLD-START FIX (2026-09-06, prod): a MUTATE policy that must travel WITH the two label
|
|
# policies above. CDI generates its own short-lived worker pods (importer-*, uploader-*,
|
|
# cloner-*) inside app namespaces; nothing in git can label them, so both policies DENIED them
|
|
# and `DataVolume unifi/unifi-rootdisk` sat in ImportScheduled with no disk. This policy adds
|
|
# the two labels to pods created by ServiceAccount cdi/cdi-sa, so the requirement is SATISFIED
|
|
# rather than bypassed. Applying it here (not later) matters: if it lands after the validating
|
|
# policies, the next import is blocked in the gap.
|
|
- ../platform/kyverno/policies/label-cdi-worker-pods.yaml
|
|
- ../platform/kyverno/admission/secret-provenance.yaml
|
|
# cilium: L2 LB IP pool + announcement policy (cilium.io CRDs already registered by Cilium)
|
|
- cilium/l2-ip-pool.yaml
|
|
- cilium/l2-announcement-policy.yaml
|
|
# longhorn/external-snapshotter: the driver.longhorn.io VolumeSnapshotClass (needs the
|
|
# snapshot.storage.k8s.io CRDs from platform/external-snapshotter). Enables CSI snapshots +
|
|
# CDI snapshot smart-clone.
|
|
- ../platform/longhorn/volumesnapshotclass.yaml
|
|
# monitoring: PrometheusRule alert rules (need monitoring.coreos.com CRDs from kube-prometheus-stack)
|
|
- ../platform/monitoring/alerts.yaml
|
|
# Cluster-wide PVC capacity rules. A SEPARATE FILE from alerts.yaml, whose only PVC-fill rule
|
|
# (MonitoringVolumeFillingUp) is scoped namespace="monitoring" — so before this, no database PVC
|
|
# was covered by any fill alert. Partitioned by namespace so the two never double-fire; see the
|
|
# file header. (2026-09-06, from the CNPG WAL/archiving incident.)
|
|
- ../platform/monitoring/pvc-capacity-alerts.yaml
|
|
# kubevirt/cdi (Phase 5): the KubeVirt + CDI CRs consume the kubevirt.io / cdi.kubevirt.io CRDs the
|
|
# operators register; the ServiceMonitors/alerts consume monitoring.coreos.com. All CRs of platform-
|
|
# installed CRDs -> applied here, gated dependsOn platform (F-MON1). The dashboard is a plain
|
|
# ConfigMap kept beside them for locality.
|
|
- ../platform/kubevirt/kubevirt-cr.yaml
|
|
- ../platform/kubevirt/cdi-cr.yaml
|
|
# *** DO NOT ADD ../platform/kubevirt/cdi-storageprofiles.yaml BACK HERE. IT DEADLOCKS THIS TIER. ***
|
|
# The CDI StorageProfiles were MOVED OUT on 2026-09-05 into their own Flux Kustomization
|
|
# (clusters/base/platform-config-cdi.yaml -> ./platform-config-cdi, dependsOn platform-config).
|
|
#
|
|
# `StorageProfile`'s CRD is NOT registered by the CDI operator INSTALL — cdi-operator.yaml ships only
|
|
# `cdis.cdi.kubevirt.io`. cdi-operator creates `storageprofiles` (and datavolumes, datasources, …)
|
|
# only AFTER it has reconciled the `CDI` CR two lines above. Flux server-side DRY-RUNS EVERY object
|
|
# in a Kustomization before applying ANY of them, so having both in this list meant: unknown kind
|
|
# `StorageProfile` -> the whole pass aborts -> the `CDI` CR is never applied -> the CRD is never
|
|
# created -> THIS TIER CAN NEVER CONVERGE, and every tier that dependsOn it is blocked forever.
|
|
# A true deadlock, not a race. Observed on prod 2026-09-05 (cdi-operator Available 12h, no CR):
|
|
# StorageProfile/longhorn dry-run failed: no matches for kind "StorageProfile" in version
|
|
# "cdi.kubevirt.io/v1beta1"
|
|
# Invisible on dev, whose CDI CRDs have existed for months. See DECISIONS-LOG 2026-09-05.
|
|
#
|
|
# THE RULE: a CR whose CRD is created by a controller AT RUNTIME (rather than by a manifest in git)
|
|
# is SECOND-ORDER and belongs in ./platform-config-cdi, not here.
|
|
- ../platform/kubevirt/servicemonitor.yaml
|
|
- ../platform/kubevirt/alerts.yaml
|
|
- ../platform/kubevirt/dashboard.yaml
|
|
# --- Phase 6 — Identity / SSO (ADR-0014) ---
|
|
# keycloak: the CNPG DB Cluster + Keycloak instance + realm import (CRs of the CNPG / Keycloak
|
|
# operator CRDs installed by platform/cnpg + platform/keycloak) + its ingress + ServiceMonitor.
|
|
# ${KEYCLOAK_REPLICAS} in keycloak-instance.yaml is resolved by this Kustomization's substituteFrom.
|
|
# cnpg: CloudNativePG platform-health alerts (WAL archiving / recoverability). PrometheusRule is a
|
|
# CR of monitoring.coreos.com, so it belongs to this pass, not the platform controllers pass —
|
|
# identical wiring to platform/kubevirt/alerts.yaml. Added 2026-09-06 after all three prod CNPG
|
|
# clusters ran 38h+ with ContinuousArchiving=False and firstRecoverabilityPoint=null, unalerted.
|
|
- ../platform/cnpg/alerts.yaml
|
|
- ../platform/keycloak/postgres-cluster.yaml
|
|
- ../platform/keycloak/keycloak-instance.yaml
|
|
- ../platform/keycloak/keycloak-realm-import.yaml
|
|
# The realm import is a ONE-SHOT that fails OPEN: an unresolved secret placeholder is
|
|
# stored LITERALLY and the import still reports success, then rests at Done=True and
|
|
# never re-runs. This CronJob is the level-triggered loop that corrects that drift
|
|
# (2026-09-06 incident; see the file header). It belongs in THIS tier, not ./platform,
|
|
# so it applies after the keycloak-realm-clients Secret (rendered by ./platform) and
|
|
# alongside the import whose failure mode it covers.
|
|
- ../platform/keycloak/client-secret-reconciler.yaml
|
|
# The DESIRED STATE for every environment-derived client URL (redirectUris, webOrigins,
|
|
# rootUrl, baseUrl, adminUrl), GENERATED from keycloak-realm-import.yaml by
|
|
# ci/render-keycloak-client-urls.py and gated by the L0 check. The reconciler pod has no
|
|
# Kubernetes API access (automountServiceAccountToken: false), so it cannot read the
|
|
# KeycloakRealmImport CR — its desired state must be MOUNTED, hence this ConfigMap.
|
|
#
|
|
# IT MUST BE RENDERED BY THIS SAME KUSTOMIZATION as the realm import above: both are then
|
|
# substituted in one pass by this Kustomization's postBuild.substituteFrom (cluster-vars),
|
|
# so a ${BASE_DOMAIN} token in either resolves to the same value and the two declarations
|
|
# cannot disagree about the domain.
|
|
- ../platform/keycloak/client-urls.yaml
|
|
- ../platform/keycloak/k8s-oidc-rbac.yaml # maps OIDC groups -> cluster roles (kubectl SSO RBAC)
|
|
- ../platform/keycloak/httproute.yaml
|
|
- ../platform/keycloak/servicemonitor.yaml
|
|
# oauth2-proxy: forward-auth httproute + ServiceMonitor MOVED OUT of platform-config. They now
|
|
# travel with the oauth2-proxy controller in its own post-platform Flux Kustomization
|
|
# (clusters/base/oauth2-proxy.yaml, dependsOn platform-config). Keeping them here — while the
|
|
# controller sat in the gating `platform` tier — was one half of the twice-proven cold-start
|
|
# deadlock (platform never Ready -> platform-config never reconciles -> these never apply).
|
|
# monitoring SSO proof: Grafana native-OIDC login ingress (Gateway API CR)
|
|
- ../platform/monitoring/grafana-httproute.yaml
|
|
# headlamp: httproute MOVED OUT of platform-config. It now travels with the headlamp component in
|
|
# its own post-platform Flux Kustomization (clusters/base/headlamp.yaml, dependsOn platform-config).
|
|
# Keeping it here — while the HelmRelease sat in the gating `platform` tier — meant a headlamp
|
|
# crashloop stalled platform-config (and everything below it). See DECISIONS-LOG "Headlamp gating stall".
|