jakman85andClaude Opus 5
a79693d3bd
kube-prometheus-stack does not scrape flux-system: the controllers are
bootstrapped, expose no Service, and no chart value covers them. So every
gotk_* alert rule in platform/monitoring/alerts.yaml could never fire —
verified on prod, count({__name__=~"gotk.*"}) returned nothing. A repo whose
entire delivery model is Flux had no alerting on reconciliation failing, a
Kustomization erroring, or a resource left suspended.
Proven live before landing: with this PodMonitor applied, gotk_suspend_status
returns 39 series and gotk_reconcile_condition 117, and FluxResourceSuspended
actually fired on a deliberate test suspension — evidence the rule is live,
not merely present.
Extracted from the wider inert-alert audit on
fix/scrape-targets-for-inert-alerts, which stalled unfinished. This is the
self-contained, verified half; Longhorn and cert-manager metrics are still
ABSENT and their alerts remain inert. It also adopts the object into git: it
existed on prod only as kubectl-client-side-apply, so a rebuild would have
lost it.
Co-Authored-By: Claude Opus 5 <[email protected]>
128 lines
9.2 KiB
YAML
128 lines
9.2 KiB
YAML
---
|
|
# platform-config — the CONFIGS half of the controllers/configs split (flux2 idiomatic).
|
|
#
|
|
# WHY: the `platform` Flux Kustomization installs the CONTROLLERS (HelmReleases) that register CRDs
|
|
# (cert-manager.io, kyverno.io, …). The custom resources BELOW use those CRDs, so applying them in
|
|
# the SAME pass fails atomically ("no matches for kind ClusterIssuer") and blocks the whole platform
|
|
# apply. This Kustomization carries the CRs and is gated `dependsOn: platform` (wait) so the CRDs
|
|
# exist first. See clusters/base/platform-config.yaml + DECISIONS-LOG.
|
|
#
|
|
# Files are referenced IN PLACE via ../platform/... (Flux builds with LoadRestrictionsNone, and the
|
|
# Kyverno policies must stay under platform/kyverno/ for the CI L0 `kyverno test` gate). Build/verify
|
|
# locally with: kubectl kustomize --load-restrictor LoadRestrictionsNone platform-config
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
resources:
|
|
# cert-manager: DNS-01 issuers (need cert-manager.io CRDs from the cert-manager HelmRelease)
|
|
- ../platform/cert-manager/clusterissuers.yaml
|
|
# gateway: the Gateway + wildcard Certificate (Certificate needs cert-manager.io; Gateway needs the
|
|
# GatewayClass, which stays in platform/gateway as a controller resource)
|
|
- ../platform/gateway/gateway.yaml
|
|
- ../platform/gateway/wildcard-certificate.yaml
|
|
# kyverno: ClusterPolicies (need kyverno.io CRDs from the Kyverno HelmRelease)
|
|
- ../platform/kyverno/policies/require-default-deny-baseline.yaml
|
|
- ../platform/kyverno/policies/validate-default-deny-shape.yaml
|
|
- ../platform/kyverno/policies/restrict-host-namespaces.yaml
|
|
- ../platform/kyverno/policies/require-egress-class-label.yaml
|
|
- ../platform/kyverno/policies/require-auth-mode-label.yaml
|
|
# COLD-START FIX (2026-09-06, prod): a MUTATE policy that must travel WITH the two label
|
|
# policies above. CDI generates its own short-lived worker pods (importer-*, uploader-*,
|
|
# cloner-*) inside app namespaces; nothing in git can label them, so both policies DENIED them
|
|
# and `DataVolume unifi/unifi-rootdisk` sat in ImportScheduled with no disk. This policy adds
|
|
# the two labels to pods created by ServiceAccount cdi/cdi-sa, so the requirement is SATISFIED
|
|
# rather than bypassed. Applying it here (not later) matters: if it lands after the validating
|
|
# policies, the next import is blocked in the gap.
|
|
- ../platform/kyverno/policies/label-cdi-worker-pods.yaml
|
|
- ../platform/kyverno/admission/secret-provenance.yaml
|
|
# cilium: L2 LB IP pool + announcement policy (cilium.io CRDs already registered by Cilium)
|
|
- cilium/l2-ip-pool.yaml
|
|
- cilium/l2-announcement-policy.yaml
|
|
# longhorn/external-snapshotter: the driver.longhorn.io VolumeSnapshotClass (needs the
|
|
# snapshot.storage.k8s.io CRDs from platform/external-snapshotter). Enables CSI snapshots +
|
|
# CDI snapshot smart-clone.
|
|
- ../platform/longhorn/volumesnapshotclass.yaml
|
|
# monitoring: PrometheusRule alert rules (need monitoring.coreos.com CRDs from kube-prometheus-stack)
|
|
- ../platform/monitoring/alerts.yaml
|
|
# Cluster-wide PVC capacity rules. A SEPARATE FILE from alerts.yaml, whose only PVC-fill rule
|
|
# (MonitoringVolumeFillingUp) is scoped namespace="monitoring" — so before this, no database PVC
|
|
# was covered by any fill alert. Partitioned by namespace so the two never double-fire; see the
|
|
# file header. (2026-09-06, from the CNPG WAL/archiving incident.)
|
|
- ../platform/monitoring/pvc-capacity-alerts.yaml
|
|
# PodMonitor for the six Flux controllers — the ONLY thing that puts `gotk_*` in the TSDB.
|
|
# kube-prometheus-stack does not scrape flux-system (the controllers are bootstrapped, expose no
|
|
# Service, and no chart value covers them), so until this landed the five gotk_* rules in
|
|
# alerts.yaml above were structurally inert: verified on prod 2026-09-13,
|
|
# `count({__name__=~"gotk.*"})` returned nothing. A CR of monitoring.coreos.com -> this tier, same
|
|
# as alerts.yaml. (2026-09-13, from the inert-alert audit.)
|
|
- ../platform/monitoring/flux-podmonitor.yaml
|
|
# kubevirt/cdi (Phase 5): the KubeVirt + CDI CRs consume the kubevirt.io / cdi.kubevirt.io CRDs the
|
|
# operators register; the ServiceMonitors/alerts consume monitoring.coreos.com. All CRs of platform-
|
|
# installed CRDs -> applied here, gated dependsOn platform (F-MON1). The dashboard is a plain
|
|
# ConfigMap kept beside them for locality.
|
|
- ../platform/kubevirt/kubevirt-cr.yaml
|
|
- ../platform/kubevirt/cdi-cr.yaml
|
|
# *** DO NOT ADD ../platform/kubevirt/cdi-storageprofiles.yaml BACK HERE. IT DEADLOCKS THIS TIER. ***
|
|
# The CDI StorageProfiles were MOVED OUT on 2026-09-05 into their own Flux Kustomization
|
|
# (clusters/base/platform-config-cdi.yaml -> ./platform-config-cdi, dependsOn platform-config).
|
|
#
|
|
# `StorageProfile`'s CRD is NOT registered by the CDI operator INSTALL — cdi-operator.yaml ships only
|
|
# `cdis.cdi.kubevirt.io`. cdi-operator creates `storageprofiles` (and datavolumes, datasources, …)
|
|
# only AFTER it has reconciled the `CDI` CR two lines above. Flux server-side DRY-RUNS EVERY object
|
|
# in a Kustomization before applying ANY of them, so having both in this list meant: unknown kind
|
|
# `StorageProfile` -> the whole pass aborts -> the `CDI` CR is never applied -> the CRD is never
|
|
# created -> THIS TIER CAN NEVER CONVERGE, and every tier that dependsOn it is blocked forever.
|
|
# A true deadlock, not a race. Observed on prod 2026-09-05 (cdi-operator Available 12h, no CR):
|
|
# StorageProfile/longhorn dry-run failed: no matches for kind "StorageProfile" in version
|
|
# "cdi.kubevirt.io/v1beta1"
|
|
# Invisible on dev, whose CDI CRDs have existed for months. See DECISIONS-LOG 2026-09-05.
|
|
#
|
|
# THE RULE: a CR whose CRD is created by a controller AT RUNTIME (rather than by a manifest in git)
|
|
# is SECOND-ORDER and belongs in ./platform-config-cdi, not here.
|
|
- ../platform/kubevirt/servicemonitor.yaml
|
|
- ../platform/kubevirt/alerts.yaml
|
|
- ../platform/kubevirt/dashboard.yaml
|
|
# --- Phase 6 — Identity / SSO (ADR-0014) ---
|
|
# keycloak: the CNPG DB Cluster + Keycloak instance + realm import (CRs of the CNPG / Keycloak
|
|
# operator CRDs installed by platform/cnpg + platform/keycloak) + its ingress + ServiceMonitor.
|
|
# ${KEYCLOAK_REPLICAS} in keycloak-instance.yaml is resolved by this Kustomization's substituteFrom.
|
|
# cnpg: CloudNativePG platform-health alerts (WAL archiving / recoverability). PrometheusRule is a
|
|
# CR of monitoring.coreos.com, so it belongs to this pass, not the platform controllers pass —
|
|
# identical wiring to platform/kubevirt/alerts.yaml. Added 2026-09-06 after all three prod CNPG
|
|
# clusters ran 38h+ with ContinuousArchiving=False and firstRecoverabilityPoint=null, unalerted.
|
|
- ../platform/cnpg/alerts.yaml
|
|
- ../platform/keycloak/postgres-cluster.yaml
|
|
- ../platform/keycloak/keycloak-instance.yaml
|
|
- ../platform/keycloak/keycloak-realm-import.yaml
|
|
# The realm import is a ONE-SHOT that fails OPEN: an unresolved secret placeholder is
|
|
# stored LITERALLY and the import still reports success, then rests at Done=True and
|
|
# never re-runs. This CronJob is the level-triggered loop that corrects that drift
|
|
# (2026-09-06 incident; see the file header). It belongs in THIS tier, not ./platform,
|
|
# so it applies after the keycloak-realm-clients Secret (rendered by ./platform) and
|
|
# alongside the import whose failure mode it covers.
|
|
- ../platform/keycloak/client-secret-reconciler.yaml
|
|
# The DESIRED STATE for every environment-derived client URL (redirectUris, webOrigins,
|
|
# rootUrl, baseUrl, adminUrl), GENERATED from keycloak-realm-import.yaml by
|
|
# ci/render-keycloak-client-urls.py and gated by the L0 check. The reconciler pod has no
|
|
# Kubernetes API access (automountServiceAccountToken: false), so it cannot read the
|
|
# KeycloakRealmImport CR — its desired state must be MOUNTED, hence this ConfigMap.
|
|
#
|
|
# IT MUST BE RENDERED BY THIS SAME KUSTOMIZATION as the realm import above: both are then
|
|
# substituted in one pass by this Kustomization's postBuild.substituteFrom (cluster-vars),
|
|
# so a ${BASE_DOMAIN} token in either resolves to the same value and the two declarations
|
|
# cannot disagree about the domain.
|
|
- ../platform/keycloak/client-urls.yaml
|
|
- ../platform/keycloak/k8s-oidc-rbac.yaml # maps OIDC groups -> cluster roles (kubectl SSO RBAC)
|
|
- ../platform/keycloak/httproute.yaml
|
|
- ../platform/keycloak/servicemonitor.yaml
|
|
# oauth2-proxy: forward-auth httproute + ServiceMonitor MOVED OUT of platform-config. They now
|
|
# travel with the oauth2-proxy controller in its own post-platform Flux Kustomization
|
|
# (clusters/base/oauth2-proxy.yaml, dependsOn platform-config). Keeping them here — while the
|
|
# controller sat in the gating `platform` tier — was one half of the twice-proven cold-start
|
|
# deadlock (platform never Ready -> platform-config never reconciles -> these never apply).
|
|
# monitoring SSO proof: Grafana native-OIDC login ingress (Gateway API CR)
|
|
- ../platform/monitoring/grafana-httproute.yaml
|
|
# headlamp: httproute MOVED OUT of platform-config. It now travels with the headlamp component in
|
|
# its own post-platform Flux Kustomization (clusters/base/headlamp.yaml, dependsOn platform-config).
|
|
# Keeping it here — while the HelmRelease sat in the gating `platform` tier — meant a headlamp
|
|
# crashloop stalled platform-config (and everything below it). See DECISIONS-LOG "Headlamp gating stall".
|