---
# PodMonitor for the Flux controllers — the `gotk_*` metric family (R-OBS-04, docs/10).
#
# WHY THIS FILE EXISTS: platform/monitoring/alerts.yaml ships five rules that query `gotk_*` —
# FluxReconciliationFailure (critical), FluxResourceSuspended, ImageUpdateAutomationFailing,
# ImagePolicyStalled, UpdateHealthGateFailed — and NONE of them could ever fire.
# VERIFIED live on prod 2026-09-13:
#     count({__name__=~"gotk.*"})   => (empty)
#     count(gotk_reconcile_condition) => (empty)
#     count(gotk_suspend_status)      => (empty)
#     count by (job) (up)            => 19 jobs, none in flux-system
# kube-prometheus-stack does NOT scrape flux-system: the Flux controllers are bootstrapped by
# `flux bootstrap` (not by a HelmRelease with a `serviceMonitor` value), they expose no Service at
# all, and nothing in this repo declared a monitor for them. So the whole gotk_* family was absent
# from the TSDB. On a cluster whose ENTIRE delivery model is Flux, that means reconciliation
# stopping, a Kustomization failing, or a resource left `suspend: true` after a debugging session
# was invisible to alerting — the failure mode is silent drift between git and the cluster, which is
# the one thing R-GITOPS-01 promises cannot happen quietly.
#
# WHY A PodMonitor AND NOT A ServiceMonitor: the controllers have no Service. Each controller Pod
# exposes its Prometheus endpoint directly on a container port named `http-prom`. CONFIRMED against
# prod 2026-09-13 — all six controllers (helm-, source-, kustomize-, notification-,
# image-automation-, image-reflector-controller) carry `app: <controller-name>` and `http-prom=8080`:
#     kubectl -n flux-system get pods -o jsonpath='{...ports[*].name}={...containerPort}'
#       helm-controller-…              http-prom=8080 healthz=9440
#       image-automation-controller-…  http-prom=8080 healthz=9440
#       image-reflector-controller-…   http-prom=8080 healthz=9440
#       kustomize-controller-…         http-prom=8080 healthz=9440
#       notification-controller-…      http=9090 http-webhook=9292 http-prom=8080 healthz=9440
#       source-controller-…            http=9090 http-prom=8080 healthz=9440
# This is the shape upstream Flux documents for kube-prometheus-stack (the flux2-monitoring-example
# `podmonitor.yaml`), reproduced here rather than vendored so the selector list is reviewable and
# matches THIS cluster's controller set exactly.
#
# `app` (not app.kubernetes.io/name) is deliberate: that is the label `flux bootstrap` actually
# stamps on the controller Pods, as printed above. A matchLabels on app.kubernetes.io/name would
# select nothing and reproduce the defect this file fixes — a monitor that exists, discovers zero
# targets, and leaves the alerts inert (exactly how the apps/ SLI ServiceMonitors fail today: they
# name a `metrics` port their Service does not have).
#
# WHICH TIER: this is a CR of monitoring.coreos.com, whose CRDs are installed by the
# kube-prometheus-stack HelmRelease in ./platform. It is therefore applied by the `platform-config`
# Flux Kustomization (dependsOn: platform), NOT the controllers pass — identical wiring to
# platform/monitoring/alerts.yaml, platform/kubevirt/servicemonitor.yaml and
# platform/cnpg/alerts.yaml. The file lives here, beside the alerts.yaml rules that consume gotk_*.
#
# No labels are needed for selection: platform/monitoring/kube-prometheus-stack.yaml sets
# podMonitorSelectorNilUsesHelmValues: false, so Prometheus selects every PodMonitor in every
# namespace regardless of labels. `release: kube-prometheus-stack` is set anyway, matching the
# convention of the sibling monitoring CRs in this directory.
#
# Cost: 6 scrape targets, one gotk_* set per controller (reconcile_condition, suspend_status,
# reconcile_duration_seconds + the controller-runtime/Go families) — tens of series per controller.
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: flux-system
  namespace: flux-system
  labels:
    release: kube-prometheus-stack
    platform-health: "true"
spec:
  namespaceSelector:
    matchNames:
      - flux-system
  selector:
    matchExpressions:
      # The six controllers `flux bootstrap` installs on this cluster. An explicit list rather than
      # `matchLabels: {}`: flux-system also runs short-lived Jobs and (in future) the notification
      # receiver, and a blanket selector would add permanently-down targets — the noise the k3s
      # control-plane scrape trim in kube-prometheus-stack.yaml exists to avoid.
      - key: app
        operator: In
        values:
          - helm-controller
          - source-controller
          - kustomize-controller
          - notification-controller
          - image-automation-controller
          - image-reflector-controller
  podMetricsEndpoints:
    - port: http-prom
      relabelings:
        # Carry the node name onto every gotk_* series. Upstream's documented relabeling; it is what
        # lets a "reconciliation is failing" signal be correlated with a node-level fault (the
        # prod-1 disk/etcd-starvation pattern in DECISIONS-LOG), where the controller's own labels
        # say nothing about where it was running.
        #
        # Upstream's example spells this with `regex: ^(.*)$` / `replacement: $1`. Both are the
        # Prometheus DEFAULTS for action `replace`, and they are omitted here ON PURPOSE: this file
        # is rendered by a Flux Kustomization with postBuild.substituteFrom, and a bare `$1` in a
        # substituted path is the shape the escape contract in clusters/base/apps-vms.yaml exists to
        # reason about. envsubst does leave `$1` alone (it is not a braced expansion and does not
        # start with a name character), so it would be safe — but relying on that to carry a value
        # that is already the default is a needless hostage to the next toolchain bump. Same
        # behaviour, nothing to escape.
        - sourceLabels: [__meta_kubernetes_pod_node_name]
          targetLabel: node
          action: replace