--- # Flux Kustomization: kured (Kubernetes Reboot Daemon) — reconciles ./platform/kured. # Performs the cordon / drain / reboot / uncordon of a node that has a pending OS reboot, strictly # one node at a time, inside a nightly window, gated on Longhorn volume health. Design rationale, # every tuned value, and the honest list of what is NOT mitigated: platform/kured/helmrelease.yaml. # # WHY ITS OWN KUSTOMIZATION, AND NOT ONE LINE IN platform/kustomization.yaml # # The same argument the Cilium split-out recorded in platform/kustomization.yaml and intel-gpu-plugin # reused, except the consequence is worse here than for either of them: # # * `platform` is `wait: true` over twelve-plus components, so its Ready condition cannot tell you # "the reboot daemon is wedged" from "longhorn is slow". # * The only emergency valve would be `flux suspend kustomization platform` — freezing storage, # TLS, DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a DaemonSet. # * That tier's 10m interval / 5m timeout are called out there as WRONG for a 3-node DaemonSet # rollout. This is a 3-node DaemonSet rollout. # * And `platform` gates platform-config, which gates apps-wave-0 (wait:true), which gates # apps-wave-1. A reboot daemon that failed to go healthy inside that tier would stall EVERY # APPLICATION in the cluster on "dependency not ready". A component added to make maintenance # safer must not acquire the ability to cause an outage; that trade is the whole argument. # # dependsOn: [platform] — AND THIS ONE IS A REAL EDGE, NOT CONVENTION # # Not `cilium` (what intel-gpu-plugin uses) and not `platform-config` (what the leaf UIs use). It is # `platform`, for two independent reasons: # # 1. CRD ORDERING (F-MON1). platform/kured ships a PrometheusRule, and the chart generates a # ServiceMonitor. Both are CRs whose CRDs (monitoring.coreos.com) are installed by # kube-prometheus-stack INSIDE the `platform` tier. Flux dry-runs every object in a pass before # applying any of them, so a pass containing an unknown kind ABORTS — the failure that made the # CDI StorageProfiles a true deadlock on prod on 2026-09-05 (see # clusters/base/platform-config-cdi.yaml). Depending on `platform` guarantees the CRDs exist # before this pass is attempted. # 2. FUNCTIONAL PREREQUISITE. kured's reboot gate QUERIES Prometheus # (`prometheusUrl` in the HelmRelease). Prometheus lives in platform/monitoring. If kured ran # first it would get connection errors, treat the gate as unevaluable and skip every reboot — # the correct fail-safe direction, but silent. Longhorn, whose volume health IS the gate, is # also in that tier. # # It does NOT need platform-config: no Keycloak, no TLS, no split-horizon DNS, no Gateway, no # HTTPRoute. kured serves no HTTP to users. # # wait: true — A HEALTH SIGNAL, NOT A GATE. Following intel-gpu-plugin rather than the leaf-UI # template. A leaf UI is legitimately not-Ready for long stretches of a cold start while it waits on # Keycloak and DNS, so its health is uninformative; kured has no such dependency — it is a DaemonSet # that opens a socket and watches a file, so on healthy nodes it is Ready in seconds and not-Ready # genuinely means BROKEN. This turns `flux get kustomization kured` into a one-line answer to "is # reboot automation live on all three nodes", which is the question the operator must be able to # answer before trusting a kernel upgrade to it. # # NOTHING IN THE GRAPH dependsOn THIS, so wait:true gates nothing — the blast radius of an unhealthy # kured is this Kustomization's own Ready condition. Do not add `dependsOn: [kured]` anywhere: that # would hand a node-rebooting DaemonSet the power to stall whatever depends on it, which is precisely # what the placement above exists to prevent. # # prune: true is safe and deliberate. If this Kustomization is deleted, kured's DaemonSet goes with # it and reboots simply stop happening — nodes are left uncordoned and running. The one caveat is # that pruning mid-reboot would leave a node CORDONED with no daemon to uncordon it; that is a # `kubectl uncordon` away and is covered in runbooks/node-reboot-automation.md. apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: kured namespace: flux-system spec: # A HelmRelease plus five static manifests, nothing to re-render; kustomize-controller is # source-triggered, so this is only the drift-correction floor. interval: 10m retryInterval: 2m # 10m, not the 5m the `platform` tier uses: a 3-node DaemonSet rollout under `wait: true` can # exceed 5m on a cold start (image pull on three nodes), which is one of the reasons the Cilium # split-out gave for leaving that tier. timeout: 10m prune: true # See the header: a health SIGNAL for a component nothing depends on, not a gate. wait: true dependsOn: - name: platform # monitoring CRDs + Prometheus + Longhorn; see the header, this is load-bearing sourceRef: kind: GitRepository name: flux-system path: ./platform/kured # REQUIRED, not boilerplate. platform/kured/alerts.yaml renders `${BASE_DOMAIN}` into the # `runbook_url` annotation of all three rules. Without this block that token becomes the EMPTY # STRING and the Kustomization still reports Ready — Flux's substitution fails OPEN — so every # alert would ship a dead link to `https:///runbooks/node-reboot-automation`. The gate stage # `flux-postbuild-substitution` asserts the converse (a rendered token implies this block), and # BASE_DOMAIN is declared in clusters/base/cluster-vars.yaml. # # Nothing else in this path contains `${...}` or a bare `$`: the HelmRelease's alertFilterRegexp is # a plain anchored alternation, and there are no shell snippets anywhere in the component — which # is deliberate, because envsubst would eat a braced shell variable and `$$` would trip the # dollar-collapse assertion in the same check. postBuild: substituteFrom: - kind: ConfigMap name: cluster-vars