jakman85andClaude Opus 5
a2440a3125
Two halves of one repeatable upgrade process. Neither works alone. (a) OS upgrades and reboots platform/kured/ — Kubernetes Reboot Daemon, chart 6.1.0/app 1.23.0, in its own Flux Kustomization (dependsOn platform, wait:true). NOT in platform/kustomization.yaml: that tier gates platform-config and every apps wave, so a wedged reboot daemon there could stall every application. A component added to make maintenance safer must not be able to cause an outage. infra/ansible/roles/unattended_upgrades/ — new role (node_baseline is touched by other open branches, and this is a different lifecycle), enabled for prod in group_vars. The human gate is the SENTINEL, which is how this honours node_baseline's "IT DOES NOT REBOOT" (k3s-prod-2 hung on a warm reboot 2026-09-03) and cluster-upgrade-plan.md's rejection of automating away the gate between nodes: u-u is security-only with the kernel blacklisted, so it essentially never writes /var/run/reboot-required. That file appears when an operator deliberately runs node-baseline.yml --tags kernel. kured then does only the mechanical half, including the ordering rule node_baseline has to shout. Quorum: concurrent reboots are structurally impossible, not merely unlikely. concurrency:1 plus kured's single lock annotation (API-server optimistic concurrency), and lockTtl:0 — an expiring lock is exactly the 3-node etcd catastrophe, so a hung node holds it forever and nothing cascades. Longhorn: measured 25 volumes x 3 replicas, hard anti-affinity, 600s replenishment wait, serialized rebuild — so every drain degrades ALL volumes to 2/3 and the inter-node gate must be volume robustness, not pod readiness. Doubled: lockReleaseDelay 30m (unconditional) plus kured's Prometheus alert gate (semantic). alertFilterMatchOnly:true is load-bearing — the flag inverts the regexp's meaning, and under the chart default kured would have been blocked on day one by the five alerts measured firing with no end in sight, and never rebooted anything. Kernel floor (kernel >= 6.13, EFI memattr): excluded TWICE. The running kernel comes from o=Debian Backports, which is not an allowed origin, so u-u cannot see a kernel upgrade; and ^linux- is blacklisted anyway so widening origins later cannot re-arm it. Asserted against apt-config dump, not against the file just written. 6.12 kernels are still installed and bootable on all three nodes, and nothing was held. Closes a silent no-op: update-notifier-common was absent on all three nodes, so NOTHING could create the sentinel. kured alone would have idled forever at 1/1 Running. platform/kured/alerts.yaml — three rules over kured's only metric, with a companion promtool suite pinning each both ways, plus runbooks/node-reboot-automation.md. Window is DAILY 03:00-06:00 because a weekly one forces for:8d, longer than this cluster's 7d Prometheus retention. (b) Image/dependency automation Renovate was aspirational: renovate.json present and complete, with no workflow, no in-cluster workload, and its docker+kubernetes managers disabled — so every plain image: tag was maintained by nothing while seven files claimed otherwise. .gitea/workflows/renovate.yaml — scheduled daily, dry_run default true for a safe first run, and a preflight that FAILS (not skips) without RENOVATE_TOKEN, because a skipped Gitea job reports success and that is how this stayed inert. Renovate now owns both halves; Flux image-automation is formally closed: it pushes commits rather than opening PRs (bypassing CLAUDE.md §1), and it cannot work here anyway — main is protected user_can_push:false and the flux-system GitRepository has no secretRef. Its controllers are installed and have idled 43 days with zero CRs. conformance/: +34/-0, one new SCRAPE_OWNERS entry claiming kured_. Weakens nothing; tightens the check for this component (requires service.create+metrics.create, both chart-default false, plus a companion NetworkPolicy requirement). Required because assertion (3) fails an alert over an unclaimed metric and KNOWN_UNWIRED_METRICS is shrink-only. Sensitive paths touched: clusters/**, conformance/**, .gitea/** — needs a non-author approving review; /approve-override deliberately NOT posted. NOT verified: no OS upgrade run, no node rebooted, no lock observed, the Ansible role never executed, and whether CNPG switches its primary on a drain-induced eviction (the most likely way the first real reboot fails). Full list in the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
99 lines
6.0 KiB
YAML
99 lines
6.0 KiB
YAML
---
|
|
# Flux Kustomization: kured (Kubernetes Reboot Daemon) — reconciles ./platform/kured.
|
|
# Performs the cordon / drain / reboot / uncordon of a node that has a pending OS reboot, strictly
|
|
# one node at a time, inside a nightly window, gated on Longhorn volume health. Design rationale,
|
|
# every tuned value, and the honest list of what is NOT mitigated: platform/kured/helmrelease.yaml.
|
|
#
|
|
# WHY ITS OWN KUSTOMIZATION, AND NOT ONE LINE IN platform/kustomization.yaml
|
|
#
|
|
# The same argument the Cilium split-out recorded in platform/kustomization.yaml and intel-gpu-plugin
|
|
# reused, except the consequence is worse here than for either of them:
|
|
#
|
|
# * `platform` is `wait: true` over twelve-plus components, so its Ready condition cannot tell you
|
|
# "the reboot daemon is wedged" from "longhorn is slow".
|
|
# * The only emergency valve would be `flux suspend kustomization platform` — freezing storage,
|
|
# TLS, DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a DaemonSet.
|
|
# * That tier's 10m interval / 5m timeout are called out there as WRONG for a 3-node DaemonSet
|
|
# rollout. This is a 3-node DaemonSet rollout.
|
|
# * And `platform` gates platform-config, which gates apps-wave-0 (wait:true), which gates
|
|
# apps-wave-1. A reboot daemon that failed to go healthy inside that tier would stall EVERY
|
|
# APPLICATION in the cluster on "dependency not ready". A component added to make maintenance
|
|
# safer must not acquire the ability to cause an outage; that trade is the whole argument.
|
|
#
|
|
# dependsOn: [platform] — AND THIS ONE IS A REAL EDGE, NOT CONVENTION
|
|
#
|
|
# Not `cilium` (what intel-gpu-plugin uses) and not `platform-config` (what the leaf UIs use). It is
|
|
# `platform`, for two independent reasons:
|
|
#
|
|
# 1. CRD ORDERING (F-MON1). platform/kured ships a PrometheusRule, and the chart generates a
|
|
# ServiceMonitor. Both are CRs whose CRDs (monitoring.coreos.com) are installed by
|
|
# kube-prometheus-stack INSIDE the `platform` tier. Flux dry-runs every object in a pass before
|
|
# applying any of them, so a pass containing an unknown kind ABORTS — the failure that made the
|
|
# CDI StorageProfiles a true deadlock on prod on 2026-09-05 (see
|
|
# clusters/base/platform-config-cdi.yaml). Depending on `platform` guarantees the CRDs exist
|
|
# before this pass is attempted.
|
|
# 2. FUNCTIONAL PREREQUISITE. kured's reboot gate QUERIES Prometheus
|
|
# (`prometheusUrl` in the HelmRelease). Prometheus lives in platform/monitoring. If kured ran
|
|
# first it would get connection errors, treat the gate as unevaluable and skip every reboot —
|
|
# the correct fail-safe direction, but silent. Longhorn, whose volume health IS the gate, is
|
|
# also in that tier.
|
|
#
|
|
# It does NOT need platform-config: no Keycloak, no TLS, no split-horizon DNS, no Gateway, no
|
|
# HTTPRoute. kured serves no HTTP to users.
|
|
#
|
|
# wait: true — A HEALTH SIGNAL, NOT A GATE. Following intel-gpu-plugin rather than the leaf-UI
|
|
# template. A leaf UI is legitimately not-Ready for long stretches of a cold start while it waits on
|
|
# Keycloak and DNS, so its health is uninformative; kured has no such dependency — it is a DaemonSet
|
|
# that opens a socket and watches a file, so on healthy nodes it is Ready in seconds and not-Ready
|
|
# genuinely means BROKEN. This turns `flux get kustomization kured` into a one-line answer to "is
|
|
# reboot automation live on all three nodes", which is the question the operator must be able to
|
|
# answer before trusting a kernel upgrade to it.
|
|
#
|
|
# NOTHING IN THE GRAPH dependsOn THIS, so wait:true gates nothing — the blast radius of an unhealthy
|
|
# kured is this Kustomization's own Ready condition. Do not add `dependsOn: [kured]` anywhere: that
|
|
# would hand a node-rebooting DaemonSet the power to stall whatever depends on it, which is precisely
|
|
# what the placement above exists to prevent.
|
|
#
|
|
# prune: true is safe and deliberate. If this Kustomization is deleted, kured's DaemonSet goes with
|
|
# it and reboots simply stop happening — nodes are left uncordoned and running. The one caveat is
|
|
# that pruning mid-reboot would leave a node CORDONED with no daemon to uncordon it; that is a
|
|
# `kubectl uncordon` away and is covered in runbooks/node-reboot-automation.md.
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: kured
|
|
namespace: flux-system
|
|
spec:
|
|
# A HelmRelease plus five static manifests, nothing to re-render; kustomize-controller is
|
|
# source-triggered, so this is only the drift-correction floor.
|
|
interval: 10m
|
|
retryInterval: 2m
|
|
# 10m, not the 5m the `platform` tier uses: a 3-node DaemonSet rollout under `wait: true` can
|
|
# exceed 5m on a cold start (image pull on three nodes), which is one of the reasons the Cilium
|
|
# split-out gave for leaving that tier.
|
|
timeout: 10m
|
|
prune: true
|
|
# See the header: a health SIGNAL for a component nothing depends on, not a gate.
|
|
wait: true
|
|
dependsOn:
|
|
- name: platform # monitoring CRDs + Prometheus + Longhorn; see the header, this is load-bearing
|
|
sourceRef:
|
|
kind: GitRepository
|
|
name: flux-system
|
|
path: ./platform/kured
|
|
# REQUIRED, not boilerplate. platform/kured/alerts.yaml renders `${BASE_DOMAIN}` into the
|
|
# `runbook_url` annotation of all three rules. Without this block that token becomes the EMPTY
|
|
# STRING and the Kustomization still reports Ready — Flux's substitution fails OPEN — so every
|
|
# alert would ship a dead link to `https:///runbooks/node-reboot-automation`. The gate stage
|
|
# `flux-postbuild-substitution` asserts the converse (a rendered token implies this block), and
|
|
# BASE_DOMAIN is declared in clusters/base/cluster-vars.yaml.
|
|
#
|
|
# Nothing else in this path contains `${...}` or a bare `$`: the HelmRelease's alertFilterRegexp is
|
|
# a plain anchored alternation, and there are no shell snippets anywhere in the component — which
|
|
# is deliberate, because envsubst would eat a braced shell variable and `$$` would trip the
|
|
# dollar-collapse assertion in the same check.
|
|
postBuild:
|
|
substituteFrom:
|
|
- kind: ConfigMap
|
|
name: cluster-vars
|