home-network-k3s-impl/platform/kured/helmrelease.yaml

---
# kured (Kubernetes Reboot Daemon) — coordinated, one-node-at-a-time reboots of the three prod
# control-plane nodes.
#
# ═══════════════════════════════════════════════════════════════════════════════════════════════
# READ THIS FIRST: kured DOES NOT DECIDE *WHETHER* TO REBOOT. IT ONLY DECIDES *WHEN* AND *HOW*.
# ═══════════════════════════════════════════════════════════════════════════════════════════════
# kured reboots a node if and only if the file `/var/run/reboot-required` exists on it. On these
# nodes exactly one thing creates that file: a deliberate operator run of
# `playbooks/node-baseline.yml --tags kernel` (which advances the backports kernel). The
# unattended-upgrades role this PR also adds installs SECURITY updates only and BLACKLISTS
# `linux-image-*`/`linux-headers-*`, so it can never produce a reboot-required sentinel on its own —
# see infra/ansible/roles/unattended_upgrades/defaults/main.yml for that argument in full.
#
# That is the whole answer to this repo's two standing objections to reboot automation:
#
#   * infra/ansible/roles/node_baseline/tasks/main.yml:61-67 — "IT DOES NOT REBOOT." k3s-prod-2 hung
#     on a warm reboot on 2026-09-03 (powered, off the LAN ~38 minutes, physical power button
#     required), so rebooting was made "a SERIALIZED, SUPERVISED operator step ... and a role cannot
#     enforce that." STILL TRUE, and nothing here changes it: node_baseline continues not to reboot.
#   * docs/plans/cluster-upgrade-plan.md:291-299 — system-upgrade-controller was rejected because it
#     "automates away the human gate between nodes", and the operator's hard requirement is an
#     explicit PASS/STOP verification BETWEEN every node.
#
# THE HUMAN GATE IS PRESERVED, AND IT IS THE SENTINEL. The operator decides whether a kernel moves
# and on which hosts (`--limit k3s-prod-1`), which is what writes the sentinel on those hosts and
# only those hosts. kured then performs the cordon/drain/reboot/uncordon that the operator would
# otherwise have typed by hand — the error-prone mechanical half, including the ordering rule
# node_baseline's own message has to shout ("NEVER node 2 first"). Run the kernel play one host at a
# time and the operator keeps per-node PASS/STOP control exactly as before; run it on all three and
# kured still does them strictly one at a time, just in an order it chooses. Both are safe for
# quorum; only the first preserves the ordering preference. See runbooks/node-reboot-automation.md.
#
# cluster-upgrade-plan.md:329-330 and :545-548 name the condition under which its own rejection
# reverses — "SUC becomes viable once the cluster is on fast disks ... where drain-driven rebuild is
# cheap" — and prod is that cluster. This component takes that clause up, but NOT its drain
# assumption: see the Longhorn section below, where drain-driven rebuild is measurably NOT free even
# on SSD, and is gated on explicitly rather than waved through.
#
# WHAT IS STILL UNMITIGATED, STATED PLAINLY: kured cannot press a power button. If a node hangs on
# warm reboot the way k3s-prod-2 did in September, kured will not recover it. The blast radius is
# bounded by design rather than by luck — `concurrency: 1` plus `lockTtl: 0` means the hung node
# keeps the cluster lock forever, so NO second node is ever rebooted behind it; etcd keeps 2/3 quorum
# and the cluster stays up, degraded, until a human attends. The KuredRebootPending alert in
# alerts.yaml is what tells them. That is strictly better than the previous automation story (none)
# but it is NOT the same as "safe to ignore at 3am on a Sunday".
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: kured
  namespace: kured
spec:
  interval: 30m
  chart:
    spec:
      chart: kured
      # Chart 6.1.0 -> appVersion 1.23.0 (published 2026-06-30; newest of the 80 entries in
      # https://kubereboot.github.io/charts/index.yaml, read 2026-10-17).
      #
      # THE `<6.2.0` CEILING IS A DELIBERATE CHOICE HERE, NOT THE REFLEX IT IS ELSEWHERE. This PR's
      # own audit found that 9 of 12 HelmReleases in this repo pin a minor-locking range, and that
      # this is a MAIN CAUSE of pin rot: Renovate can then only ever offer patches inside a line it
      # has been told not to leave (kube-prometheus-stack is stuck 4 chart majors back that way).
      # For a component whose job is to reboot control-plane nodes, that ceiling is worth paying on
      # purpose: a kured MINOR may change drain or lock semantics, and this is the one component
      # where "a human looked at the changelog" should be a precondition, not a nice-to-have.
      # Renovate will raise the minor as a non-automerged PR (see the kured packageRule in
      # renovate.json), so the ceiling is a review prompt, not a wall.
      version: ">=6.1.0 <6.2.0"
      sourceRef:
        kind: HelmRepository
        name: kured
        namespace: kured
  install:
    remediation:
      retries: 3
  upgrade:
    remediation:
      retries: 3
  values:
    # ── IDENTITY / PLACEMENT ───────────────────────────────────────────────────────────────────
    # No tolerations. Measured 2026-10-17: all three prod nodes carry `spec.taints` EMPTY (k3s does
    # not taint its servers), so a DaemonSet reaches every node untainted. If a future node IS
    # tainted, kured silently stops covering it and that node then never reboots — invisible,
    # because the other two keep reporting. KuredNodeCoverageGap in alerts.yaml is the detector: it
    # compares kured's per-node series count against the node count rather than merely asking
    # whether kured exists.
    priorityClassName: system-node-critical
    # The chart's defaults are kept: `hostPID: true` and `containerSecurityContext.privileged: true`
    # with `readOnlyRootFilesystem: true`. Both are load-bearing and neither is reducible:
    # `rebootMethod: command` runs the reboot by nsenter'ing /proc/1/ns/mnt, which needs the host PID
    # namespace to see PID 1 and the privilege to enter its mount namespace. The documented
    # lower-privilege alternative (privileged:false + CAP_KILL) applies ONLY to
    # `rebootMethod: signal`, which signals a host-side kured-reboot.service that does not exist on
    # these nodes. Choosing `signal` would mean shipping and maintaining that unit via Ansible for a
    # capability reduction on a DaemonSet that is already cluster-admin-adjacent by function; it is a
    # reasonable future hardening step and is explicitly NOT taken here.
    resources:
      requests:
        cpu: 10m
        memory: 32Mi
      limits:
        # No CPU limit on purpose (the repo's node-pressure work treats CPU limits on tiny system
        # agents as a throttling hazard). Memory is capped: kured holds no state beyond one
        # annotation and a gauge, and 128Mi is ~4x its idle RSS upstream reports.
        memory: 128Mi

    configuration:
      # ── THE QUORUM CONSTRAINT: TWO NODES MUST NEVER BE DOWN AT ONCE ──────────────────────────
      # All three prod nodes are control-plane + etcd (measured 2026-10-17: every node reports
      # `control-plane,etcd`). etcd quorum is 2 of 3, so one node down is degraded-but-up and two
      # nodes down is a LOST CLUSTER that does not self-heal. Everything in this block exists to
      # make the second case structurally unreachable rather than merely unlikely.
      #
      # concurrency: 1 — kured's cluster-wide lock is a single annotation
      # (`weave.works/kured-node-lock`) on kured's OWN DaemonSet object, and a node reboots only
      # while it holds that lock. One lock + concurrency 1 = one reboot, enforced by the API server's
      # optimistic concurrency rather than by timing. This is the chart default; it is written out
      # explicitly because it is the single most important value in this file.
      concurrency: 1
      # lockTtl: 0 (DISABLED) — THIS IS A SAFETY DECISION, NOT A DEFAULT WE FORGOT TO TUNE.
      # A TTL makes the lock EXPIRE, which exists for the case "the lock holder died and we want
      # reboots to continue". On a 3-node etcd cluster that is precisely the catastrophe: if node A
      # has hung on a warm reboot (which has happened here — see the header) and its lock expires,
      # node B becomes eligible and takes the cluster from 2/3 to 1/3. We would rather the reboot
      # cycle STOP FOREVER and page a human than have it step over a corpse. The cost is that a
      # genuinely stale lock needs manual clearing; runbooks/node-reboot-automation.md has the
      # procedure, and KuredRebootPending is the alert that surfaces it.
      lockTtl: 0
      # lockReleaseDelay: 30m — hold the lock for 30 minutes AFTER this node is back and uncordoned,
      # before any other node may take it. See the Longhorn section: this is the STRUCTURAL half of
      # the inter-node gate, deliberately not dependent on an alert's `for:` timer having latched.
      lockReleaseDelay: 30m

      # ── THE MAINTENANCE WINDOW: 03:00-06:00 EVERY DAY, America/New_York ──────────────────────
      # The timezone and the 03:00-06:00 band match renovate.json's existing
      # `"schedule": ["after 3am and before 6am"]` / `"timezone": "America/New_York"`, so every
      # unattended change in this repo lands in the same quiet 3-hour window.
      #
      # DAILY, NOT WEEKLY, AND THE REASON IS THE ALERT — not impatience. A weekly (`- sun`) window
      # was the first draft and it breaks the only alert that can tell you this mechanism has
      # stalled. With a Sunday-only window, a kernel upgraded on a Monday legitimately waits six
      # days, so "a reboot has been pending too long" cannot be distinguished from "it is Tuesday"
      # until the threshold exceeds a full 7-day cycle. That forces `for: 8d`+, which is longer than
      # this cluster's Prometheus retention (MONITORING_PROMETHEUS_RETENTION is 7d) and is reset by
      # any Prometheus restart beyond the 1h default for-outage-tolerance — i.e. the alert would be
      # unreliable in exactly the direction that matters. A daily window caps the legitimate wait at
      # under 24h, which makes KuredRebootPending's `for: 36h` both robust and meaningful: 36 hours
      # with a sentinel still set means at least one full window passed and SOMETHING BLOCKED IT.
      #
      # Daily does NOT weaken the quorum guarantee — that is `concurrency: 1` plus the lock, not the
      # calendar. It does mean a 3-node kernel rollout can finish in one to three nights instead of
      # three weeks, with each node still gated on the previous one's Longhorn rebuilds (below).
      rebootDays:
        - su
        - mo
        - tu
        - we
        - th
        - fr
        - sa
      startTime: "03:00"
      endTime: "06:00"
      timeZone: America/New_York
      # How often to re-evaluate (sentinel + window + alerts + lock). 1h is the chart default; at a
      # 3h window that is three evaluation opportunities per node per night.
      period: 1h

      # ── THE SENTINEL ─────────────────────────────────────────────────────────────────────────
      # Written ONLY by a deliberate operator kernel upgrade (see the header). `update-notifier-common`
      # — the package that owns /etc/kernel/postinst.d and actually creates this file on Debian — was
      # measured ABSENT on all three nodes on 2026-10-17, which is why the unattended_upgrades role
      # installs it. Without that package nothing ever writes this path, kured correctly never
      # reboots, every probe looks healthy, and the entire mechanism is a silent no-op. That failure
      # mode is the reason KuredNotReporting exists.
      #
      # THE PATH kured LOGS IS NOT THIS PATH, and that trips people debugging it. The chart
      # defaults `useRebootSentinelHostPath: true` (kept), which bind-mounts the DIRNAME of this
      # value onto /sentinel read-only and renders the flag as
      # `--reboot-sentinel=/sentinel/reboot-required` (daemonset.yaml:134). So the container watches
      # /sentinel/reboot-required and the host file is /var/run/reboot-required. Same file, two
      # names. Set this value as the HOST path, as below.
      rebootSentinel: /var/run/reboot-required

      # ── THE LONGHORN GATE: WHY POD READINESS IS NOT A SUFFICIENT INTER-NODE SIGNAL ────────────
      # Measured on prod 2026-10-17, and this is the finding that shaped this file:
      #   * 25 Longhorn volumes, ALL `numberOfReplicas: 3`, all healthy, with exactly 25 replicas on
      #     each of the 3 nodes. So there is no single-replica volume to lose today.
      #   * BUT `replica-soft-anti-affinity: false` (replicas may not share a node) and
      #     `disable-scheduling-on-cordoned-node: true`. With 3 nodes and 3 replicas, cordoning ONE
      #     node leaves nowhere to place a replacement — so every drain takes ALL 25 volumes to
      #     degraded 2/3 for the whole reboot.
      #   * Recovery is slow and serialized: `replica-replenishment-wait-interval: 600` (10 minutes
      #     before a rebuild even starts) and `concurrent-replica-rebuild-per-node-limit: 1`.
      # Therefore: if the next node is rebooted before rebuilds finish, those volumes go to 1/3, and
      # the drain after that genuinely CAN hit Longhorn's `node-drain-policy:
      # block-if-contains-last-replica` and block forever. The inter-node gate must be VOLUME
      # ROBUSTNESS, not "are the pods Ready again".
      #
      # It is gated twice, on purpose, because each half covers the other's blind spot:
      #   1. lockReleaseDelay: 30m (above) — unconditional. Covers the 600s replenishment wait with
      #      margin and does not depend on any alert having fired.
      #   2. The Prometheus alert gate below — semantic. Covers a rebuild that takes LONGER than 30
      #      minutes (25 serialized rebuilds on one node can), which a fixed delay cannot.
      prometheusUrl: http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090
      # Only consider alerts that are FIRING, not merely pending. A `pending` alert is a 10-minute
      # guess; blocking a weekly reboot on one would make the gate fire on noise.
      alertFiringOnly: true
      # ── alertFilterMatchOnly: true IS LOAD-BEARING. DO NOT DROP IT. ──────────────────────────
      # kured's two alert modes are near-opposites and the flag name does not advertise it:
      #   * false (the chart default): alertFilterRegexp is an IGNORE list — kured blocks on ANY
      #     firing alert except those matching.
      #   * true: alertFilterRegexp is an ALLOW list — kured blocks ONLY on alerts that match.
      # We need `true`, because this cluster always has something firing. Measured via the Prometheus
      # API on 2026-10-17, these five were firing with no reboot pending and no end in sight:
      # CNPGPrimariesCoLocated, KubeDeploymentReplicasMismatch, KubeDeploymentRolloutStuck,
      # KubePodNotReady, OAuth2ProxyGateUnobserved (the last three all trace to the pm-agent pod,
      # ImagePullBackOff on a placeholder digest for 12 days). Under the default `false`, kured would
      # have been blocked by those on day one and would never have rebooted anything — a silent
      # no-op indistinguishable, from outside, from "no reboots were needed".
      alertFilterMatchOnly: true
      # The ALLOW list: the conditions under which rebooting ANOTHER node risks data or quorum.
      # Anchored and explicit — a prefix match here would silently re-admit the chronic five above.
      #   LonghornVolumeDegraded  — the inter-node gate (see above). `longhorn_volume_robustness
      #                             {state="degraded"} == 1`, for: 10m, in platform/monitoring/alerts.yaml.
      #   LonghornVolumeFaulted   — data already at risk; never reboot into this.
      #   KubeNodeNotReady        — a node is already down. Rebooting a second one loses quorum. This
      #                             is the single most important entry for the etcd constraint.
      #   NodeExporterDown        — a node we cannot see the health of. Treated as "assume the worst".
      # DELIBERATELY NOT INCLUDED: NodeEtcdRequestsSlow and the NodeIOPressure* family. They are
      # exactly the conditions you would want to pause for, but prod-1 has a documented history of
      # etcd WAL I/O starvation, so including them risks a permanent wedge that looks like safety.
      # Revisit once those are quiet for a sustained period; KuredRebootPending makes a wedge visible
      # either way.
      alertFilterRegexp: ^(LonghornVolumeDegraded|LonghornVolumeFaulted|KubeNodeNotReady|NodeExporterDown)$

      # ── THE DRAIN ────────────────────────────────────────────────────────────────────────────
      # NOTE ON FLAGS THAT DO NOT EXIST: there is no `--ignore-daemonsets`, no
      # `--delete-emptydir-data` and no `--force` in kured 1.23.0. Read from
      # cmd/kured/main.go:438-449 on 2026-10-17, kured HARDCODES
      # `IgnoreAllDaemonSets: true`, `DeleteEmptyDirData: true` and `Force: true` in the drain
      # helper. Passing them as extraArgs would CrashLoop the DaemonSet on an unknown flag. This
      # matters because all three are genuinely REQUIRED here and it is easy to assume they must be
      # configured: 12 DaemonSets and 135 emptyDir-carrying pods were measured on 2026-10-17, and a
      # drain without those three settings refuses to start.
      #
      # drainTimeout: 15m — the chart/kured default is 0, meaning INFINITE, and infinite is the wrong
      # answer here. Measured 2026-10-17, six PodDisruptionBudgets currently allow ZERO voluntary
      # disruptions, and a plain drain loops on them indefinitely:
      #   * 3x longhorn-system/instance-manager-* — one per node, `minAvailable: 1` against a
      #     selector pinned to a single node's single pod, so disruptionsAllowed is STRUCTURALLY 0
      #     forever. This is the mechanism by which Longhorn's node-drain-policy blocks: the
      #     longhorn-manager deletes the node's instance-manager PDB only once it judges the drain
      #     safe. The drain is SUPPOSED to wait on this one.
      #   * 3x CNPG `<cluster>-primary` (keycloak-db, nextcloud-db, semaphore-db) — `minAvailable: 1`
      #     against a selector matching exactly one pod (the primary), so again structurally 0 until
      #     the CNPG operator switches the primary away. All three primaries were on k3s-prod-1.
      # With a timeout, a drain that is genuinely stuck ABORTS, the lock is released, and kured
      # retries next period instead of holding the cluster lock forever on a wedged node. 15m is
      # chosen to be longer than a CNPG switchover and longer than the `unifi` VM's eviction (below)
      # usually needs, while still being well inside the 3h window.
      drainTimeout: 15m
      # Taint a node PreferNoSchedule while its reboot is pending, so the scheduler stops placing new
      # pods on a node that is about to go away. With concurrency 1 this is a small win (there is no
      # second rebooting node to shed pods onto) but it is free and it shortens the drain.
      preferNoScheduleTaint: weave.works/kured-node-reboot
      # Record reboot state on the Node objects (`weave.works/kured-reboot-in-progress`,
      # `weave.works/kured-most-recent-reboot-needed`). Off by default. Turned on because the only
      # other observability kured offers is ONE gauge, and during an incident `kubectl get node -o
      # yaml` is the thing an operator can reach without Prometheus.
      annotateNodes: true
      logFormat: json

    # ── METRICS: BOTH OF THESE MUST BE true OR THE ALERTS IN alerts.yaml CANNOT FIRE ───────────
    # The chart ships `service.create: false` AND `metrics.create: false`. Leaving either at its
    # default means no Service, or a ServiceMonitor with nothing to select, so `kured_reboot_required`
    # is never scraped — and a PrometheusRule over a series that no target produces is INERT: quiet
    # forever, indistinguishable from healthy. This repo has shipped that exact bug twice
    # (VeleroNoRecentBackup over an absent series; the inert gotk_* rules in
    # runbooks/image-automation.md), which is why platform/monitoring/tests/run.sh exists at all.
    # KuredNotReporting (alerts.yaml) is the standing detector for this pair regressing.
    # Prometheus selects ServiceMonitors cluster-wide: measured 2026-10-17, `serviceMonitorSelector`
    # and `serviceMonitorNamespaceSelector` are both `{}`, so no label plumbing is needed here.
    service:
      create: true
      port: 8080
    metrics:
      create: true
      interval: 60s