Merge pull request 'feat(upgrades): automate node reboots (kured) and wire Renovate, which had never run' (#73) from feat/automated-upgrades into main
Reviewed-on: #73 Reviewed-by: Keith Jackson <[email protected]>
This commit was merged in pull request #73.
This commit is contained in:
jakman85 committed
commit
ee196c7bcd
22 files changed
+2661
-5
No files matched your search
@@ -0,0 +1,127 @@
|
||||
# platform/kured
|
||||
|
||||
**kured (Kubernetes Reboot Daemon)** — coordinated, one-node-at-a-time OS reboots of the three prod
|
||||
control-plane nodes.
|
||||
|
||||
- **Flux tier:** `clusters/base/kured.yaml` (its own Kustomization, `dependsOn: [platform]`,
|
||||
`wait: true`)
|
||||
- **Chart:** `kured` 6.1.0 → appVersion 1.23.0, from `https://kubereboot.github.io/charts`
|
||||
- **The other half:** `infra/ansible/roles/unattended_upgrades` — the package upgrades. **Neither
|
||||
half is useful alone.**
|
||||
- **Runbook:** `/runbooks/node-reboot-automation`
|
||||
- **Every tuned value and its reasoning:** `helmrelease.yaml`. That file is the argument; this one is
|
||||
the orientation.
|
||||
- **Design record:** `decisions/2026-10-17-automated-upgrades.md`
|
||||
|
||||
## What this component owns
|
||||
|
||||
The *mechanics* of a reboot: take a cluster-wide lock, cordon, drain, reboot, uncordon, hold the
|
||||
lock a while longer. Nothing else.
|
||||
|
||||
**It does not decide whether to reboot.** kured reboots a node if and only if
|
||||
`/var/run/reboot-required` exists there. On these nodes that file is written by a deliberate
|
||||
operator kernel upgrade (`playbooks/node-baseline.yml --tags kernel`) — unattended-upgrades is
|
||||
security-only with the kernel blacklisted, so it essentially never produces one. **The human gate is
|
||||
the sentinel, not the daemon.**
|
||||
|
||||
## The problem this fixes
|
||||
|
||||
Rebooting these three nodes was a manual, serialized, supervised procedure, and
|
||||
`roles/node_baseline` has to end its kernel run by shouting the rules at the operator: *"Reboot prod
|
||||
nodes ONE AT A TIME, verify each returns before touching the next, and NEVER node 2 first"*. That is
|
||||
a correct procedure and an error-prone one, and nothing enforced it.
|
||||
|
||||
What it has to get right, measured on prod 2026-10-17:
|
||||
|
||||
- **etcd quorum is 2 of 3** — all three nodes are control-plane+etcd. Two down is a lost cluster.
|
||||
- **Every drain degrades all 25 Longhorn volumes to 2/3**, because 3 nodes × 3 replicas ×
|
||||
`replica-soft-anti-affinity: false` leaves nowhere to place a replacement, and rebuild is
|
||||
serialized behind a 600s wait. Reboot the next node too early and volumes hit 1/3, where
|
||||
`block-if-contains-last-replica` can block a drain forever.
|
||||
- **Six PodDisruptionBudgets allow zero voluntary disruptions** — 3 Longhorn `instance-manager-*`
|
||||
(structurally 0 by design; this is how Longhorn gates the drain) and 3 CNPG `*-primary`.
|
||||
|
||||
## Placement — why its own Kustomization
|
||||
|
||||
`platform` is `wait: true` over twelve components and gates `platform-config` → every apps wave. A
|
||||
reboot daemon that failed to go healthy there would stall **every application in the cluster** on
|
||||
"dependency not ready". A component added to make maintenance safer must not be able to cause an
|
||||
outage. `wait: true` here is a health **signal** for something nothing depends on — and nothing
|
||||
should ever `dependsOn: [kured]`.
|
||||
|
||||
The `dependsOn: [platform]` edge is load-bearing, not conventional: it supplies the
|
||||
`monitoring.coreos.com` CRDs this component's `PrometheusRule` needs (F-MON1 — a pass containing an
|
||||
unknown kind aborts), plus the Prometheus that kured **queries** before every reboot.
|
||||
|
||||
## How concurrent reboots are made impossible
|
||||
|
||||
| mechanism | value | why |
|
||||
|---|---|---|
|
||||
| cluster-wide lock | one annotation on kured's own DaemonSet | a node reboots only while holding it; enforced by API-server optimistic concurrency, not timing |
|
||||
| `concurrency` | `1` | one lock holder at a time |
|
||||
| **`lockTtl`** | **`0` (disabled)** | a TTL lets reboots continue after the holder dies — on 3 etcd nodes that is the catastrophe. A hung node holds the lock **forever** and nothing cascades. |
|
||||
| `lockReleaseDelay` | `30m` | the next node waits out Longhorn's 600s replenishment even if no alert latched |
|
||||
| alert gate | `alertFilterMatchOnly: true` + anchored allow-list | refuses to proceed while a volume is degraded/faulted or a node is down |
|
||||
| window | 03:00–06:00 daily, America/New_York | matches `renovate.json`'s window; daily (not weekly) is what makes `KuredRebootPending`'s `for: 36h` meaningful |
|
||||
|
||||
## Deliberate non-choices
|
||||
|
||||
- **No `--ignore-daemonsets` / `--delete-emptydir-data` / `--force`.** Those flags **do not exist**
|
||||
in kured 1.23.0 — it hardcodes all three in the drain helper (`cmd/kured/main.go:438-449`).
|
||||
Passing them would CrashLoop the DaemonSet. All three are genuinely required here (12 DaemonSets,
|
||||
135 emptyDir pods).
|
||||
- **`rebootMethod: command`, not `signal`.** `signal` would allow `privileged: false` + `CAP_KILL`,
|
||||
but needs a host-side `kured-reboot.service` that does not exist on these nodes. A real future
|
||||
hardening step; explicitly not taken here.
|
||||
- **No tolerations.** No prod node is tainted (measured). A future tainted node would be silently
|
||||
uncovered — hence `KuredNodeCoverageGap`.
|
||||
- **`NodeEtcdRequestsSlow` / `NodeIOPressure*` are NOT in the alert gate**, despite being exactly
|
||||
what you would want to pause for. prod-1 has a documented etcd WAL I/O starvation history and
|
||||
including them risks a permanent wedge that looks like safety.
|
||||
- **No Slack/shoutrrr notifications.** Would require a SOPS Secret and a decryption patch; the three
|
||||
alerts cover the same ground through Alertmanager.
|
||||
|
||||
## Versioning
|
||||
|
||||
Chart pinned `>=6.1.0 <6.2.0`. The minor ceiling is **deliberate here**, unlike elsewhere in this
|
||||
repo where it is a main cause of pin rot: a kured minor may change drain or lock semantics, and this
|
||||
is the one component where "a human read the changelog" should be a precondition. `renovate.json`'s
|
||||
`blast-radius` rule additionally forbids auto-merging it, so the ceiling is a review prompt, not a
|
||||
wall.
|
||||
|
||||
## What breaks silently, and what catches it
|
||||
|
||||
Every failure mode of this component leaves a cluster that looks healthy — kured stays 1/1 Running
|
||||
while rebooting nothing. `alerts.yaml` ships three rules over kured's single metric, with a
|
||||
companion promtool suite:
|
||||
|
||||
| alert | catches |
|
||||
|---|---|
|
||||
| `KuredNotReporting` | the scrape pipeline is dead (chart defaults `service.create`/`metrics.create` to **false**; a NetworkPolicy can time the scrape out) — and the other two are inert while this is true |
|
||||
| `KuredNodeCoverageGap` | kured missing on *some* node, which `absent()` cannot see |
|
||||
| `KuredRebootPending` | a reboot was offered and blocked for >36h |
|
||||
|
||||
## Post-merge verification
|
||||
|
||||
```bash
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds,pod,svc,servicemonitor -o wide'
|
||||
infra/devcontainer/dev.sh run bash -c 'flux --context k3s-prod get kustomization kured'
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured logs ds/kured --tail=40'
|
||||
```
|
||||
|
||||
Expect `3/3` ready, the Kustomization `Ready=True`, and kured logging a reboot-check decision each
|
||||
period. Then confirm the scrape is real — this is the step that proves the alerts are not inert:
|
||||
|
||||
```bash
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n monitoring exec \
|
||||
sts/prometheus-kube-prometheus-stack-prometheus -c prometheus -- \
|
||||
wget -qO- "http://localhost:9090/api/v1/query?query=count(kured_reboot_required)"'
|
||||
```
|
||||
|
||||
**Expect `3`.** Anything else (and especially an empty result) means the Service, ServiceMonitor or
|
||||
NetworkPolicy is wrong and all three alerts are silently dead.
|
||||
|
||||
**Before trusting the timer with a real kernel upgrade**, do one watched drain to settle the open
|
||||
question the design record flags: whether CNPG actually switches its primary away in response to a
|
||||
drain-induced eviction. If it does not, draining `k3s-prod-1` blocks on three 0-disruption PDBs at
|
||||
once.
|
||||
@@ -0,0 +1,282 @@
|
||||
---
|
||||
# promtool unit tests for the three rules in platform/kured/alerts.yaml.
|
||||
#
|
||||
# WHY THIS FILE EXISTS. The rules it covers are the ONLY thing that can tell an operator that the
|
||||
# reboot automation has stopped working, and every failure mode they describe is silent — kured stays
|
||||
# 1/1 Running while rebooting nothing. An alert that cannot fire is therefore not a cosmetic defect
|
||||
# here, it is the removal of the only signal. This repo has shipped that bug in both directions
|
||||
# already (a permanently-firing VPNTunnelDown; a structurally inert VeleroNoRecentBackup over an
|
||||
# absent series), which is why the harness exists at all. Each rule below is pinned BOTH ways: quiet
|
||||
# on the healthy state, firing on the constructed bad one, with the `for:` boundary asserted rather
|
||||
# than assumed.
|
||||
#
|
||||
# HOW TO RUN. `platform/monitoring/tests/run.sh` — the one repo-wide rule-unit harness. It finds this
|
||||
# file by the COMPANION convention (`<manifest>.promtool-test.yaml` beside `<manifest>.yaml`), so the
|
||||
# manifest under test is exactly platform/kured/alerts.yaml and nothing else. Run just this file with
|
||||
# platform/monitoring/tests/run.sh platform/kured/alerts.promtool-test.yaml
|
||||
# The harness flattens the PrometheusRule CR's `.spec.groups` into the bare rules file promtool wants,
|
||||
# `promtool check rules` it first so a non-parsing file cannot pass vacuously, and runs it all in a
|
||||
# temp dir. It leaves Flux's `${BASE_DOMAIN}` in the runbook_url annotations verbatim, which is safe
|
||||
# because the assertions below query the `ALERTS` series and never compare annotations.
|
||||
#
|
||||
# WHY THE ASSERTIONS QUERY `ALERTS` RATHER THAN USING `exp_alerts`. promtool's alert_rule_test
|
||||
# compares annotations VERBATIM, which would mean pasting each rule's description prose in here and
|
||||
# re-pasting it on every wording change. Querying the synthetic `ALERTS` series proves what actually
|
||||
# matters — the alert reaches `pending` and then `firing` across its `for:` boundary, with the right
|
||||
# labels — and stays decoupled from annotation text. Same choice as
|
||||
# platform/cnpg/alerts.promtool-test.yaml.
|
||||
#
|
||||
# THE SERIES SHAPES ARE REAL. `kured_reboot_required{node=...}` is a GaugeVec with exactly one label,
|
||||
# `node` — read from kured's cmd/kured/main.go:92-96 (the only registered metric in the project).
|
||||
# `kube_node_info{node=...}` was measured on prod Prometheus on 2026-10-17 returning
|
||||
# `count(kube_node_info) == 3`, matching the three nodes. The extra kube_node_info labels prod
|
||||
# carries are omitted deliberately: every rule aggregates with `count()`, so they cannot affect the
|
||||
# result, and inventing plausible-but-wrong values would be worse than leaving them out.
|
||||
#
|
||||
# evaluation_interval is 5m, not the usual 1m: KuredRebootPending has `for: 36h`, and at a 1m
|
||||
# interval promtool would evaluate that test over 2000+ steps for no added coverage.
|
||||
rule_files:
|
||||
- alerts.rules.yaml
|
||||
evaluation_interval: 5m
|
||||
|
||||
tests:
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
# 1. THE HEALTHY STEADY STATE. Three nodes, kured running on all three, nothing pending.
|
||||
# ALL THREE RULES MUST BE SILENT. This is the case that a badly-written `absent()` or a
|
||||
# `count() <` comparison with an off-by-one gets wrong, and it is also the state the cluster
|
||||
# is in ~100% of the time, so a false positive here would train the operator to ignore the
|
||||
# whole group.
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
- interval: 5m
|
||||
name: healthy cluster - no reboots pending, kured on every node, all three rules quiet
|
||||
input_series:
|
||||
- series: 'kured_reboot_required{node="k3s-prod-1"}'
|
||||
values: '0x60'
|
||||
- series: 'kured_reboot_required{node="k3s-prod-2"}'
|
||||
values: '0x60'
|
||||
- series: 'kured_reboot_required{node="k3s-prod-3"}'
|
||||
values: '0x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-1"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-2"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-3"}'
|
||||
values: '1x60'
|
||||
promql_expr_test:
|
||||
# Nothing from this group may be pending or firing, at any point in the window.
|
||||
- expr: 'count(ALERTS{alertname=~"Kured.*"})'
|
||||
eval_time: 4h
|
||||
exp_samples: []
|
||||
# And the underlying expressions must each be genuinely false, not merely unevaluated:
|
||||
# absent() yields no series because the vector is populated...
|
||||
- expr: 'absent(kured_reboot_required)'
|
||||
eval_time: 4h
|
||||
exp_samples: []
|
||||
# ...coverage is exact (3 == 3), so the `<` comparison drops the sample...
|
||||
- expr: 'count(kured_reboot_required) < count(kube_node_info)'
|
||||
eval_time: 4h
|
||||
exp_samples: []
|
||||
# ...and no node reports a pending reboot.
|
||||
- expr: 'max by (node) (kured_reboot_required) == 1'
|
||||
eval_time: 4h
|
||||
exp_samples: []
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
# 2. KuredNotReporting — THE WHOLE PIPELINE IS DEAD. No kured series at all, which is what the
|
||||
# cluster looks like if the chart's `service.create`/`metrics.create` regress to their FALSE
|
||||
# defaults, if the ServiceMonitor is dropped, or if a NetworkPolicy silently times out the
|
||||
# scrape (the Longhorn precedent). The nodes still exist, so kube_node_info is present —
|
||||
# proving the rule keys on kured's absence and not on a dead cluster.
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
- interval: 5m
|
||||
name: no kured series anywhere - KuredNotReporting fires after 30m
|
||||
input_series:
|
||||
- series: 'kube_node_info{node="k3s-prod-1"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-2"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-3"}'
|
||||
values: '1x60'
|
||||
promql_expr_test:
|
||||
# The expression itself yields 1 immediately.
|
||||
- expr: 'absent(kured_reboot_required)'
|
||||
eval_time: 10m
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: 1
|
||||
# `for: 30m` is honoured — still only pending at 25m.
|
||||
- expr: 'ALERTS{alertname="KuredNotReporting"}'
|
||||
eval_time: 25m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="KuredNotReporting", alertstate="pending", severity="warning"}'
|
||||
value: 1
|
||||
# ...and firing once past it.
|
||||
- expr: 'ALERTS{alertname="KuredNotReporting", alertstate="firing"}'
|
||||
eval_time: 35m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="KuredNotReporting", alertstate="firing", severity="warning"}'
|
||||
value: 1
|
||||
# THE GUARD CLAUSE: with kured absent, the other two rules are inert — exactly the reason
|
||||
# this rule has to exist. KuredRebootPending cannot fire (no series to be 1) and
|
||||
# KuredNodeCoverageGap cannot either, because `count()` over an empty vector returns NO
|
||||
# SAMPLE rather than 0, so the `<` comparison has nothing to compare. Asserting that here
|
||||
# pins the dependency: if a future edit made coverage-gap fire in this state, the overlap
|
||||
# would be deliberate rather than accidental.
|
||||
- expr: 'ALERTS{alertname="KuredRebootPending"}'
|
||||
eval_time: 35m
|
||||
exp_samples: []
|
||||
- expr: 'ALERTS{alertname="KuredNodeCoverageGap"}'
|
||||
eval_time: 35m
|
||||
exp_samples: []
|
||||
- expr: 'count(kured_reboot_required) < count(kube_node_info)'
|
||||
eval_time: 35m
|
||||
exp_samples: []
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
# 3. KuredNodeCoverageGap — kured COVERS ONLY TWO OF THREE NODES. k3s-prod-3's series stops at
|
||||
# 1h and never returns (a node taint this DaemonSet does not tolerate, a node that never came
|
||||
# back from a reboot, a DaemonSet nodeSelector edit). This also pins the `for: 2h` choice,
|
||||
# which exists SO THAT AN ORDINARY REBOOT DOES NOT TRIP IT: the gap is real from ~1h05 but
|
||||
# the alert is only pending an hour later, and a healthy reboot is minutes.
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
- interval: 5m
|
||||
name: kured missing on one node - KuredNodeCoverageGap fires after 2h, not before
|
||||
input_series:
|
||||
- series: 'kured_reboot_required{node="k3s-prod-1"}'
|
||||
values: '0x60'
|
||||
- series: 'kured_reboot_required{node="k3s-prod-2"}'
|
||||
values: '0x60'
|
||||
# Ends after 1h (13 samples at 5m) and is stale thereafter.
|
||||
- series: 'kured_reboot_required{node="k3s-prod-3"}'
|
||||
values: '0x12'
|
||||
- series: 'kube_node_info{node="k3s-prod-1"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-2"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-3"}'
|
||||
values: '1x60'
|
||||
promql_expr_test:
|
||||
# Before the gap opens, coverage is exact and nothing is pending.
|
||||
- expr: 'ALERTS{alertname="KuredNodeCoverageGap"}'
|
||||
eval_time: 30m
|
||||
exp_samples: []
|
||||
# The shortfall is real well before the alert fires — 2 < 3.
|
||||
- expr: 'count(kured_reboot_required)'
|
||||
eval_time: 2h
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: 2
|
||||
# A SHORT OUTAGE DOES NOT ALERT: still only pending at 2h, an hour after the gap opened.
|
||||
- expr: 'ALERTS{alertname="KuredNodeCoverageGap"}'
|
||||
eval_time: 2h
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="KuredNodeCoverageGap", alertstate="pending", severity="warning"}'
|
||||
value: 1
|
||||
# Firing once the gap has persisted past `for: 2h`.
|
||||
- expr: 'ALERTS{alertname="KuredNodeCoverageGap", alertstate="firing"}'
|
||||
eval_time: 3h15m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="KuredNodeCoverageGap", alertstate="firing", severity="warning"}'
|
||||
value: 1
|
||||
# KuredNotReporting must stay QUIET: two nodes are still publishing, so the vector is not
|
||||
# absent. This is the distinction between "kured is gone" and "kured is incomplete", and
|
||||
# getting it wrong in either direction would collapse two different diagnoses into one.
|
||||
- expr: 'ALERTS{alertname="KuredNotReporting"}'
|
||||
eval_time: 3h15m
|
||||
exp_samples: []
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
# 4. KuredRebootPending — A REBOOT IS BLOCKED. k3s-prod-1 reports a pending reboot continuously:
|
||||
# the sentinel was written (an operator kernel upgrade) and the reboot has not happened across
|
||||
# more than one 03:00-06:00 window. Asserts the 36h boundary in both directions, which is the
|
||||
# assertion that makes the DAILY-window decision load-bearing rather than cosmetic: on a
|
||||
# weekly window this threshold could not distinguish "blocked" from "waiting for Sunday".
|
||||
# Series run to 38h20m (456 samples at 5m) so the post-boundary evaluation has real data.
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
- interval: 5m
|
||||
name: reboot pending on one node for over 36h - KuredRebootPending fires, per-node label kept
|
||||
input_series:
|
||||
- series: 'kured_reboot_required{node="k3s-prod-1"}'
|
||||
values: '1x456'
|
||||
- series: 'kured_reboot_required{node="k3s-prod-2"}'
|
||||
values: '0x456'
|
||||
- series: 'kured_reboot_required{node="k3s-prod-3"}'
|
||||
values: '0x456'
|
||||
- series: 'kube_node_info{node="k3s-prod-1"}'
|
||||
values: '1x456'
|
||||
- series: 'kube_node_info{node="k3s-prod-2"}'
|
||||
values: '1x456'
|
||||
- series: 'kube_node_info{node="k3s-prod-3"}'
|
||||
values: '1x456'
|
||||
promql_expr_test:
|
||||
# The expression selects ONLY the pending node — the operator needs to know which box.
|
||||
- expr: 'max by (node) (kured_reboot_required) == 1'
|
||||
eval_time: 1h
|
||||
exp_samples:
|
||||
- labels: '{node="k3s-prod-1"}'
|
||||
value: 1
|
||||
# A NORMAL WAIT IS NOT AN ALERT. At 24h the node is still inside its legitimate
|
||||
# once-per-day window wait, so the alert may be pending but must NOT be firing.
|
||||
- expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}'
|
||||
eval_time: 24h
|
||||
exp_samples: []
|
||||
# Still not firing just short of the boundary.
|
||||
- expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}'
|
||||
eval_time: 35h30m
|
||||
exp_samples: []
|
||||
# Firing past it, carrying the node label through to the notification.
|
||||
- expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}'
|
||||
eval_time: 36h30m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="KuredRebootPending", alertstate="firing", node="k3s-prod-1", severity="warning"}'
|
||||
value: 1
|
||||
# The two healthy nodes must NOT be alerted on. A rule written with `kured_reboot_required`
|
||||
# unaggregated, or compared with `>= 0`, would raise all three and bury the real one.
|
||||
- expr: 'count(ALERTS{alertname="KuredRebootPending", alertstate="firing"})'
|
||||
eval_time: 36h30m
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: 1
|
||||
# Liveness rules stay quiet: kured is present and covering all three nodes. A pending reboot
|
||||
# is NOT a wiring fault and must not be reported as one.
|
||||
- expr: 'ALERTS{alertname="KuredNotReporting"}'
|
||||
eval_time: 36h30m
|
||||
exp_samples: []
|
||||
- expr: 'ALERTS{alertname="KuredNodeCoverageGap"}'
|
||||
eval_time: 36h30m
|
||||
exp_samples: []
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
# 5. THE RULE MUST STOP. A reboot that SUCCEEDS clears the sentinel, and the alert must clear
|
||||
# with it — the other half of the failure space this harness exists for (a permanently-firing
|
||||
# rule is how VPNTunnelDown taught this repo the lesson). k3s-prod-1 is pending for ~40h,
|
||||
# long enough to fire, then the reboot happens and the gauge returns to 0.
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
- interval: 5m
|
||||
name: reboot completes after firing - KuredRebootPending resolves
|
||||
input_series:
|
||||
# 1 for 480 samples (40h), then 0 for the remainder.
|
||||
- series: 'kured_reboot_required{node="k3s-prod-1"}'
|
||||
values: '1x480 0x120'
|
||||
- series: 'kured_reboot_required{node="k3s-prod-2"}'
|
||||
values: '0x600'
|
||||
- series: 'kured_reboot_required{node="k3s-prod-3"}'
|
||||
values: '0x600'
|
||||
- series: 'kube_node_info{node="k3s-prod-1"}'
|
||||
values: '1x600'
|
||||
- series: 'kube_node_info{node="k3s-prod-2"}'
|
||||
values: '1x600'
|
||||
- series: 'kube_node_info{node="k3s-prod-3"}'
|
||||
values: '1x600'
|
||||
promql_expr_test:
|
||||
# Firing while the reboot is still outstanding.
|
||||
- expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}'
|
||||
eval_time: 39h
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="KuredRebootPending", alertstate="firing", node="k3s-prod-1", severity="warning"}'
|
||||
value: 1
|
||||
# Cleared once the sentinel is gone. If this assertion ever fails, the rule has become
|
||||
# self-sustaining and the operator will start ignoring it.
|
||||
- expr: 'ALERTS{alertname=~"Kured.*"}'
|
||||
eval_time: 45h
|
||||
exp_samples: []
|
||||
@@ -0,0 +1,124 @@
|
||||
---
|
||||
# Alert rules for the kured reboot automation.
|
||||
#
|
||||
# WHY THESE THREE, AND WHY THEY ARE NOT OPTIONAL. Reboot automation's characteristic failure is
|
||||
# SILENCE, not noise. Every way this component can break leaves a cluster that looks perfectly
|
||||
# healthy: a sentinel nothing writes, a Service the chart did not create, a NetworkPolicy that drops
|
||||
# the scrape, a taint that excludes a node, a drain wedged on a PodDisruptionBudget, a cluster lock
|
||||
# held forever by a node that hung on warm reboot. In none of those cases does anything turn red on
|
||||
# its own. kured itself stays 1/1 Running throughout.
|
||||
#
|
||||
# kured EXPOSES EXACTLY ONE METRIC. Read from cmd/kured/main.go:92-96 and :113 on 2026-10-17 — the
|
||||
# only `prometheus.MustRegister` in the tree is a GaugeVec `Subsystem: "kured"`, `Name:
|
||||
# "reboot_required"`, labelled `{node}`. So the full vocabulary available is
|
||||
# `kured_reboot_required{node="..."}`, 1 when the sentinel exists and 0 when it does not. Each rule
|
||||
# below is built from that one series plus `kube_node_info` (kube-state-metrics), and nothing else.
|
||||
# There is deliberately no rule over an invented metric name: this repo has shipped a
|
||||
# structurally-inert alert before (VeleroNoRecentBackup over an absent series, which is why
|
||||
# platform/monitoring/tests/run.sh exists), and an alert that cannot fire is worse than no alert
|
||||
# because it also removes the suspicion that would have caught it.
|
||||
#
|
||||
# THESE RULES LIVE HERE, NOT IN platform/monitoring/alerts.yaml, for two reasons. (1) The unit-test
|
||||
# harness treats a `<manifest>.promtool-test.yaml` COMPANION as covering just its sibling, whereas a
|
||||
# `tests/*.test.yaml` SUITE covers every PrometheusRule in its directory — so shipping them beside
|
||||
# the component keeps the test obligation local and legible (the platform/cnpg precedent). (2)
|
||||
# platform/monitoring/alerts.yaml is concurrently edited by other open branches; adding 3 rules to
|
||||
# its tail would create a merge conflict for no benefit.
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: kured
|
||||
namespace: kured
|
||||
labels:
|
||||
app.kubernetes.io/name: kured
|
||||
spec:
|
||||
groups:
|
||||
- name: platform-health.node-reboots
|
||||
rules:
|
||||
# ── 1. IS THE MECHANISM ALIVE AT ALL? ────────────────────────────────────────────────
|
||||
- alert: KuredNotReporting
|
||||
# `absent()` yields a series only when the ENTIRE vector is empty, so this fires when no
|
||||
# node at all is publishing — kured gone, or its Service/ServiceMonitor/NetworkPolicy
|
||||
# regressed so nothing is scraped. It stays quiet while a single node is mid-reboot
|
||||
# (the other two still report), which is what makes it specific to a wiring failure
|
||||
# rather than to a reboot in progress.
|
||||
#
|
||||
# THIS IS THE RULE THAT GUARDS THE OTHER TWO. Both of them read `kured_reboot_required`,
|
||||
# so if that series stops existing they go quiet and look like good news. This one turns
|
||||
# the absence itself into the signal. The specific regression it is aimed at: the kured
|
||||
# chart defaults `service.create` and `metrics.create` to FALSE, and Longhorn has already
|
||||
# demonstrated on this cluster (see the companion note in
|
||||
# conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh) that a
|
||||
# NetworkPolicy can let targets be DISCOVERED while every scrape times out and the TSDB
|
||||
# stays empty.
|
||||
expr: absent(kured_reboot_required)
|
||||
# 30m absorbs a HelmRelease upgrade rolling the DaemonSet and a Prometheus restart,
|
||||
# without tolerating a real wiring break for more than one quiet night's worth of time.
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: kured is publishing no reboot_required series on any node
|
||||
description: >-
|
||||
No node is reporting kured_reboot_required, so the reboot automation is either not
|
||||
running or not being scraped. While this is true, OS reboots are not happening and
|
||||
the two alerts that would say so (KuredRebootPending, KuredNodeCoverageGap) are
|
||||
themselves inert. Nodes will silently accumulate pending kernel upgrades.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-reboot-automation"
|
||||
|
||||
# ── 2. IS IT ALIVE ON *EVERY* NODE? ──────────────────────────────────────────────────
|
||||
- alert: KuredNodeCoverageGap
|
||||
# A per-node agent that silently covers only some nodes is the subtler half of the
|
||||
# liveness question, and `absent()` cannot see it. kured publishes one series per node it
|
||||
# runs on; kube-state-metrics publishes one `kube_node_info` per node that exists
|
||||
# (measured 2026-10-17: 3, matching the 3 nodes). A shortfall means at least one node has
|
||||
# no reboot automation — most likely a new or re-provisioned node carrying a taint this
|
||||
# DaemonSet does not tolerate (it declares none, because no prod node is tainted today).
|
||||
# That node would then never reboot, with nothing else to indicate it.
|
||||
expr: count(kured_reboot_required) < count(kube_node_info)
|
||||
# 2h, chosen to sit comfortably ABOVE the duration of a healthy reboot. A node being
|
||||
# rebooted stops publishing for a few minutes and legitimately creates a shortfall; a
|
||||
# node that is still missing after two hours is either genuinely uncovered or has hung on
|
||||
# reboot, and both of those want a human.
|
||||
for: 2h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: kured is not running on every node
|
||||
description: >-
|
||||
Fewer nodes are reporting kured_reboot_required than exist in the cluster, so at
|
||||
least one node has no reboot automation and will never be rebooted by it. Most
|
||||
likely a node carries a taint this DaemonSet does not tolerate, or has hung during
|
||||
a reboot.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-reboot-automation"
|
||||
|
||||
# ── 3. IS A REBOOT STUCK? ────────────────────────────────────────────────────────────
|
||||
- alert: KuredRebootPending
|
||||
# A node has wanted a reboot for longer than a full maintenance window plus margin, so
|
||||
# the reboot was OFFERED and did not happen. The causes are all real and all invisible
|
||||
# otherwise: the Prometheus alert gate is blocking on a degraded Longhorn volume that is
|
||||
# not recovering; the drain hit `drainTimeout` against a 0-disruption PDB; another node
|
||||
# holds the cluster lock and never released it (`lockTtl: 0` means a lock held by a hung
|
||||
# node is held FOREVER — that is deliberate, see helmrelease.yaml, and this is the alert
|
||||
# that makes the resulting stall visible).
|
||||
expr: max by (node) (kured_reboot_required) == 1
|
||||
# 36h, and the window being DAILY is what makes this number meaningful — see the window
|
||||
# block in helmrelease.yaml. With a 03:00-06:00 window every day, a pending reboot waits
|
||||
# at most ~24h legitimately, so 36h means at least one full window elapsed without the
|
||||
# reboot completing. On a weekly window this threshold would have to exceed 7 days, which
|
||||
# is longer than this cluster's 7d Prometheus retention and is reset by any Prometheus
|
||||
# restart past the 1h for-outage-tolerance — i.e. unreliable in the one direction that
|
||||
# matters. Note this alert can still be LATE for that reason, but it cannot be spurious.
|
||||
for: 36h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: Node {{ $labels.node }} has needed a reboot for over 36h
|
||||
description: >-
|
||||
kured reports a pending reboot on {{ $labels.node }} that has survived at least one
|
||||
full 03:00-06:00 maintenance window. The reboot is being blocked, not merely
|
||||
waiting: likely a degraded Longhorn volume holding the Prometheus alert gate, a
|
||||
drain timing out against a PodDisruptionBudget, or a stale cluster lock held by a
|
||||
node that did not come back. The kernel or security update that needed the reboot is
|
||||
installed but NOT running.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-reboot-automation"
|
||||
@@ -0,0 +1,289 @@
|
||||
---
|
||||
# kured (Kubernetes Reboot Daemon) — coordinated, one-node-at-a-time reboots of the three prod
|
||||
# control-plane nodes.
|
||||
#
|
||||
# ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# READ THIS FIRST: kured DOES NOT DECIDE *WHETHER* TO REBOOT. IT ONLY DECIDES *WHEN* AND *HOW*.
|
||||
# ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# kured reboots a node if and only if the file `/var/run/reboot-required` exists on it. On these
|
||||
# nodes exactly one thing creates that file: a deliberate operator run of
|
||||
# `playbooks/node-baseline.yml --tags kernel` (which advances the backports kernel). The
|
||||
# unattended-upgrades role this PR also adds installs SECURITY updates only and BLACKLISTS
|
||||
# `linux-image-*`/`linux-headers-*`, so it can never produce a reboot-required sentinel on its own —
|
||||
# see infra/ansible/roles/unattended_upgrades/defaults/main.yml for that argument in full.
|
||||
#
|
||||
# That is the whole answer to this repo's two standing objections to reboot automation:
|
||||
#
|
||||
# * infra/ansible/roles/node_baseline/tasks/main.yml:61-67 — "IT DOES NOT REBOOT." k3s-prod-2 hung
|
||||
# on a warm reboot on 2026-09-03 (powered, off the LAN ~38 minutes, physical power button
|
||||
# required), so rebooting was made "a SERIALIZED, SUPERVISED operator step ... and a role cannot
|
||||
# enforce that." STILL TRUE, and nothing here changes it: node_baseline continues not to reboot.
|
||||
# * docs/plans/cluster-upgrade-plan.md:291-299 — system-upgrade-controller was rejected because it
|
||||
# "automates away the human gate between nodes", and the operator's hard requirement is an
|
||||
# explicit PASS/STOP verification BETWEEN every node.
|
||||
#
|
||||
# THE HUMAN GATE IS PRESERVED, AND IT IS THE SENTINEL. The operator decides whether a kernel moves
|
||||
# and on which hosts (`--limit k3s-prod-1`), which is what writes the sentinel on those hosts and
|
||||
# only those hosts. kured then performs the cordon/drain/reboot/uncordon that the operator would
|
||||
# otherwise have typed by hand — the error-prone mechanical half, including the ordering rule
|
||||
# node_baseline's own message has to shout ("NEVER node 2 first"). Run the kernel play one host at a
|
||||
# time and the operator keeps per-node PASS/STOP control exactly as before; run it on all three and
|
||||
# kured still does them strictly one at a time, just in an order it chooses. Both are safe for
|
||||
# quorum; only the first preserves the ordering preference. See runbooks/node-reboot-automation.md.
|
||||
#
|
||||
# cluster-upgrade-plan.md:329-330 and :545-548 name the condition under which its own rejection
|
||||
# reverses — "SUC becomes viable once the cluster is on fast disks ... where drain-driven rebuild is
|
||||
# cheap" — and prod is that cluster. This component takes that clause up, but NOT its drain
|
||||
# assumption: see the Longhorn section below, where drain-driven rebuild is measurably NOT free even
|
||||
# on SSD, and is gated on explicitly rather than waved through.
|
||||
#
|
||||
# WHAT IS STILL UNMITIGATED, STATED PLAINLY: kured cannot press a power button. If a node hangs on
|
||||
# warm reboot the way k3s-prod-2 did in September, kured will not recover it. The blast radius is
|
||||
# bounded by design rather than by luck — `concurrency: 1` plus `lockTtl: 0` means the hung node
|
||||
# keeps the cluster lock forever, so NO second node is ever rebooted behind it; etcd keeps 2/3 quorum
|
||||
# and the cluster stays up, degraded, until a human attends. The KuredRebootPending alert in
|
||||
# alerts.yaml is what tells them. That is strictly better than the previous automation story (none)
|
||||
# but it is NOT the same as "safe to ignore at 3am on a Sunday".
|
||||
apiVersion: helm.toolkit.fluxcd.io/v2
|
||||
kind: HelmRelease
|
||||
metadata:
|
||||
name: kured
|
||||
namespace: kured
|
||||
spec:
|
||||
interval: 30m
|
||||
chart:
|
||||
spec:
|
||||
chart: kured
|
||||
# Chart 6.1.0 -> appVersion 1.23.0 (published 2026-06-30; newest of the 80 entries in
|
||||
# https://kubereboot.github.io/charts/index.yaml, read 2026-10-17).
|
||||
#
|
||||
# THE `<6.2.0` CEILING IS A DELIBERATE CHOICE HERE, NOT THE REFLEX IT IS ELSEWHERE. This PR's
|
||||
# own audit found that 9 of 12 HelmReleases in this repo pin a minor-locking range, and that
|
||||
# this is a MAIN CAUSE of pin rot: Renovate can then only ever offer patches inside a line it
|
||||
# has been told not to leave (kube-prometheus-stack is stuck 4 chart majors back that way).
|
||||
# For a component whose job is to reboot control-plane nodes, that ceiling is worth paying on
|
||||
# purpose: a kured MINOR may change drain or lock semantics, and this is the one component
|
||||
# where "a human looked at the changelog" should be a precondition, not a nice-to-have.
|
||||
# Renovate will raise the minor as a non-automerged PR (see the kured packageRule in
|
||||
# renovate.json), so the ceiling is a review prompt, not a wall.
|
||||
version: ">=6.1.0 <6.2.0"
|
||||
sourceRef:
|
||||
kind: HelmRepository
|
||||
name: kured
|
||||
namespace: kured
|
||||
install:
|
||||
remediation:
|
||||
retries: 3
|
||||
upgrade:
|
||||
remediation:
|
||||
retries: 3
|
||||
values:
|
||||
# ── IDENTITY / PLACEMENT ───────────────────────────────────────────────────────────────────
|
||||
# No tolerations. Measured 2026-10-17: all three prod nodes carry `spec.taints` EMPTY (k3s does
|
||||
# not taint its servers), so a DaemonSet reaches every node untainted. If a future node IS
|
||||
# tainted, kured silently stops covering it and that node then never reboots — invisible,
|
||||
# because the other two keep reporting. KuredNodeCoverageGap in alerts.yaml is the detector: it
|
||||
# compares kured's per-node series count against the node count rather than merely asking
|
||||
# whether kured exists.
|
||||
priorityClassName: system-node-critical
|
||||
# The chart's defaults are kept: `hostPID: true` and `containerSecurityContext.privileged: true`
|
||||
# with `readOnlyRootFilesystem: true`. Both are load-bearing and neither is reducible:
|
||||
# `rebootMethod: command` runs the reboot by nsenter'ing /proc/1/ns/mnt, which needs the host PID
|
||||
# namespace to see PID 1 and the privilege to enter its mount namespace. The documented
|
||||
# lower-privilege alternative (privileged:false + CAP_KILL) applies ONLY to
|
||||
# `rebootMethod: signal`, which signals a host-side kured-reboot.service that does not exist on
|
||||
# these nodes. Choosing `signal` would mean shipping and maintaining that unit via Ansible for a
|
||||
# capability reduction on a DaemonSet that is already cluster-admin-adjacent by function; it is a
|
||||
# reasonable future hardening step and is explicitly NOT taken here.
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
# No CPU limit on purpose (the repo's node-pressure work treats CPU limits on tiny system
|
||||
# agents as a throttling hazard). Memory is capped: kured holds no state beyond one
|
||||
# annotation and a gauge, and 128Mi is ~4x its idle RSS upstream reports.
|
||||
memory: 128Mi
|
||||
|
||||
configuration:
|
||||
# ── THE QUORUM CONSTRAINT: TWO NODES MUST NEVER BE DOWN AT ONCE ──────────────────────────
|
||||
# All three prod nodes are control-plane + etcd (measured 2026-10-17: every node reports
|
||||
# `control-plane,etcd`). etcd quorum is 2 of 3, so one node down is degraded-but-up and two
|
||||
# nodes down is a LOST CLUSTER that does not self-heal. Everything in this block exists to
|
||||
# make the second case structurally unreachable rather than merely unlikely.
|
||||
#
|
||||
# concurrency: 1 — kured's cluster-wide lock is a single annotation
|
||||
# (`weave.works/kured-node-lock`) on kured's OWN DaemonSet object, and a node reboots only
|
||||
# while it holds that lock. One lock + concurrency 1 = one reboot, enforced by the API server's
|
||||
# optimistic concurrency rather than by timing. This is the chart default; it is written out
|
||||
# explicitly because it is the single most important value in this file.
|
||||
concurrency: 1
|
||||
# lockTtl: 0 (DISABLED) — THIS IS A SAFETY DECISION, NOT A DEFAULT WE FORGOT TO TUNE.
|
||||
# A TTL makes the lock EXPIRE, which exists for the case "the lock holder died and we want
|
||||
# reboots to continue". On a 3-node etcd cluster that is precisely the catastrophe: if node A
|
||||
# has hung on a warm reboot (which has happened here — see the header) and its lock expires,
|
||||
# node B becomes eligible and takes the cluster from 2/3 to 1/3. We would rather the reboot
|
||||
# cycle STOP FOREVER and page a human than have it step over a corpse. The cost is that a
|
||||
# genuinely stale lock needs manual clearing; runbooks/node-reboot-automation.md has the
|
||||
# procedure, and KuredRebootPending is the alert that surfaces it.
|
||||
lockTtl: 0
|
||||
# lockReleaseDelay: 30m — hold the lock for 30 minutes AFTER this node is back and uncordoned,
|
||||
# before any other node may take it. See the Longhorn section: this is the STRUCTURAL half of
|
||||
# the inter-node gate, deliberately not dependent on an alert's `for:` timer having latched.
|
||||
lockReleaseDelay: 30m
|
||||
|
||||
# ── THE MAINTENANCE WINDOW: 03:00-06:00 EVERY DAY, America/New_York ──────────────────────
|
||||
# The timezone and the 03:00-06:00 band match renovate.json's existing
|
||||
# `"schedule": ["after 3am and before 6am"]` / `"timezone": "America/New_York"`, so every
|
||||
# unattended change in this repo lands in the same quiet 3-hour window.
|
||||
#
|
||||
# DAILY, NOT WEEKLY, AND THE REASON IS THE ALERT — not impatience. A weekly (`- sun`) window
|
||||
# was the first draft and it breaks the only alert that can tell you this mechanism has
|
||||
# stalled. With a Sunday-only window, a kernel upgraded on a Monday legitimately waits six
|
||||
# days, so "a reboot has been pending too long" cannot be distinguished from "it is Tuesday"
|
||||
# until the threshold exceeds a full 7-day cycle. That forces `for: 8d`+, which is longer than
|
||||
# this cluster's Prometheus retention (MONITORING_PROMETHEUS_RETENTION is 7d) and is reset by
|
||||
# any Prometheus restart beyond the 1h default for-outage-tolerance — i.e. the alert would be
|
||||
# unreliable in exactly the direction that matters. A daily window caps the legitimate wait at
|
||||
# under 24h, which makes KuredRebootPending's `for: 36h` both robust and meaningful: 36 hours
|
||||
# with a sentinel still set means at least one full window passed and SOMETHING BLOCKED IT.
|
||||
#
|
||||
# Daily does NOT weaken the quorum guarantee — that is `concurrency: 1` plus the lock, not the
|
||||
# calendar. It does mean a 3-node kernel rollout can finish in one to three nights instead of
|
||||
# three weeks, with each node still gated on the previous one's Longhorn rebuilds (below).
|
||||
rebootDays:
|
||||
- su
|
||||
- mo
|
||||
- tu
|
||||
- we
|
||||
- th
|
||||
- fr
|
||||
- sa
|
||||
startTime: "03:00"
|
||||
endTime: "06:00"
|
||||
timeZone: America/New_York
|
||||
# How often to re-evaluate (sentinel + window + alerts + lock). 1h is the chart default; at a
|
||||
# 3h window that is three evaluation opportunities per node per night.
|
||||
period: 1h
|
||||
|
||||
# ── THE SENTINEL ─────────────────────────────────────────────────────────────────────────
|
||||
# Written ONLY by a deliberate operator kernel upgrade (see the header). `update-notifier-common`
|
||||
# — the package that owns /etc/kernel/postinst.d and actually creates this file on Debian — was
|
||||
# measured ABSENT on all three nodes on 2026-10-17, which is why the unattended_upgrades role
|
||||
# installs it. Without that package nothing ever writes this path, kured correctly never
|
||||
# reboots, every probe looks healthy, and the entire mechanism is a silent no-op. That failure
|
||||
# mode is the reason KuredNotReporting exists.
|
||||
#
|
||||
# THE PATH kured LOGS IS NOT THIS PATH, and that trips people debugging it. The chart
|
||||
# defaults `useRebootSentinelHostPath: true` (kept), which bind-mounts the DIRNAME of this
|
||||
# value onto /sentinel read-only and renders the flag as
|
||||
# `--reboot-sentinel=/sentinel/reboot-required` (daemonset.yaml:134). So the container watches
|
||||
# /sentinel/reboot-required and the host file is /var/run/reboot-required. Same file, two
|
||||
# names. Set this value as the HOST path, as below.
|
||||
rebootSentinel: /var/run/reboot-required
|
||||
|
||||
# ── THE LONGHORN GATE: WHY POD READINESS IS NOT A SUFFICIENT INTER-NODE SIGNAL ────────────
|
||||
# Measured on prod 2026-10-17, and this is the finding that shaped this file:
|
||||
# * 25 Longhorn volumes, ALL `numberOfReplicas: 3`, all healthy, with exactly 25 replicas on
|
||||
# each of the 3 nodes. So there is no single-replica volume to lose today.
|
||||
# * BUT `replica-soft-anti-affinity: false` (replicas may not share a node) and
|
||||
# `disable-scheduling-on-cordoned-node: true`. With 3 nodes and 3 replicas, cordoning ONE
|
||||
# node leaves nowhere to place a replacement — so every drain takes ALL 25 volumes to
|
||||
# degraded 2/3 for the whole reboot.
|
||||
# * Recovery is slow and serialized: `replica-replenishment-wait-interval: 600` (10 minutes
|
||||
# before a rebuild even starts) and `concurrent-replica-rebuild-per-node-limit: 1`.
|
||||
# Therefore: if the next node is rebooted before rebuilds finish, those volumes go to 1/3, and
|
||||
# the drain after that genuinely CAN hit Longhorn's `node-drain-policy:
|
||||
# block-if-contains-last-replica` and block forever. The inter-node gate must be VOLUME
|
||||
# ROBUSTNESS, not "are the pods Ready again".
|
||||
#
|
||||
# It is gated twice, on purpose, because each half covers the other's blind spot:
|
||||
# 1. lockReleaseDelay: 30m (above) — unconditional. Covers the 600s replenishment wait with
|
||||
# margin and does not depend on any alert having fired.
|
||||
# 2. The Prometheus alert gate below — semantic. Covers a rebuild that takes LONGER than 30
|
||||
# minutes (25 serialized rebuilds on one node can), which a fixed delay cannot.
|
||||
prometheusUrl: http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090
|
||||
# Only consider alerts that are FIRING, not merely pending. A `pending` alert is a 10-minute
|
||||
# guess; blocking a weekly reboot on one would make the gate fire on noise.
|
||||
alertFiringOnly: true
|
||||
# ── alertFilterMatchOnly: true IS LOAD-BEARING. DO NOT DROP IT. ──────────────────────────
|
||||
# kured's two alert modes are near-opposites and the flag name does not advertise it:
|
||||
# * false (the chart default): alertFilterRegexp is an IGNORE list — kured blocks on ANY
|
||||
# firing alert except those matching.
|
||||
# * true: alertFilterRegexp is an ALLOW list — kured blocks ONLY on alerts that match.
|
||||
# We need `true`, because this cluster always has something firing. Measured via the Prometheus
|
||||
# API on 2026-10-17, these five were firing with no reboot pending and no end in sight:
|
||||
# CNPGPrimariesCoLocated, KubeDeploymentReplicasMismatch, KubeDeploymentRolloutStuck,
|
||||
# KubePodNotReady, OAuth2ProxyGateUnobserved (the last three all trace to the pm-agent pod,
|
||||
# ImagePullBackOff on a placeholder digest for 12 days). Under the default `false`, kured would
|
||||
# have been blocked by those on day one and would never have rebooted anything — a silent
|
||||
# no-op indistinguishable, from outside, from "no reboots were needed".
|
||||
alertFilterMatchOnly: true
|
||||
# The ALLOW list: the conditions under which rebooting ANOTHER node risks data or quorum.
|
||||
# Anchored and explicit — a prefix match here would silently re-admit the chronic five above.
|
||||
# LonghornVolumeDegraded — the inter-node gate (see above). `longhorn_volume_robustness
|
||||
# {state="degraded"} == 1`, for: 10m, in platform/monitoring/alerts.yaml.
|
||||
# LonghornVolumeFaulted — data already at risk; never reboot into this.
|
||||
# KubeNodeNotReady — a node is already down. Rebooting a second one loses quorum. This
|
||||
# is the single most important entry for the etcd constraint.
|
||||
# NodeExporterDown — a node we cannot see the health of. Treated as "assume the worst".
|
||||
# DELIBERATELY NOT INCLUDED: NodeEtcdRequestsSlow and the NodeIOPressure* family. They are
|
||||
# exactly the conditions you would want to pause for, but prod-1 has a documented history of
|
||||
# etcd WAL I/O starvation, so including them risks a permanent wedge that looks like safety.
|
||||
# Revisit once those are quiet for a sustained period; KuredRebootPending makes a wedge visible
|
||||
# either way.
|
||||
alertFilterRegexp: ^(LonghornVolumeDegraded|LonghornVolumeFaulted|KubeNodeNotReady|NodeExporterDown)$
|
||||
|
||||
# ── THE DRAIN ────────────────────────────────────────────────────────────────────────────
|
||||
# NOTE ON FLAGS THAT DO NOT EXIST: there is no `--ignore-daemonsets`, no
|
||||
# `--delete-emptydir-data` and no `--force` in kured 1.23.0. Read from
|
||||
# cmd/kured/main.go:438-449 on 2026-10-17, kured HARDCODES
|
||||
# `IgnoreAllDaemonSets: true`, `DeleteEmptyDirData: true` and `Force: true` in the drain
|
||||
# helper. Passing them as extraArgs would CrashLoop the DaemonSet on an unknown flag. This
|
||||
# matters because all three are genuinely REQUIRED here and it is easy to assume they must be
|
||||
# configured: 12 DaemonSets and 135 emptyDir-carrying pods were measured on 2026-10-17, and a
|
||||
# drain without those three settings refuses to start.
|
||||
#
|
||||
# drainTimeout: 15m — the chart/kured default is 0, meaning INFINITE, and infinite is the wrong
|
||||
# answer here. Measured 2026-10-17, six PodDisruptionBudgets currently allow ZERO voluntary
|
||||
# disruptions, and a plain drain loops on them indefinitely:
|
||||
# * 3x longhorn-system/instance-manager-* — one per node, `minAvailable: 1` against a
|
||||
# selector pinned to a single node's single pod, so disruptionsAllowed is STRUCTURALLY 0
|
||||
# forever. This is the mechanism by which Longhorn's node-drain-policy blocks: the
|
||||
# longhorn-manager deletes the node's instance-manager PDB only once it judges the drain
|
||||
# safe. The drain is SUPPOSED to wait on this one.
|
||||
# * 3x CNPG `<cluster>-primary` (keycloak-db, nextcloud-db, semaphore-db) — `minAvailable: 1`
|
||||
# against a selector matching exactly one pod (the primary), so again structurally 0 until
|
||||
# the CNPG operator switches the primary away. All three primaries were on k3s-prod-1.
|
||||
# With a timeout, a drain that is genuinely stuck ABORTS, the lock is released, and kured
|
||||
# retries next period instead of holding the cluster lock forever on a wedged node. 15m is
|
||||
# chosen to be longer than a CNPG switchover and longer than the `unifi` VM's eviction (below)
|
||||
# usually needs, while still being well inside the 3h window.
|
||||
drainTimeout: 15m
|
||||
# Taint a node PreferNoSchedule while its reboot is pending, so the scheduler stops placing new
|
||||
# pods on a node that is about to go away. With concurrency 1 this is a small win (there is no
|
||||
# second rebooting node to shed pods onto) but it is free and it shortens the drain.
|
||||
preferNoScheduleTaint: weave.works/kured-node-reboot
|
||||
# Record reboot state on the Node objects (`weave.works/kured-reboot-in-progress`,
|
||||
# `weave.works/kured-most-recent-reboot-needed`). Off by default. Turned on because the only
|
||||
# other observability kured offers is ONE gauge, and during an incident `kubectl get node -o
|
||||
# yaml` is the thing an operator can reach without Prometheus.
|
||||
annotateNodes: true
|
||||
logFormat: json
|
||||
|
||||
# ── METRICS: BOTH OF THESE MUST BE true OR THE ALERTS IN alerts.yaml CANNOT FIRE ───────────
|
||||
# The chart ships `service.create: false` AND `metrics.create: false`. Leaving either at its
|
||||
# default means no Service, or a ServiceMonitor with nothing to select, so `kured_reboot_required`
|
||||
# is never scraped — and a PrometheusRule over a series that no target produces is INERT: quiet
|
||||
# forever, indistinguishable from healthy. This repo has shipped that exact bug twice
|
||||
# (VeleroNoRecentBackup over an absent series; the inert gotk_* rules in
|
||||
# runbooks/image-automation.md), which is why platform/monitoring/tests/run.sh exists at all.
|
||||
# KuredNotReporting (alerts.yaml) is the standing detector for this pair regressing.
|
||||
# Prometheus selects ServiceMonitors cluster-wide: measured 2026-10-17, `serviceMonitorSelector`
|
||||
# and `serviceMonitorNamespaceSelector` are both `{}`, so no label plumbing is needed here.
|
||||
service:
|
||||
create: true
|
||||
port: 8080
|
||||
metrics:
|
||||
create: true
|
||||
interval: 60s
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
# HelmRepository for the official kured chart (github.com/kubereboot/charts). Renovate maintains the
|
||||
# pinned chart version in helmrelease.yaml via its `flux` manager, whose managerFilePatterns already
|
||||
# cover `platform/**` (renovate.json). Same Flux source pattern as platform/headlamp.
|
||||
apiVersion: source.toolkit.fluxcd.io/v1
|
||||
kind: HelmRepository
|
||||
metadata:
|
||||
name: kured
|
||||
namespace: kured
|
||||
spec:
|
||||
interval: 1h
|
||||
url: https://kubereboot.github.io/charts
|
||||
@@ -0,0 +1,36 @@
|
||||
---
|
||||
# platform/kured — the Kubernetes Reboot Daemon.
|
||||
#
|
||||
# Rendered by its OWN Flux Kustomization (clusters/base/kured.yaml, dependsOn platform), NOT by a
|
||||
# line in platform/kustomization.yaml. The reasoning is the same one the Cilium split-out wrote down
|
||||
# and intel-gpu-plugin reused: inside the shared `wait: true` platform tier, this component's health
|
||||
# would be indistinguishable from eleven others, the only emergency valve would be suspending the
|
||||
# whole tier, and that tier's 5m timeout is wrong for a 3-node DaemonSet rollout. It matters more
|
||||
# than usual here, because `platform` gates platform-config which gates every apps wave — so a wedged
|
||||
# reboot daemon placed there could stall every application in the cluster. A component whose entire
|
||||
# purpose is to make maintenance safer must not be able to cause an outage.
|
||||
#
|
||||
# `namespace: kured` applies to every resource below, so none of them repeats it. Keep it: the
|
||||
# CiliumNetworkPolicy in networkpolicy.yaml is a CR whose namespace kustomize would otherwise leave
|
||||
# to chance.
|
||||
#
|
||||
# NO SECRET OF ANY KIND. kured authenticates to the API server with its own ServiceAccount token and
|
||||
# talks to nothing off-cluster — no registry credential, no webhook URL, no notification token. So
|
||||
# neither clusters/dev nor clusters/prod adds a SOPS decryption patch for this tier (same as
|
||||
# homepage/runbooks/intel-gpu-plugin). If Slack or shoutrrr notifications are ever wanted, the chart
|
||||
# takes them via `configuration.notifyUrl`, and THAT would need a SOPS-encrypted Secret plus a
|
||||
# decryption patch — do not inline a webhook URL here.
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
namespace: kured
|
||||
resources:
|
||||
- namespace.yaml
|
||||
- helmrepository.yaml
|
||||
- helmrelease.yaml # chart installs the DaemonSet / RBAC / SA / Service / ServiceMonitor
|
||||
# default-deny baseline + the Prometheus scrape ingress + the apiserver/Prometheus egress.
|
||||
# The scrape ingress is NOT optional decoration: without it the ServiceMonitor's targets are
|
||||
# discovered and every scrape times out, leaving the TSDB empty and all three rules in alerts.yaml
|
||||
# permanently quiet. See the header in that file for the measured Longhorn precedent.
|
||||
- networkpolicy.yaml
|
||||
# Three rules over kured's single metric, with a companion promtool unit test beside them.
|
||||
- alerts.yaml
|
||||
@@ -0,0 +1,34 @@
|
||||
---
|
||||
# kured — namespace for the Kubernetes Reboot Daemon DaemonSet.
|
||||
#
|
||||
# PLATFORM INFRA, NOT AN APPLICATION NAMESPACE, and here that distinction is the difference between
|
||||
# a working component and an admission rejection. kured runs `hostPID: true` with a `privileged: true`
|
||||
# container (it `nsenter`s /proc/1/ns/mnt to run `/bin/systemctl reboot` on the host). Both are
|
||||
# forbidden by platform/kyverno/policies/restrict-host-namespaces.yaml — but that policy, like every
|
||||
# other policy in this repo, is scoped by
|
||||
# `namespaceSelector: matchLabels: homelab.keithjackson.dev/app-namespace: "true"`
|
||||
# and its own header states the carve-out: "platform components that do [need these] live in platform
|
||||
# namespaces that do NOT carry the app-namespace label, so they are out of scope."
|
||||
# Verified by reading the policies on this branch, not assumed:
|
||||
# * restrict-host-namespaces.yaml rule `disallow-host-namespaces` (hostPID)
|
||||
# * restrict-host-namespaces.yaml rule `disallow-privileged-containers` (privileged)
|
||||
#
|
||||
# We set the label EXPLICITLY to "false" rather than leaving it unset, following
|
||||
# platform/intel-gpu-plugin/namespace.yaml and platform/vpn-egress: an unset label and a deliberate
|
||||
# "false" are identical to the admission engine but not to a reviewer, and this component's
|
||||
# correctness depends entirely on the answer.
|
||||
#
|
||||
# NO PodSecurity labels. There is no cluster-wide PSA baseline in this repo — `pod-security.*`
|
||||
# appears only where a namespace opts IN (conformance/incluster, platform/kyverno/admission,
|
||||
# platform/pm-agent), all at `enforce: restricted`. Adding them here would reject this DaemonSet for
|
||||
# the same reasons Kyverno would. Leaving them off is the convention for a privileged platform
|
||||
# namespace (platform/vpn-ingress/namespace.yaml sets none either).
|
||||
#
|
||||
# NO `gateway-access` label. kured serves no HTTP to users and attaches no HTTPRoute; its only
|
||||
# listener is the :8080 metrics endpoint, reached in-cluster by Prometheus. See networkpolicy.yaml.
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: kured
|
||||
labels:
|
||||
homelab.keithjackson.dev/app-namespace: "false"
|
||||
@@ -0,0 +1,125 @@
|
||||
---
|
||||
# Default-deny baseline (R-NET-01): deny all ingress + egress, allow only DNS. `kured` is a PLATFORM
|
||||
# namespace, so Kyverno's generator — scoped to `app-namespace=true` — does not produce this; we ship
|
||||
# it explicitly, as crafty/headlamp/homepage/runbooks all do.
|
||||
#
|
||||
# THESE POLICIES DO APPLY TO kured, UNLIKE platform/vpn-ingress'. That component's identical-looking
|
||||
# baseline is honest-but-inert because its DaemonSet runs `hostNetwork: true` and therefore has no
|
||||
# Cilium endpoint for any policy to select. kured's chart defaults to `hostNetwork: false` and we
|
||||
# keep that default (it needs the host PID namespace to reboot, not the host network stack), so it
|
||||
# has a normal pod identity and every rule here is enforced.
|
||||
#
|
||||
# Cilium enforces the UNION of these policies (allow if ANY policy allows), so the CiliumNetworkPolicy
|
||||
# below adds the kube-apiserver path on top of this deny-all floor.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: default-deny
|
||||
namespace: kured
|
||||
labels:
|
||||
homelab.keithjackson.dev/baseline: default-deny
|
||||
spec:
|
||||
podSelector: {}
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress: []
|
||||
egress:
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
---
|
||||
# ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# PROMETHEUS MUST REACH :8080, OR EVERY RULE IN alerts.yaml IS INERT.
|
||||
# ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# This is not a theoretical hazard in this repo, it is a measured one. From the Longhorn entry in
|
||||
# conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh: with the ServiceMonitor
|
||||
# applied and no NetworkPolicy allowance, all three targets were DISCOVERED and every scrape failed
|
||||
# with "context deadline exceeded" after 10.0s — "a drop, not a refusal" — and the TSDB stayed
|
||||
# empty. Target discovery succeeding is therefore NOT evidence the scrape works, and the alerts
|
||||
# would have been quiet rather than broken. KuredNotReporting is the standing detector for exactly
|
||||
# this regression, and it can only work if this policy is correct.
|
||||
#
|
||||
# THE WHOLE `monitoring` NAMESPACE, NOT THE PROMETHEUS POD LABELS — following the same choice
|
||||
# longhorn/networkpolicy-metrics.yaml made and for the same reason: the prometheus-operator owns
|
||||
# those pod labels and rewrites them on upgrade (measured 2026-10-17, the pod carries
|
||||
# `app.kubernetes.io/name: prometheus` plus five operator-managed labels including a
|
||||
# controller-revision-hash), so selecting on them makes the scrape silently fail on a chart bump.
|
||||
# The namespace holds only the monitoring stack.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: kured-allow-prometheus-scrape
|
||||
namespace: kured
|
||||
labels:
|
||||
app.kubernetes.io/name: kured
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
# Set by the chart's `kured.matchLabels` helper (_helpers.tpl:70-71) with release name `kured`.
|
||||
app.kubernetes.io/name: kured
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: monitoring
|
||||
ports:
|
||||
- protocol: TCP
|
||||
# The container port the chart names `metrics` (daemonset.yaml:211-213), from
|
||||
# `configuration.metricsPort`, which we leave at the chart default of 8080 and mirror in
|
||||
# the HelmRelease's `service.port`.
|
||||
port: 8080
|
||||
---
|
||||
# Egress allows. kured talks to exactly two things and nothing else; both are required and the
|
||||
# component is useless without either.
|
||||
#
|
||||
# 1. THE KUBE-APISERVER — the lock annotation on its own DaemonSet (`weave.works/kured-node-lock`),
|
||||
# cordon/uncordon (`nodes` patch), the node annotations, listing pods, and creating
|
||||
# `pods/eviction` objects during the drain. Reached via the `kube-apiserver` Cilium entity
|
||||
# rather than an IP: on this cluster the API is fronted by a kube-vip VIP that moves between
|
||||
# nodes on every reboot, so any CIDR-based rule here would break during the exact operation
|
||||
# this component exists to perform. Same construct as platform/headlamp/networkpolicy.yaml.
|
||||
#
|
||||
# 2. PROMETHEUS, on 9090 — the alert gate. The HelmRelease sets
|
||||
# `prometheusUrl: http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090`
|
||||
# and `alertFilterMatchOnly: true`, so before taking the cluster lock kured asks Prometheus
|
||||
# whether any of LonghornVolumeDegraded / LonghornVolumeFaulted / KubeNodeNotReady /
|
||||
# NodeExporterDown is firing. IF THIS EGRESS IS BLOCKED, kured cannot evaluate the gate.
|
||||
# It treats the query error as "do not proceed" and skips the reboot — which is the correct
|
||||
# fail-safe direction, but it fails SILENTLY and indefinitely: reboots simply stop happening
|
||||
# and nothing goes red except, eventually, KuredRebootPending. Do not remove this rule
|
||||
# thinking it is only for metrics; metrics ingress is the policy above, this is the gate.
|
||||
#
|
||||
# NOTHING ELSE. No egress to the internet (kured pulls no content at runtime), no DNS beyond the
|
||||
# baseline above, no Longhorn or etcd access — kured reads Longhorn's health THROUGH Prometheus
|
||||
# alerts, never from Longhorn's own API, which is why that allow is absent here.
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: kured-allow-egress
|
||||
namespace: kured
|
||||
labels:
|
||||
app.kubernetes.io/name: kured
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: kured
|
||||
egress:
|
||||
- toEntities:
|
||||
- kube-apiserver
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: monitoring
|
||||
app.kubernetes.io/name: prometheus
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: "9090"
|
||||
protocol: TCP
|
||||
Reference in new issue
Block a user