Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
  jakman85andClaude Opus 5
|
dbdb215651 | ||
|
jakman85andClaude Opus 5
|
a2440a3125 |
No files matched your search
@@ -0,0 +1,173 @@
|
||||
# Renovate — the dependency/image update automation. SCHEDULED, self-hosted against this Gitea.
|
||||
#
|
||||
# ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# WHY THIS FILE EXISTS: renovate.json HAS BEEN IN THIS REPO AND HAS NEVER ONCE RUN.
|
||||
# ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# Measured 2026-10-17: `renovate.json` is present, complete and thoughtfully written — it declares
|
||||
# managers, schedules, automerge classes and grouping. Nothing executes it. There is no Renovate
|
||||
# workflow (this is the first), no Renovate CronJob or Deployment in the cluster
|
||||
# (`kubectl get cronjob,deploy,job -A | grep -i renovate` -> nothing), and no bot account doing it
|
||||
# from outside. Meanwhile SEVEN files carry comments asserting "Renovate maintains this pin"
|
||||
# (platform/homepage/deployment.yaml:40, runbooks/serving/deployment.yaml:87,
|
||||
# platform/crafty/deployment.yaml:63, platform/vpn-ingress/daemonset.yaml:259,
|
||||
# platform/keycloak/postgres-cluster.yaml:75, apps/wave-0/nextcloud-db.yaml:302, and
|
||||
# platform/headlamp/helmrepository.yaml:2). Those comments have been false for the life of the file.
|
||||
#
|
||||
# The cost is measurable. With nothing bumping anything: the six per-app oauth2-proxy auth sidecars
|
||||
# sit on v7.8.1 while the platform oauth2-proxy in the SAME cluster runs v7.15.5; Keycloak (the IdP)
|
||||
# runs 26.0.6, a November-2024 release, 8 minors behind; Velero runs a RELEASE CANDIDATE
|
||||
# (velero-plugin-for-aws v1.14.3-rc.2) with a stable available; the CNPG Postgres image is 16.4,
|
||||
# missing every 16.5+ security release. A pin nothing maintains is not a pin, it is a fossil.
|
||||
#
|
||||
# ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# WHY RENOVATE AND NOT FLUX IMAGE-AUTOMATION (the decision, and it reverses ADR-0009's split)
|
||||
# ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# ADR-0009 divided the work: Renovate owns Helm chart/dependency bumps, Flux image-automation owns
|
||||
# per-service container image TAGS, deliberately non-overlapping. The second half of that split
|
||||
# cannot be delivered on this repo's terms, for three measured reasons:
|
||||
#
|
||||
# 1. IT WOULD BYPASS THE MERGE GATE, WHICH CLAUDE.md §1 FORBIDS. ImageUpdateAutomation does not
|
||||
# open pull requests; it COMMITS AND PUSHES. Its default target is the source's own branch —
|
||||
# `main` — so it would write straight past the PR flow. It can be pointed at a side branch
|
||||
# instead, but then nothing opens the PR and bumps silently accumulate on a branch nobody
|
||||
# merges. Either way a human or a second bot has to close the loop.
|
||||
# 2. ON THIS CLUSTER IT CANNOT EVEN DO THAT. Measured 2026-10-17: `main` is protected with
|
||||
# `user_can_push: false`, and the `flux-system` GitRepository has NO `spec.secretRef` at all —
|
||||
# the source is anonymous and read-only. Enabling write-back means minting an in-cluster git
|
||||
# WRITE credential for the repository that defines the cluster, which is a materially larger
|
||||
# blast radius than a CI-scoped token.
|
||||
# 3. RENOVATE ALREADY OPENS PRs NATIVELY, so every change it proposes goes through the gate with
|
||||
# no extra machinery.
|
||||
#
|
||||
# So: Renovate owns BOTH halves, and Flux image-automation stays disabled. Note this is a change of
|
||||
# plan, not a change of state — image-automation has been commented out at
|
||||
# platform/kustomization.yaml:52 since it was authored, and the cluster has ZERO
|
||||
# ImageRepository/ImagePolicy/ImageUpdateAutomation objects (measured). Its two controllers ARE
|
||||
# installed and have idled for 43 days. Nothing is being torn down here; a path that was never
|
||||
# taken is being formally closed, and renovate.json's `docker`/`kubernetes` managers — disabled
|
||||
# purely to avoid fighting a controller that never ran — are switched on to fill the gap.
|
||||
#
|
||||
# ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# THIS WORKFLOW MUST NOT BE ABLE TO SUCCEED SILENTLY
|
||||
# ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# A SKIPPED Gitea job reports `success` (CLAUDE.md §2), so "green" is not evidence this ran. Two
|
||||
# guards:
|
||||
# * The preflight step FAILS HARD (exit 1) if RENOVATE_TOKEN is absent, rather than skipping.
|
||||
# A missing credential on a dependency bot is indistinguishable from "nothing to update", and
|
||||
# that is precisely how renovate.json stayed inert for months.
|
||||
# * The summary step records what ran into the job summary unconditionally.
|
||||
# What this still does NOT give you is an alert when the SCHEDULE stops firing. See the handoff
|
||||
# entry's "What I did NOT verify" — that gap is stated, not papered over.
|
||||
#
|
||||
# OPERATOR PREREQUISITE — ONE SECRET, NOT COMMITTED HERE:
|
||||
# `RENOVATE_TOKEN` — a Gitea access token for the bot account, scopes `write:repository` and
|
||||
# `read:user`. Set it as an Actions secret on this repository. Nothing in this repo mints,
|
||||
# stores or references its value; it is consumed by name only.
|
||||
# Until it exists, this workflow FAILS LOUDLY on every scheduled run. That is intended: a
|
||||
# visible red is the correct state for "automation configured but not credentialed".
|
||||
#
|
||||
# THE FIRST RUN WILL WANT TO OPEN A LOT OF PRs. ~30 dependencies are behind. renovate.json caps the
|
||||
# blast radius (`prConcurrentLimit: 5`, `prHourlyLimit: 2`), but run it manually with
|
||||
# `dry_run: true` FIRST and read the log before letting the schedule loose. That is what
|
||||
# workflow_dispatch's default is for.
|
||||
name: renovate
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# 03:15 UTC daily. renovate.json ALSO carries `"schedule": ["after 3am and before 6am"]` in
|
||||
# America/New_York, and that inner schedule is the authoritative one — Renovate re-checks it and
|
||||
# will decline to open PRs outside it. The two are deliberately both present: this cron decides
|
||||
# when the process RUNS, renovate.json decides when it may ACT. Keeping the cron slightly ahead
|
||||
# of the repo window means a run is always in flight when the window opens.
|
||||
- cron: "15 3 * * *"
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
dry_run:
|
||||
description: "Dry run (no PRs, no branches). Leave true for the first run."
|
||||
required: false
|
||||
default: "true"
|
||||
log_level:
|
||||
description: "Renovate LOG_LEVEL (info | debug)"
|
||||
required: false
|
||||
default: "info"
|
||||
|
||||
concurrency:
|
||||
# Never two Renovate runs at once: concurrent runs race on the same branches and can open
|
||||
# duplicate PRs. Not cancel-in-progress — a half-finished Renovate run can leave a branch pushed
|
||||
# with no PR, so let the first one finish.
|
||||
group: renovate
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
renovate:
|
||||
runs-on: ubuntu-latest
|
||||
# Renovate scans every manager across the whole repo; 40m is generous for this repo's size and
|
||||
# well under a runner's patience. Without a timeout a wedged run holds the `renovate`
|
||||
# concurrency group and silently blocks every subsequent scheduled run.
|
||||
timeout-minutes: 40
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Preflight - the token must exist, and its absence must be LOUD
|
||||
env:
|
||||
RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [ -z "${RENOVATE_TOKEN:-}" ]; then
|
||||
echo "::error::RENOVATE_TOKEN is not set on this repository." >&2
|
||||
cat >&2 <<'MSG'
|
||||
|
||||
Renovate cannot run without a Gitea access token, and this step fails rather than
|
||||
skipping ON PURPOSE: a skipped Gitea job reports `success`, so skipping here would make
|
||||
"dependency automation is not running" look exactly like "everything is up to date".
|
||||
That is the failure mode that let renovate.json sit inert in this repo for months while
|
||||
seven files claimed "Renovate maintains this pin".
|
||||
|
||||
TO FIX: create a Gitea access token for the bot account with scopes
|
||||
`write:repository` and `read:user`, then add it to this repository as the Actions
|
||||
secret RENOVATE_TOKEN. Do not commit it anywhere.
|
||||
MSG
|
||||
exit 1
|
||||
fi
|
||||
echo "RENOVATE_TOKEN is present (value not printed)."
|
||||
|
||||
- name: Run Renovate
|
||||
uses: renovatebot/[email protected]
|
||||
with:
|
||||
# Pinned, and Renovate maintains this pin itself: renovate.json's enabledManagers
|
||||
# includes `github-actions`, and `helpers:pinGitHubActionDigests` is in its `extends`.
|
||||
# This is the one dependency in the repo whose updater is its own updater.
|
||||
configurationFile: renovate.json
|
||||
token: ${{ secrets.RENOVATE_TOKEN }}
|
||||
env:
|
||||
# Self-hosted Gitea. `platform` + `endpoint` were the two values renovate.json's own
|
||||
# header said were "set at runtime by the operator's renovate runner" — this is that
|
||||
# runner, so they are set here rather than in the repo config, which keeps renovate.json
|
||||
# portable and free of this instance's URL.
|
||||
RENOVATE_PLATFORM: gitea
|
||||
RENOVATE_ENDPOINT: https://gitea.home.keithjackson.dev/api/v1
|
||||
RENOVATE_REPOSITORIES: Home-Network/home-network-k3s-impl
|
||||
# Renovate must never push to a protected branch; it works on its own branches and opens
|
||||
# PRs against this base. Stated explicitly rather than inherited, because this is the
|
||||
# property that keeps the whole mechanism inside CLAUDE.md §1's branch -> PR -> gate path.
|
||||
RENOVATE_BASE_BRANCHES: main
|
||||
RENOVATE_ONBOARDING: "false"
|
||||
RENOVATE_REQUIRE_CONFIG: "required"
|
||||
RENOVATE_DRY_RUN: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run == 'true' && 'full' || '' }}
|
||||
LOG_LEVEL: ${{ github.event_name == 'workflow_dispatch' && inputs.log_level || 'info' }}
|
||||
|
||||
- name: Summary
|
||||
if: always()
|
||||
run: |
|
||||
{
|
||||
echo "## Renovate"
|
||||
echo
|
||||
echo "- trigger: \`${{ github.event_name }}\`"
|
||||
echo "- dry run: \`${{ github.event_name == 'workflow_dispatch' && inputs.dry_run || 'false' }}\`"
|
||||
echo "- outcome: \`${{ job.status }}\`"
|
||||
echo
|
||||
echo "Renovate owns BOTH Helm chart bumps and container image tags here; Flux"
|
||||
echo "image-automation is deliberately disabled (platform/kustomization.yaml)."
|
||||
echo "Every change it proposes arrives as a pull request and must pass the merge gate."
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
@@ -0,0 +1,98 @@
|
||||
---
|
||||
# Flux Kustomization: kured (Kubernetes Reboot Daemon) — reconciles ./platform/kured.
|
||||
# Performs the cordon / drain / reboot / uncordon of a node that has a pending OS reboot, strictly
|
||||
# one node at a time, inside a nightly window, gated on Longhorn volume health. Design rationale,
|
||||
# every tuned value, and the honest list of what is NOT mitigated: platform/kured/helmrelease.yaml.
|
||||
#
|
||||
# WHY ITS OWN KUSTOMIZATION, AND NOT ONE LINE IN platform/kustomization.yaml
|
||||
#
|
||||
# The same argument the Cilium split-out recorded in platform/kustomization.yaml and intel-gpu-plugin
|
||||
# reused, except the consequence is worse here than for either of them:
|
||||
#
|
||||
# * `platform` is `wait: true` over twelve-plus components, so its Ready condition cannot tell you
|
||||
# "the reboot daemon is wedged" from "longhorn is slow".
|
||||
# * The only emergency valve would be `flux suspend kustomization platform` — freezing storage,
|
||||
# TLS, DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a DaemonSet.
|
||||
# * That tier's 10m interval / 5m timeout are called out there as WRONG for a 3-node DaemonSet
|
||||
# rollout. This is a 3-node DaemonSet rollout.
|
||||
# * And `platform` gates platform-config, which gates apps-wave-0 (wait:true), which gates
|
||||
# apps-wave-1. A reboot daemon that failed to go healthy inside that tier would stall EVERY
|
||||
# APPLICATION in the cluster on "dependency not ready". A component added to make maintenance
|
||||
# safer must not acquire the ability to cause an outage; that trade is the whole argument.
|
||||
#
|
||||
# dependsOn: [platform] — AND THIS ONE IS A REAL EDGE, NOT CONVENTION
|
||||
#
|
||||
# Not `cilium` (what intel-gpu-plugin uses) and not `platform-config` (what the leaf UIs use). It is
|
||||
# `platform`, for two independent reasons:
|
||||
#
|
||||
# 1. CRD ORDERING (F-MON1). platform/kured ships a PrometheusRule, and the chart generates a
|
||||
# ServiceMonitor. Both are CRs whose CRDs (monitoring.coreos.com) are installed by
|
||||
# kube-prometheus-stack INSIDE the `platform` tier. Flux dry-runs every object in a pass before
|
||||
# applying any of them, so a pass containing an unknown kind ABORTS — the failure that made the
|
||||
# CDI StorageProfiles a true deadlock on prod on 2026-09-05 (see
|
||||
# clusters/base/platform-config-cdi.yaml). Depending on `platform` guarantees the CRDs exist
|
||||
# before this pass is attempted.
|
||||
# 2. FUNCTIONAL PREREQUISITE. kured's reboot gate QUERIES Prometheus
|
||||
# (`prometheusUrl` in the HelmRelease). Prometheus lives in platform/monitoring. If kured ran
|
||||
# first it would get connection errors, treat the gate as unevaluable and skip every reboot —
|
||||
# the correct fail-safe direction, but silent. Longhorn, whose volume health IS the gate, is
|
||||
# also in that tier.
|
||||
#
|
||||
# It does NOT need platform-config: no Keycloak, no TLS, no split-horizon DNS, no Gateway, no
|
||||
# HTTPRoute. kured serves no HTTP to users.
|
||||
#
|
||||
# wait: true — A HEALTH SIGNAL, NOT A GATE. Following intel-gpu-plugin rather than the leaf-UI
|
||||
# template. A leaf UI is legitimately not-Ready for long stretches of a cold start while it waits on
|
||||
# Keycloak and DNS, so its health is uninformative; kured has no such dependency — it is a DaemonSet
|
||||
# that opens a socket and watches a file, so on healthy nodes it is Ready in seconds and not-Ready
|
||||
# genuinely means BROKEN. This turns `flux get kustomization kured` into a one-line answer to "is
|
||||
# reboot automation live on all three nodes", which is the question the operator must be able to
|
||||
# answer before trusting a kernel upgrade to it.
|
||||
#
|
||||
# NOTHING IN THE GRAPH dependsOn THIS, so wait:true gates nothing — the blast radius of an unhealthy
|
||||
# kured is this Kustomization's own Ready condition. Do not add `dependsOn: [kured]` anywhere: that
|
||||
# would hand a node-rebooting DaemonSet the power to stall whatever depends on it, which is precisely
|
||||
# what the placement above exists to prevent.
|
||||
#
|
||||
# prune: true is safe and deliberate. If this Kustomization is deleted, kured's DaemonSet goes with
|
||||
# it and reboots simply stop happening — nodes are left uncordoned and running. The one caveat is
|
||||
# that pruning mid-reboot would leave a node CORDONED with no daemon to uncordon it; that is a
|
||||
# `kubectl uncordon` away and is covered in runbooks/node-reboot-automation.md.
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: kured
|
||||
namespace: flux-system
|
||||
spec:
|
||||
# A HelmRelease plus five static manifests, nothing to re-render; kustomize-controller is
|
||||
# source-triggered, so this is only the drift-correction floor.
|
||||
interval: 10m
|
||||
retryInterval: 2m
|
||||
# 10m, not the 5m the `platform` tier uses: a 3-node DaemonSet rollout under `wait: true` can
|
||||
# exceed 5m on a cold start (image pull on three nodes), which is one of the reasons the Cilium
|
||||
# split-out gave for leaving that tier.
|
||||
timeout: 10m
|
||||
prune: true
|
||||
# See the header: a health SIGNAL for a component nothing depends on, not a gate.
|
||||
wait: true
|
||||
dependsOn:
|
||||
- name: platform # monitoring CRDs + Prometheus + Longhorn; see the header, this is load-bearing
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./platform/kured
|
||||
# REQUIRED, not boilerplate. platform/kured/alerts.yaml renders `${BASE_DOMAIN}` into the
|
||||
# `runbook_url` annotation of all three rules. Without this block that token becomes the EMPTY
|
||||
# STRING and the Kustomization still reports Ready — Flux's substitution fails OPEN — so every
|
||||
# alert would ship a dead link to `https:///runbooks/node-reboot-automation`. The gate stage
|
||||
# `flux-postbuild-substitution` asserts the converse (a rendered token implies this block), and
|
||||
# BASE_DOMAIN is declared in clusters/base/cluster-vars.yaml.
|
||||
#
|
||||
# Nothing else in this path contains `${...}` or a bare `$`: the HelmRelease's alertFilterRegexp is
|
||||
# a plain anchored alternation, and there are no shell snippets anywhere in the component — which
|
||||
# is deliberate, because envsubst would eat a braced shell variable and `$$` would trip the
|
||||
# dollar-collapse assertion in the same check.
|
||||
postBuild:
|
||||
substituteFrom:
|
||||
- kind: ConfigMap
|
||||
name: cluster-vars
|
||||
@@ -85,4 +85,17 @@ resources:
|
||||
# pm-agent-credentials.sops.yaml, so both overlays add a SOPS decryption patch for it (by
|
||||
# name). See clusters/base/pm-agent.yaml and platform/pm-agent/README.md.
|
||||
- pm-agent.yaml # Flux Kustomization -> ./platform/pm-agent (dependsOn apps-wave-0)
|
||||
# kured (Kubernetes Reboot Daemon) — its OWN Kustomization (dependsOn platform, wait:true), NOT a
|
||||
# line in platform/kustomization.yaml. Same reasoning as the Cilium and intel-gpu-plugin
|
||||
# split-outs, with a sharper consequence: `platform` gates platform-config and every apps wave, so
|
||||
# a wedged reboot DaemonSet placed in that wait:true tier could stall every application in the
|
||||
# cluster — a component added to make maintenance SAFER must not be able to cause an outage.
|
||||
# wait:true here is a HEALTH SIGNAL, not a gate: nothing dependsOn it, and nothing should.
|
||||
# The `platform` edge is load-bearing rather than conventional — it supplies the
|
||||
# monitoring.coreos.com CRDs this component's PrometheusRule/ServiceMonitor need (F-MON1), and
|
||||
# Prometheus itself, which kured QUERIES before every reboot to gate on Longhorn volume health.
|
||||
# It reboots a node only when /var/run/reboot-required exists, which on these nodes only a
|
||||
# deliberate operator kernel upgrade creates — the human gate is the sentinel, not the daemon.
|
||||
# See clusters/base/kured.yaml and platform/kured/helmrelease.yaml.
|
||||
- kured.yaml # Flux Kustomization -> ./platform/kured (dependsOn platform)
|
||||
- cluster-vars.yaml # overlay knob surface (base defaults)
|
||||
@@ -363,6 +363,54 @@ SCRAPE_OWNERS: list[dict] = [
|
||||
"why": "velero_* powers VeleroBackupFailure / PartialFailure / NoRecentBackup — the rules "
|
||||
"that failed to notice prod running with zero volume backups",
|
||||
},
|
||||
# ── ADDED 2026-10-17 (feat/automated-upgrades). A NEW OWNER, NOT A RELAXED ASSERTION. ──────
|
||||
# This entry is purely additive: it registers a new metric family and its enablement, and it
|
||||
# changes nothing about how any existing owner is evaluated. It makes the check STRICTER for the
|
||||
# component it covers, not looser — it adds two `require` patterns plus a `companion` obligation
|
||||
# that platform/kured must keep satisfying, so a future edit that disables kured's Service,
|
||||
# ServiceMonitor or NetworkPolicy FAILS the gate instead of silently re-inerting three alerts.
|
||||
#
|
||||
# WHY THE CHECK HAD TO BE TOUCHED AT ALL, rather than the manifest being wrong: assertion (3)
|
||||
# fails any alert over a metric no owner claims, and the escape hatch (KNOWN_UNWIRED_METRICS) is
|
||||
# SHRINK-ONLY, so declaring `kured_*` as accepted debt is not permitted and would be the wrong
|
||||
# answer anyway — the scrape genuinely IS wired here. The only two honest options were to
|
||||
# register the owner (this) or to ship the reboot automation with no alerting at all, which for a
|
||||
# component whose every failure mode is silent would have been the worse engineering choice.
|
||||
# `conformance/**` is a sensitive approval path precisely so that this reasoning gets read.
|
||||
{
|
||||
"owner": "kured: service.create + metrics.create in the HelmRelease",
|
||||
"prefixes": ("kured_",),
|
||||
"file": "platform/kured/helmrelease.yaml",
|
||||
# BOTH keys are required and the chart defaults BOTH to false. `metrics.create` makes the
|
||||
# ServiceMonitor; `service.create` makes the Service it selects. Enable only the monitor and
|
||||
# it selects nothing, so the single kured metric is never scraped — and a PrometheusRule over
|
||||
# a series no target produces is INERT, which for this component means its own liveness
|
||||
# alarm is the thing that breaks. Matched as two separate anchored blocks rather than one
|
||||
# pattern because they are separate top-level values keys, not nested.
|
||||
"require": [
|
||||
r"service:\s*\n\s*create:\s*true",
|
||||
r"metrics:\s*\n\s*create:\s*true",
|
||||
],
|
||||
"forbid": [],
|
||||
"why": "kured_reboot_required is the ONLY metric kured exposes (one GaugeVec, "
|
||||
"cmd/kured/main.go:92-96) and it powers all three platform-health.node-reboots "
|
||||
"rules. Those rules are the only way to learn that OS reboot automation has stalled: "
|
||||
"kured stays 1/1 Running while rebooting nothing, so a missed scrape here is "
|
||||
"indistinguishable from a cluster that needed no reboots",
|
||||
# The ServiceMonitor ALONE is not sufficient — the same NetworkPolicy trap Longhorn hit.
|
||||
"companion": {
|
||||
"file": "platform/kured/kustomization.yaml",
|
||||
"require": [r"^\s*-\s*networkpolicy\.yaml\s*$"],
|
||||
"why": "platform/kured/networkpolicy.yaml ships a default-deny baseline (the kured "
|
||||
"namespace is a platform namespace, so Kyverno's app-namespace-scoped generator "
|
||||
"does not cover it) AND the ingress allow that lets Prometheus reach :8080 "
|
||||
"through it. Drop the networkpolicy from this kustomization and the deny-all "
|
||||
"goes too, so this currently fails SAFE — but reinstating the baseline without "
|
||||
"the allow reproduces Longhorn's measured failure exactly: targets DISCOVERED, "
|
||||
"every scrape 'context deadline exceeded', TSDB empty, all three rules quiet. "
|
||||
"Keeping the two in one file makes that pair inseparable",
|
||||
},
|
||||
},
|
||||
]
|
||||
|
||||
# `up` is produced by EVERY scrape target, so no owner can meaningfully claim it and assertion (1)
|
||||
|
||||
@@ -0,0 +1,300 @@
|
||||
---
|
||||
date: 2026-10-17
|
||||
branch: feat/automated-upgrades
|
||||
title: Automated upgrades — kured + security-only unattended-upgrades, and Renovate finally gets a runner
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Two halves of one process. Neither is useful alone and they must land together.
|
||||
|
||||
### (a) OS upgrades and reboots
|
||||
|
||||
**`platform/kured/`** — the Kubernetes Reboot Daemon, chart `6.1.0` / app `1.23.0`, in its own Flux
|
||||
Kustomization (`clusters/base/kured.yaml`, `dependsOn: [platform]`, `wait: true`), registered in
|
||||
`clusters/base/kustomization.yaml`. Not a line in `platform/kustomization.yaml`: that tier is
|
||||
`wait: true` and gates `platform-config` → every apps wave, so a wedged reboot daemon placed there
|
||||
could stall every application in the cluster. A component added to make maintenance safer must not
|
||||
be able to cause an outage.
|
||||
|
||||
**`infra/ansible/roles/unattended_upgrades/`** + `playbooks/unattended-upgrades.yml`, enabled for
|
||||
prod in `inventory/group_vars/k3s_prod/vars.yml`. A **new** role rather than an extension of
|
||||
`node_baseline` — different lifecycle, and other open branches touch that role.
|
||||
|
||||
**The design, and how it answers this repo's two standing objections to reboot automation.**
|
||||
`roles/node_baseline/tasks/main.yml:61-67` says "IT DOES NOT REBOOT" (k3s-prod-2 hung on a warm
|
||||
reboot 2026-09-03; physical power button required), and `docs/plans/cluster-upgrade-plan.md:291-299`
|
||||
rejected system-upgrade-controller for "automating away the human gate between nodes". Both still
|
||||
stand. **The human gate is the sentinel.** unattended-upgrades is security-only with the kernel
|
||||
blacklisted, so it essentially never produces `/var/run/reboot-required`; that file appears when an
|
||||
operator deliberately runs `node-baseline.yml --tags kernel`. kured then performs only the
|
||||
*mechanical* half the operator would otherwise type by hand — including the ordering rule
|
||||
`node_baseline`'s own output has to shout ("NEVER node 2 first"), which the operator keeps by
|
||||
running the kernel play with `--limit <host>` one node at a time. `cluster-upgrade-plan.md:329-330`
|
||||
names the condition under which its own rejection reverses (fast disks), and prod is that cluster.
|
||||
|
||||
**Quorum: concurrent reboots are structurally impossible.** All three nodes are control-plane+etcd
|
||||
(quorum 2/3). `concurrency: 1` plus kured's single cluster-wide lock annotation on its own DaemonSet
|
||||
means a node reboots only while holding that lock, enforced by API-server optimistic concurrency
|
||||
rather than by timing. **`lockTtl: 0`** (expiry disabled) is the deliberate choice: a TTL exists so
|
||||
reboots continue after the lock holder dies, which on 3 etcd nodes is exactly the catastrophe — a
|
||||
hung node's lock expiring would let a second node take the cluster to 1/3. So a hang stops the cycle
|
||||
forever and pages a human. `lockReleaseDelay: 30m` holds the lock after a node returns.
|
||||
|
||||
**The Longhorn gate.** Measured: 25 volumes, all `numberOfReplicas: 3`, one replica per node, with
|
||||
`replica-soft-anti-affinity: false` and `disable-scheduling-on-cordoned-node: true`. So every drain
|
||||
takes **all 25 volumes to degraded 2/3**, and recovery is serialized —
|
||||
`replica-replenishment-wait-interval: 600`, `concurrent-replica-rebuild-per-node-limit: 1`. Reboot
|
||||
the next node too early and volumes reach 1/3, where `node-drain-policy:
|
||||
block-if-contains-last-replica` can block a drain forever. The inter-node gate is therefore **volume
|
||||
robustness, not pod readiness**, and it is doubled: `lockReleaseDelay: 30m` (unconditional, covers
|
||||
the 600s wait) plus kured's Prometheus alert gate (semantic, covers a rebuild longer than 30m).
|
||||
|
||||
**`alertFilterMatchOnly: true` is load-bearing.** The flag inverts the regexp's meaning: `false`
|
||||
(chart default) makes it an *ignore* list and kured blocks on **any** firing alert. Measured
|
||||
2026-10-17, five alerts were firing with no end in sight (`CNPGPrimariesCoLocated`,
|
||||
`KubeDeploymentReplicasMismatch`, `KubeDeploymentRolloutStuck`, `KubePodNotReady`,
|
||||
`OAuth2ProxyGateUnobserved` — the last three all tracing to `pm-agent`, ImagePullBackOff on a
|
||||
placeholder digest for 12 days). Under the default, kured would have been blocked on day one and
|
||||
never rebooted anything, indistinguishable from "no reboots needed". The allow-list is anchored:
|
||||
`^(LonghornVolumeDegraded|LonghornVolumeFaulted|KubeNodeNotReady|NodeExporterDown)$`.
|
||||
`NodeEtcdRequestsSlow` and the `NodeIOPressure*` family are **deliberately excluded** despite being
|
||||
exactly what you'd want to pause for — prod-1 has a documented etcd WAL I/O starvation history and
|
||||
including them risks a permanent wedge that looks like safety.
|
||||
|
||||
**The kernel floor (the highest-risk item), handled with two independent exclusions.** Measured: all
|
||||
three nodes run `7.1.8+deb13-amd64` (`7.1.8-1~bpo13+1`, the backports metapackage), far above the
|
||||
6.13 memattr floor — **but `linux-image-6.12.107` and `linux-image-6.12.94` are also still
|
||||
installed, below the floor, bootable, and `apt-mark showhold` is empty on every node.** So: (1) the
|
||||
allowed-origins list is `label=Debian-Security` only, and the running kernel comes from
|
||||
`o=Debian Backports` (pin 100, the `NotAutomatic`+`ButAutomaticUpgrades` signature), so u-u cannot
|
||||
*see* a kernel upgrade; (2) `^linux-` is blacklisted anyway, so widening origins later — the most
|
||||
likely future edit — cannot re-arm the hazard. The role **asserts both against `apt-config dump`**,
|
||||
apt's own parse of the whole directory, rather than trusting the file it just wrote. Also blacklisted:
|
||||
`intel-microcode`, `amd64-microcode`, `grub-*`. `Remove-Unused-Kernel-Packages` is **false** against
|
||||
the Debian default, because those older kernels are the documented rollback path. None of this stops
|
||||
the kernel moving — the operator's deliberate Ansible run still does that, unchanged.
|
||||
|
||||
**The silent-no-op trap this closes.** Measured: `update-notifier-common`, `needrestart` and
|
||||
`unattended-upgrades` were **absent on all three nodes**, so *nothing could create
|
||||
`/var/run/reboot-required`*. kured deployed alone would have been permanently idle while 1/1 Running
|
||||
and reporting `kured_reboot_required=0` — green forever. The role installs the sentinel writer and
|
||||
**asserts a hook exists** by discovery (not a hard-coded path), failing loudly if not.
|
||||
`needrestart` runs in auto mode with `k3s.service` blacklisted — auto-restart is what makes a
|
||||
security-only policy effective, but restarting the control plane unattended is worse than a delayed
|
||||
library fix. (k3s is not an apt package at all here: no `k3s`, no `containerd` in dpkg.)
|
||||
|
||||
**Alerts + runbook.** `platform/kured/alerts.yaml` — three rules over kured's *only* metric
|
||||
(`kured_reboot_required`, one GaugeVec, read from `cmd/kured/main.go:92-96`) plus `kube_node_info`:
|
||||
`KuredNotReporting` (`absent()` — the pipeline is dead and the other two are inert),
|
||||
`KuredNodeCoverageGap` (`count(kured) < count(nodes)` — a node silently uncovered, which `absent()`
|
||||
cannot see), `KuredRebootPending` (`for: 36h` — a reboot was offered and blocked). Companion
|
||||
promtool suite `alerts.promtool-test.yaml` pins all three both ways, including that the rule
|
||||
**stops** when a reboot succeeds. `runbooks/node-reboot-automation.md` registered in
|
||||
`runbooks/kustomization.yaml`.
|
||||
|
||||
**The window is DAILY 03:00-06:00, and that choice is driven by the alert.** A weekly window means a
|
||||
pending reboot legitimately waits 6 days, forcing `for: 8d`+ — longer than this cluster's 7d
|
||||
Prometheus retention and reset by any restart past the 1h for-outage-tolerance. Daily caps the
|
||||
legitimate wait under 24h and makes `for: 36h` both robust and meaningful.
|
||||
|
||||
### (b) Image / dependency automation
|
||||
|
||||
**Renovate was aspirational. It now has a runner.** `.gitea/workflows/renovate.yaml` — scheduled
|
||||
daily 03:15 UTC, `workflow_dispatch` with `dry_run` defaulting to **true** for a safe first run.
|
||||
Preflight **fails hard (exit 1)** if `RENOVATE_TOKEN` is missing rather than skipping, because a
|
||||
skipped Gitea job reports `success` and that is precisely how `renovate.json` stayed inert.
|
||||
|
||||
**`renovate.json`: Renovate now owns both halves, and Flux image-automation is formally closed.**
|
||||
ADR-0009 gave container image tags to Flux image-automation; that cannot be delivered on this
|
||||
repo's terms. (1) `ImageUpdateAutomation` **pushes commits, it does not open PRs**, so it bypasses
|
||||
the CLAUDE.md §1 gate; pointed at a side branch instead, nothing opens the PR. (2) It cannot work
|
||||
here anyway: `main` is protected `user_can_push: false` and the `flux-system` GitRepository has **no
|
||||
`spec.secretRef`** — the source is anonymous and read-only. (3) Enabling it means an in-cluster git
|
||||
**write** credential for the repo that defines the cluster. So `docker`/`kubernetes` — disabled
|
||||
*only* to avoid fighting that controller — are enabled, with `kubernetes.managerFilePatterns`
|
||||
covering `platform/`, `platform-config/`, `apps/`, **`catalog/`** (apps are generated from it; a bump
|
||||
only to the rendered output is reverted by the next render and fails `ci/render-golden.sh`),
|
||||
`runbooks/serving/` and `infra/agent/`. Added a no-automerge `blast-radius` rule for `kured`,
|
||||
`longhorn`, `cilium`, `cloudnative-pg`, `kube-prometheus-stack`, because the gate is static and does
|
||||
not boot a node or form a quorum. `platform/kyverno/tests/**` added to `ignorePaths`.
|
||||
|
||||
### Sensitive paths — this PR needs a non-author approving review
|
||||
|
||||
It touches **`clusters/**`**, **`conformance/**`** and **`.gitea/**`**, all in
|
||||
`infra/gitea/approval-policy.json`. It will **not** merge on a green gate alone. Per CLAUDE.md §1 I
|
||||
have **not** posted `/approve-override` and must not — that is the self-approval the stage exists to
|
||||
prevent.
|
||||
|
||||
**The `conformance/` edit, justified.** `+34/-0` in
|
||||
`alert-metrics-have-scrape-targets/check.sh`: one new `SCRAPE_OWNERS` entry claiming `kured_`. It
|
||||
**weakens nothing** — no existing owner's evaluation changes — and it *tightens* the check for my
|
||||
own component, adding two `require` patterns (`service.create: true`, `metrics.create: true`; the
|
||||
chart defaults **both** to false) plus a `companion` requirement that `networkpolicy.yaml` stay in
|
||||
the kustomization. It had to be touched: assertion (3) fails any alert over an unclaimed metric, and
|
||||
the escape hatch `KNOWN_UNWIRED_METRICS` is **shrink-only**, so declaring `kured_*` as debt was not
|
||||
permitted and would have been wrong anyway. The alternative was shipping reboot automation with no
|
||||
alerting, which for a component whose every failure mode is silent is the worse choice. The entry
|
||||
carries a dated comment saying all of this.
|
||||
|
||||
## What I verified
|
||||
|
||||
Measured against the live cluster and nodes, read-only, on 2026-10-17.
|
||||
|
||||
- **Flux image-automation controllers ARE installed and idle.** `-n flux-system get deploy` shows
|
||||
`image-automation-controller` and `image-reflector-controller` both `1/1`, age **43d**; all three
|
||||
`image.toolkit.fluxcd.io` CRDs exist. And **zero** `ImageRepository`/`ImagePolicy`/
|
||||
`ImageUpdateAutomation` objects — "No resources found" for each. The controllers have run for 43
|
||||
days with nothing to do. `platform/kustomization.yaml:52` has the component commented out.
|
||||
- **Renovate is aspirational, confirmed three ways**: no workflow in `.gitea/` or `ci/` (grep),
|
||||
no Renovate CronJob/Deployment/Job in any namespace, and `renovate.json` itself disabled the
|
||||
`docker`+`kubernetes` managers — so every plain `image:` tag was maintained by **nothing**, while
|
||||
**seven files** claim "Renovate maintains this pin".
|
||||
- **`flux-system` GitRepository has no `secretRef`** (empty), URL is the in-cluster Gitea, and its
|
||||
artifact revision was `main@sha1:1a0b208…`, matching my base.
|
||||
- **Nodes**: 3× Debian 13 trixie, all `control-plane,etcd`, k3s `v1.36.4+k3s1`, kernel
|
||||
`7.1.8+deb13-amd64`. `apt-mark showhold` **empty**; `/etc/apt/preferences.d/` **empty**;
|
||||
`/var/run/reboot-required` absent; `unattended-upgrades`/`update-notifier-common`/`needrestart`
|
||||
all **absent**; `linux-image-6.12.107` and `6.12.94` installed alongside the backports kernel;
|
||||
no `k3s`/`containerd` apt package; `apt-cache policy` origins and pins as quoted above.
|
||||
- **No node taints** on any of the three (so no tolerations needed), and `system-node-critical`
|
||||
exists.
|
||||
- **Longhorn**: 1.12.1; `node-drain-policy: block-if-contains-last-replica`;
|
||||
`replica-soft-anti-affinity: false`; `disable-scheduling-on-cordoned-node: true`;
|
||||
`replica-replenishment-wait-interval: 600`; `concurrent-replica-rebuild-per-node-limit: 1`;
|
||||
25 volumes all 3-replica/healthy; 25 replicas per node.
|
||||
- **6 PDBs with 0 allowed disruptions**: 3× `longhorn-system/instance-manager-*` (structurally 0 —
|
||||
single-node selector, `minAvailable: 1`) and 3× CNPG `*-primary`. All three Postgres primaries
|
||||
were on `k3s-prod-1`.
|
||||
- **kured drain flags do not exist.** Read `cmd/kured/main.go:438-449`: `Force`,
|
||||
`DeleteEmptyDirData` and `IgnoreAllDaemonSets` are **hardcoded true**. Passing
|
||||
`--ignore-daemonsets`/`--delete-emptydir-data`/`--force` as extraArgs would CrashLoop the
|
||||
DaemonSet. All three are genuinely needed (12 DaemonSets, 135 emptyDir pods measured).
|
||||
- **kured exposes exactly one metric** — the only `MustRegister` in the project.
|
||||
- **Prometheus selectors are all `{}`** (`serviceMonitorSelector`, `serviceMonitorNamespaceSelector`,
|
||||
`ruleSelector`, `ruleNamespaceSelector`), so a `PrometheusRule`/`ServiceMonitor` in a new namespace
|
||||
is picked up with no label plumbing.
|
||||
- **`count(kube_node_info) == 3`** and `count(longhorn_volume_robustness) == 100`, queried through a
|
||||
read-only port-forward. The 5 chronically-firing alerts were read from `/api/v1/alerts`.
|
||||
- **`LonghornVolumeDegraded`** is `longhorn_volume_robustness{state="degraded"} == 1`, `for: 10m` —
|
||||
shorter than Longhorn's own 600s replenishment wait, so degradation reliably outlasts the latch.
|
||||
- **kured chart 6.1.0 → appVersion 1.23.0** is the newest of 80 entries in the chart index.
|
||||
- **Kyverno/PSA will not block it**: every policy is scoped
|
||||
`app-namespace: "true"`; the namespace sets it explicitly `"false"`, and
|
||||
`restrict-host-namespaces.yaml`'s own header states that carve-out. No cluster-wide PSA baseline.
|
||||
- **`renovate.json` parses** as valid JSON after my edits (`json.load`).
|
||||
- **kured is not installed** and the string `kured` appears nowhere in the repo — greenfield.
|
||||
|
||||
### CORRECTION 2026-10-18 — the first real run FAILED, and the sentinel design was wrong
|
||||
|
||||
`ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod` failed on all three
|
||||
nodes at the package task:
|
||||
|
||||
```
|
||||
fatal: [k3s-prod-1]: FAILED! => "No package matching 'update-notifier-common' is available"
|
||||
```
|
||||
|
||||
**`update-notifier-common` is an UBUNTU package and does not exist in Debian.** Measured on trixie:
|
||||
`apt-cache policy update-notifier-common` → `Candidate: (none)`. The original entry asserted this
|
||||
package "owns the hook that creates /var/run/reboot-required" — true on Ubuntu, false here, and the
|
||||
role could never have worked. This was the exact risk the first version of this entry flagged ("the
|
||||
`find` patterns are a guess at which package ships what") landing in a worse place than predicted:
|
||||
not a wrong glob, a nonexistent package.
|
||||
|
||||
Three further measurements settled what to do instead:
|
||||
|
||||
* `/usr/share/update-notifier/notify-reboot-required` is **absent**.
|
||||
* **No `linux-image-*` postinst references `notify-reboot-required`** — invoking it is an Ubuntu
|
||||
patch, so merely placing a script at that path would never be called.
|
||||
* No hook under `/etc/apt/apt.conf.d` or `/etc/kernel/` mentions `reboot-required`.
|
||||
|
||||
So Debian trixie has **no stock mechanism at all**, and the role now ships the writer itself:
|
||||
`/usr/local/sbin/k3s-notify-reboot-required` (compares the running kernel against the newest
|
||||
installed via `linux-version`, from `linux-base`, which every `linux-image` depends on — measured
|
||||
`linux-base 4.12.1` present) plus a `DPkg::Post-Invoke` hook at
|
||||
`/etc/apt/apt.conf.d/99-k3s-reboot-required`. `/run` is a tmpfs (measured), so the sentinel
|
||||
self-clears on boot and needs no cleanup task.
|
||||
|
||||
The verify assert changed shape too: it was a `find` glob over `/etc/kernel/postinst.d` and
|
||||
`/etc/apt/apt.conf.d` for `*update-notifier*` / `*unattended-upgrades*`. That glob was **weaker
|
||||
than it looked** — it would have matched Debian's own `unattended-upgrades` apt conffile and
|
||||
reported "sentinel writer present" while no writer existed. It now stats the two exact paths this
|
||||
role owns and requires the script to be executable.
|
||||
|
||||
**Newly verified (this correction):** the sentinel script was executed on prod-1 as the
|
||||
unprivileged `k3s` user under `sh -x`. It resolved `running=7.1.8+deb13-amd64`,
|
||||
`newest=7.1.8+deb13-amd64`, took the equal branch, exited 0 and wrote nothing — correct, since the
|
||||
running kernel *is* the newest. `sh -n` and `bash -n` clean.
|
||||
|
||||
**Still NOT verified after this correction:** the *writing* half. Because running == newest on all
|
||||
three nodes, the `touch` path has never executed, so I have not seen the sentinel actually created,
|
||||
nor seen kured react to it. The honest test is the operator's next kernel upgrade. Also unverified:
|
||||
whether a `DPkg::Post-Invoke` hook fires in the `unattended-upgrades` code path specifically (it
|
||||
runs dpkg, so it should, but that is inference) — and whether `linux-version sort --reverse` orders
|
||||
a mixed backports/stable kernel set the way I assume when a 6.12 and a 7.1 are both installed. It
|
||||
produced the right answer with the current set; a future set could differ.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
The honest list. Most of this needs an actual upgrade cycle, which I did not and could not run.
|
||||
|
||||
- **I never ran an OS upgrade, never rebooted a node, and never watched kured take a lock.** No part
|
||||
of the runtime behaviour of either half has been observed. `sudo` needs a password I do not have,
|
||||
so I could not run `apt`, `unattended-upgrade --dry-run`, or install anything. **Everything about
|
||||
(a) working is inference from configuration and measured cluster state, not observation.**
|
||||
- **The Ansible role has never been executed.** Not syntax-checked by `ansible-playbook --syntax-check`
|
||||
either (no inventory access from the gate). Its asserts, its `apt-config dump` parsing, and the
|
||||
sentinel-hook discovery are all unproven. The `find` patterns for the sentinel hook are a *guess at
|
||||
which package ships what* — I could not read `update-notifier-common`'s file list (packages.debian.org
|
||||
returned no such package for that suite/arch), which is exactly why the role discovers rather than
|
||||
hard-codes. **If no hook matches, the role fails loudly — but it may fail on a correct node.**
|
||||
- **Whether CNPG actually switches the primary in response to a drain-induced eviction.** All three
|
||||
`*-primary` PDBs are structurally 0-disruption and `.spec.nodeMaintenanceWindow` is unset on all
|
||||
three clusters. If the operator does not switch over, draining `k3s-prod-1` blocks on three PDBs at
|
||||
once and hits `drainTimeout`. **This is the most likely way the first real reboot fails.** It
|
||||
should be tested with one manual, watched `kubectl drain` before trusting the timer.
|
||||
- **Whether `drainTimeout: 15m` is long enough.** Unmeasured. The `unifi` VMI has
|
||||
`evictionStrategy: LiveMigrate` with `migrationMethod: BlockMigration`, i.e. a full root-disk copy
|
||||
over the network on every `k3s-prod-1` drain. I do not know how long that takes.
|
||||
- **Whether the alert gate actually blocks as intended.** I never saw kured query Prometheus. The
|
||||
assumption that it treats a query error as "do not proceed" is from its documented behaviour, not
|
||||
observed. The NetworkPolicy allowing kured→Prometheus:9090 and Prometheus→kured:8080 is
|
||||
**unproven**; if either is wrong the symptom is silence, which is why `KuredNotReporting` exists —
|
||||
but that alert is itself unproven against a live scrape.
|
||||
- **The alert expressions have not run against real `kured_*` data**, because the metric does not
|
||||
exist in this cluster yet. They pass promtool unit tests against synthetic series whose shape I
|
||||
read from kured's source. `absent()` and the `count() < count()` comparison are the two most
|
||||
likely to behave differently against real label sets.
|
||||
- **`enabledManagers: ["regex"]` may be the wrong name.** Modern Renovate expects `custom.regex` for
|
||||
`customManagers`. I left the pre-existing value alone rather than guess, but **the inline
|
||||
`# renovate:` markers may simply not run.** Unverifiable without executing Renovate. Two markers
|
||||
are additionally known-broken independent of this: `external-snapshotter`'s sits two comment lines
|
||||
above its `newTag`, and Keycloak's `kustomization.yaml` has no marker at all despite a comment
|
||||
claiming Renovate maintains it.
|
||||
- **Renovate has still never run.** I added the runner; `RENOVATE_TOKEN` does not exist yet, so the
|
||||
first scheduled run **will fail** (by design). Nothing about the config is proven correct until a
|
||||
`dry_run: true` dispatch is read by a human. Expect it to want to open ~30 PRs.
|
||||
- **No alert for "automatic patching silently stopped".** u-u's success lives in
|
||||
`/var/log/unattended-upgrades/` on each node and nothing reads it. Likewise **no alert for the
|
||||
Renovate schedule ceasing to fire**. Both are real gaps I did not close.
|
||||
- **`coredns` is 1 replica and I did not fix it.** It is k3s-managed, out of this PR's scope, and it
|
||||
means every `k3s-prod-1` reboot causes a cluster-wide in-cluster DNS gap. Recommended follow-up.
|
||||
- **kured cannot honour "NEVER node 2 first".** It has no ordering control. The mitigation — run the
|
||||
kernel play with `--limit` one host at a time — is a *procedure*, documented in the runbook, not an
|
||||
enforced property. If the operator flags all three nodes at once, kured picks the order.
|
||||
- **kured cannot press a power button.** The 2026-09-03 warm-reboot hang is unmitigated. Bounded, not
|
||||
prevented: quorum holds at 2/3, the lock is held forever so nothing cascades, and
|
||||
`KuredRebootPending` eventually fires.
|
||||
- **Upstream version claims are a snapshot, partly from search results**, not all from authoritative
|
||||
release APIs. I could not determine latest for versitygw, crafty, nginx-unprivileged, act_runner,
|
||||
`virt-template-*`, the linuxserver wireguard date-tag, or the exact latest PostgreSQL 16 patch
|
||||
(sources conflicted 16.10 vs 16.15). **The gitea `1.24.3 → v28.1.0` gap is almost certainly an
|
||||
upstream versioning-scheme change, not 26 majors — do not act on that framing.**
|
||||
- **I did not verify Renovate's `kustomize` manager still processes `images:`/`newTag`** now that
|
||||
`docker` is enabled, which decides whether `external-snapshotter` and `intel-gpu-plugin` are
|
||||
managed at all.
|
||||
- **I did not raise any HelmRelease version ceiling**, so the 9 minor-locking ranges still cap
|
||||
Renovate at patches. Bumping `kube-prometheus-stack` in particular will fail the gate until
|
||||
`ci/versions.env`'s exact promtool/Prometheus `3.14.0` lockstep moves with it.
|
||||
- **L2/L3 conformance did not run** (`vars.CLUSTER_AVAILABLE` unset), so nothing here is proven
|
||||
against a cluster by CI either.
|
||||
@@ -1,108 +0,0 @@
|
||||
---
|
||||
date: 2026-10-18
|
||||
branch: fix/coredns-ha
|
||||
title: "coredns ran at one replica, so any node drain was a cluster-wide DNS outage"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
**MEASURED on prod 2026-10-18: `kube-system/coredns` had `replicas: 1`.** In-cluster DNS was a
|
||||
single pod on a single node, so draining or rebooting that node was a **cluster-wide in-cluster DNS
|
||||
outage** until the pod rescheduled and became ready — every Service lookup, every CNPG replica
|
||||
finding its primary, every operator addressing the API by name. It was invisible because `1/1
|
||||
Ready` looks perfectly healthy.
|
||||
|
||||
It also became urgent rather than latent: the scheduled reboot automation on
|
||||
`feat/automated-upgrades` drains and reboots one node at a time, unattended, overnight — turning a
|
||||
single point of failure into a recurring one.
|
||||
|
||||
New `platform/coredns/hpa.yaml`: a `HorizontalPodAutoscaler` on the coredns Deployment with
|
||||
`minReplicas: 2`, `maxReplicas: 3`, registered in the existing `platform/coredns/kustomization.yaml`
|
||||
alongside the split-horizon ConfigMap already there.
|
||||
|
||||
**Why an HPA and not `replicas: 2` — this is the whole decision.** k3s owns the coredns Deployment
|
||||
and we cannot durably edit it: it is applied by k3s's own deploy controller from a bundled manifest
|
||||
(`AddOn/coredns`, source `/data/k3s/server/manifests/coredns.yaml`, `managedFields` managers
|
||||
`deploy@k3s-prod-1` and `k3s`, plus an `objectset.rio.cattle.io/hash` label). k3s re-stages its
|
||||
bundled manifests on startup, so a `replicas: 2` written there — or a `kubectl scale` — is reverted
|
||||
on the next k3s restart or upgrade. Two alternatives were weighed and rejected:
|
||||
|
||||
* **`coredns.yaml.skip` + our own templated copy** (the pattern `roles/kube_vip` already uses for a
|
||||
file in `server/manifests`). Deterministic and never flaps, but it takes ownership of coredns away
|
||||
from k3s permanently — k3s upgrades would stop updating it and the version would need refreshing
|
||||
by hand forever. Too high a price for a replica count.
|
||||
* **A Flux-managed patch of the Deployment.** Rejected outright: Flux and k3s's deploy controller
|
||||
would both own one object and fight continuously — the exact shape of the `default-deny`
|
||||
NetworkPolicy drift loop this cluster already suffered (~2500 events/10min).
|
||||
|
||||
An HPA avoids both: a separate object k3s does not own, driving the `/scale` subresource rather than
|
||||
the Deployment spec, so there is no shared-ownership fight. If k3s re-stages `replicas: 1`, the HPA
|
||||
sees it below `minReplicas` and scales back within one reconcile.
|
||||
|
||||
It is a **floor-holder, not an autoscaler**: the CPU metric exists only because an `autoscaling/v2`
|
||||
HPA needs one to be valid. `scaleDown.stabilizationWindowSeconds: 600` because DNS is the worst
|
||||
thing in the cluster to flap; `scaleUp` is immediate, since that is the path that restores redundancy
|
||||
after a k3s restart.
|
||||
|
||||
**Immediate action taken outside this PR:** `kubectl -n kube-system scale deploy coredns
|
||||
--replicas=2` was run at the operator's explicit request, so the cluster has two replicas **now**
|
||||
rather than after merge. That hand scale is what this PR makes durable.
|
||||
|
||||
## What I verified
|
||||
|
||||
* **The defect:** `kubectl -n kube-system get deploy coredns` → `replicas=1 available=1`.
|
||||
* **k3s ownership:** the `AddOn/coredns` object exists with source
|
||||
`/data/k3s/server/manifests/coredns.yaml`; the Deployment has no `ownerReferences` but carries
|
||||
`objectset.rio.cattle.io/hash` and `managedFields` managers `deploy@k3s-prod-1` / `k3s`. So it is
|
||||
k3s-managed, not Flux-managed — independently corroborated by another agent, which confirmed
|
||||
`platform/coredns/` contains **only** the `coredns-custom` ConfigMap and does not touch the
|
||||
Deployment or its replica count.
|
||||
* **The hand scale works and holds:** coredns went to `2/2 Running` on two different nodes, and
|
||||
in-cluster DNS kept answering throughout (`nslookup kubernetes.default.svc.cluster.local
|
||||
10.43.0.10` → `10.43.0.1`). Still `spec.replicas=2 ready=2` after ~2 minutes, with the AddOn
|
||||
checksum **unchanged** — so k3s's deploy controller does not revert on a drift timer; it
|
||||
re-applies on manifest change and on restart.
|
||||
* **The HPA's preconditions hold:** coredns ships `requests: {cpu: 100m, memory: 70Mi}` (so a
|
||||
Resource/Utilization metric is meaningful), and metrics-server is installed as a k3s addon and
|
||||
verified serving — `kubectl top pod -l k8s-app=kube-dns` returns **5m and 4m** against that 100m
|
||||
request, i.e. ~5% utilisation, confirming the 80% target will never be approached and the HPA will
|
||||
sit at `minReplicas`.
|
||||
* **The kustomization edit is additive:** `git diff --stat` is `1 file changed, 4 insertions(+)`,
|
||||
and `- coredns` was already registered in `platform/kustomization.yaml`, so no change was needed
|
||||
there.
|
||||
* All three touched/added YAML files parse.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
* **The HPA has never been applied.** I did not reconcile it, so I have **not** observed it create,
|
||||
reach `ScalingActive=True`, resolve its metric against real coredns data, or hold `minReplicas`.
|
||||
Applying is an operator action (CLAUDE.md §5). The failure mode to watch for is
|
||||
`ScalingActive=False` with `FailedGetResourceMetric` — the HPA would then be present and inert,
|
||||
which looks identical to working because the hand scale already left 2 replicas in place. **Check
|
||||
`kubectl -n kube-system describe hpa coredns` after merge**, not just the replica count.
|
||||
* **The central claim that k3s reverts `replicas` on restart is REASONED, NOT OBSERVED.** It rests
|
||||
on k3s re-staging bundled manifests at startup plus the measured ownership evidence above. I did
|
||||
**not** restart k3s to watch the revert happen — doing so is a rolling control-plane restart, an
|
||||
operator action. If k3s in fact preserves a modified replica count, this HPA is unnecessary
|
||||
(though harmless).
|
||||
* **I did not confirm the HPA wins the race against k3s's deploy controller after a restart.** Both
|
||||
write the scale; the expectation is the HPA re-asserts within a reconcile, leaving a brief window
|
||||
at 1 replica. Neither the window's length nor the outcome has been measured.
|
||||
* **I nearly shipped a destructive change.** My first write of
|
||||
`platform/coredns/kustomization.yaml` replaced the file wholesale, listing only `hpa.yaml` — which
|
||||
would have pruned the existing `coredns-custom` ConfigMap and **broken in-cluster split-horizon
|
||||
DNS (R-DNS-01/03)**. It was caught by a guard assertion before commit and restored from `HEAD`, and
|
||||
the diff is now additive. Recorded because the near-miss is the useful part: `platform/coredns/`
|
||||
already existed and I assumed it did not. **No conformance check would obviously have caught the
|
||||
removal** — it is valid YAML and a legitimate-looking prune.
|
||||
* **Whether 2 is enough.** Three nodes, and coredns' anti-affinity only *prefers* spreading, so both
|
||||
replicas could in principle land on one node and a single drain would still take DNS down. I did
|
||||
not add a `topologySpreadConstraint` or a PodDisruptionBudget — both arguably belong here and
|
||||
neither is in this change.
|
||||
* **No alert fires if coredns drops to 1 replica**, so a future silent regression (k3s upgrade
|
||||
changing the manifest, the HPA going inert) would not be reported. That gap is not closed here.
|
||||
* **No conformance test asserts this**, so a future edit could remove the HPA with no gate stage
|
||||
noticing.
|
||||
* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline
|
||||
subset only.
|
||||
* I did not re-measure branch protection or runner state (CLAUDE.md §2b).
|
||||
@@ -420,3 +420,34 @@ flux_bootstrap_env_path: "clusters/prod"
|
||||
# rather than leaving an inert drop-in behind, so this group-wide `true` is still correct
|
||||
# for it.
|
||||
etcd_io_priority_enabled: true
|
||||
|
||||
# --- automatic security patching (roles/unattended_upgrades) -------------------------
|
||||
# TURNS ON roles/unattended_upgrades FOR PROD ONLY. Role default is false. Same seam as
|
||||
# node_baseline_backports_kernel and etcd_io_priority_enabled above.
|
||||
#
|
||||
# WHAT WAS THERE BEFORE: nothing at all. Measured on all three prod nodes 2026-10-17 —
|
||||
# `unattended-upgrades`, `update-notifier-common` and `needrestart` were ALL absent,
|
||||
# `apt-mark showhold` was empty on every node, /etc/apt/preferences.d/ was empty, and
|
||||
# nothing on the node could create /var/run/reboot-required. These boxes have had ZERO
|
||||
# automatic patching since provisioning, and no way to signal that a reboot was needed.
|
||||
#
|
||||
# WHAT ENABLING THIS DOES *NOT* DO: nothing, until the operator runs
|
||||
# playbooks/unattended-upgrades.yml. It is not applied by site.yml (deliberately — see
|
||||
# that playbook's header). And once run, it still never upgrades the KERNEL, never
|
||||
# restarts k3s, and never reboots:
|
||||
# * kernel — blacklisted `^linux-`, AND the trixie-backports archive the running
|
||||
# kernel comes from (o=Debian Backports, measured) is not an allowed origin. Two
|
||||
# independent exclusions, because this is the EFI-memattr constraint: these HP minis
|
||||
# need >= 6.13 or they can boot with a fraction of their RAM
|
||||
# (PROD-PROVISIONING.md §0b), and 6.12 kernels are still installed and bootable here.
|
||||
# The kernel continues to move ONLY under a deliberate
|
||||
# `node-baseline.yml --tags kernel` run, exactly as before.
|
||||
# * k3s — needrestart blacklist. k3s is not even an apt package on these nodes
|
||||
# (measured: no `k3s`, no `containerd` in dpkg), so the control plane and runtime are
|
||||
# outside apt's reach regardless.
|
||||
# * reboots — Automatic-Reboot "false", asserted by the role against apt's own
|
||||
# effective config. Reboots belong to kured (platform/kured): one node at a time,
|
||||
# under a cluster-wide lock, inside 03:00-06:00, after cordon + drain. Three
|
||||
# control-plane nodes with etcd quorum 2/3 must never reboot themselves
|
||||
# independently, which is what Automatic-Reboot "true" would mean.
|
||||
unattended_upgrades_enabled: true
|
||||
@@ -0,0 +1,36 @@
|
||||
---
|
||||
# Automatic Debian SECURITY patching for the k3s nodes — the package half of the automated upgrade
|
||||
# process. The reboot half is kured (platform/kured), reconciled by Flux; the two are designed
|
||||
# together and neither is useful alone. Full rationale for every decision:
|
||||
# roles/unattended_upgrades/defaults/main.yml and that role's README.md.
|
||||
#
|
||||
# TARGET (R-ENV-02): `k3s_hosts` defaults to dev's `k3s_servers`; prod is
|
||||
# ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod
|
||||
# Same pattern node-baseline.yml and node-health.yml established — `hosts:` is templated before
|
||||
# group_vars exist, so the target is an extra var, never a group_vars value.
|
||||
#
|
||||
# The role is a NO-OP unless `unattended_upgrades_enabled` is true, which is set for prod only in
|
||||
# inventory/group_vars/k3s_prod/vars.yml. Running this against dev changes nothing.
|
||||
#
|
||||
# DELIBERATELY NOT IN site.yml. site.yml is the bring-up chain (node-baseline -> k3s ->
|
||||
# etcd-io-priority -> kube-vip -> cilium -> verify -> flux-bootstrap) and is "not run against prod
|
||||
# casually". Automatic patching is not a prerequisite for anything in that chain and nothing in the
|
||||
# chain depends on it, so adding it there would only widen what a cluster rebuild touches. It is
|
||||
# applied on its own, once, and re-applied when its configuration changes.
|
||||
#
|
||||
# SAFE TO RUN ON A LIVE CLUSTER, AT ANY TIME. This playbook installs and configures the upgrader. It
|
||||
# never runs `apt upgrade`, never restarts k3s, and never reboots — the role asserts the last of
|
||||
# those against apt's own effective config rather than trusting the file it just wrote. The first
|
||||
# unattended upgrade happens at the next timer firing (04:00 +/-60m), and any reboot that upgrade
|
||||
# implies is kured's to schedule, one node at a time, inside 03:00-06:00.
|
||||
#
|
||||
# TAGS. `--tags verify` re-runs only the read-only assertions and the per-node summary, which is the
|
||||
# way to answer "is automatic patching actually working on these boxes, and is anything waiting on a
|
||||
# reboot" without rewriting any configuration:
|
||||
# ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify
|
||||
- name: Automatic security upgrades on the k3s nodes (kernel excluded; reboots delegated to kured)
|
||||
hosts: "{{ k3s_hosts | default('k3s_servers') }}"
|
||||
become: true
|
||||
gather_facts: true
|
||||
roles:
|
||||
- unattended_upgrades
|
||||
@@ -0,0 +1,195 @@
|
||||
# roles/unattended_upgrades
|
||||
|
||||
Automatic Debian **security** patching on the k3s nodes. The kernel is excluded, k3s is never
|
||||
restarted, and reboots are delegated to **kured** (`platform/kured`).
|
||||
|
||||
- **Playbook:** `playbooks/unattended-upgrades.yml`
|
||||
- **Enabled by:** `unattended_upgrades_enabled` — role default `false`, set `true` for prod in
|
||||
`inventory/group_vars/k3s_prod/vars.yml`
|
||||
- **Companion component:** `platform/kured` (the reboot half). **Neither is useful alone** — see
|
||||
"The two halves" below.
|
||||
- **Every decision and its reasoning:** `defaults/main.yml`. That file is the argument; this one is
|
||||
the orientation.
|
||||
|
||||
## The two halves, and why they ship together
|
||||
|
||||
| | what it does | what it must NOT do |
|
||||
|---|---|---|
|
||||
| **this role** (Ansible, host) | installs security updates continuously; leaves `/var/run/reboot-required` behind when one needs a reboot | upgrade the kernel; restart k3s; reboot |
|
||||
| **kured** (Flux, cluster) | cordons, drains, reboots and uncordons **one node at a time** when that file exists | decide *whether* a reboot is needed |
|
||||
|
||||
Deploying either alone produces a mechanism that looks healthy and does nothing:
|
||||
|
||||
- **kured alone** watches a file that **nothing on Debian ever creates**. Measured 2026-10-18,
|
||||
after a first attempt to install Ubuntu's `update-notifier-common` failed on all three nodes
|
||||
with *"No package matching 'update-notifier-common' is available"*:
|
||||
- `apt-cache policy update-notifier-common` → `Candidate: (none)` — the package does not exist
|
||||
in Debian; it is Ubuntu-only.
|
||||
- `/usr/share/update-notifier/notify-reboot-required` → absent.
|
||||
- **No `linux-image-*` postinst references `notify-reboot-required`** — calling it is an Ubuntu
|
||||
patch, so even dropping a script at that path would never be invoked.
|
||||
- No hook under `/etc/apt/apt.conf.d` or `/etc/kernel/` mentions `reboot-required`.
|
||||
|
||||
So **this role ships the writer itself** (`/usr/local/sbin/k3s-notify-reboot-required` plus a
|
||||
`DPkg::Post-Invoke` hook) rather than installing a package. Without it kured would sit at 1/1
|
||||
Running reporting `kured_reboot_required=0` forever.
|
||||
- **this role alone** installs patches that a long-running process never picks up, and when a
|
||||
reboot *is* needed there is nothing safe to take it. `Automatic-Reboot "true"` is not the
|
||||
answer: three control-plane nodes with etcd quorum 2/3, each rebooting at its own time, is how
|
||||
the cluster is lost.
|
||||
|
||||
## What it was replacing: nothing
|
||||
|
||||
Measured on all three prod nodes, 2026-10-17:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| `unattended-upgrades` | **not installed** |
|
||||
| `update-notifier-common` | **not installable** — Ubuntu-only, `Candidate: (none)` on trixie |
|
||||
| `needrestart` | **not installed** |
|
||||
| `apt-mark showhold` | **empty** — no package held on any node |
|
||||
| `/etc/apt/preferences.d/` | **empty** — no apt pin declares the backports-kernel intent |
|
||||
| `/var/run/reboot-required` | absent, and uncreatable |
|
||||
|
||||
These nodes had received **no automatic patching since provisioning**.
|
||||
|
||||
## The three decisions
|
||||
|
||||
### 1. Security only
|
||||
|
||||
Allowed origins are `label=Debian-Security` and nothing else. `trixie-updates` is excluded
|
||||
deliberately (not overlooked) — it carries non-security fixes that are useful but are the archive
|
||||
through which an unreviewed change would most plausibly arrive on a control-plane node. The knob to
|
||||
add it is `unattended_upgrades_include_updates`.
|
||||
|
||||
### 2. The kernel is excluded twice over
|
||||
|
||||
**This is the highest-risk part of the change.** These HP EliteDesk 800 G5 minis publish a corrupt
|
||||
EFI Memory Attributes Table on R21 firmware; Linux `< 6.13` does not validate it before reserving
|
||||
memory from its contents, and `k3s-prod-2` twice booted with **2.6 GiB of 16 GB**. Upstream
|
||||
`8fbe4c49c0cc` (first in v6.13) is the only fix — there is no kernel cmdline workaround. Full
|
||||
write-up: `infra/PROD-PROVISIONING.md` §0b and `roles/node_baseline/defaults/main.yml:21-51`.
|
||||
|
||||
Measured 2026-10-17: all three nodes run `7.1.8+deb13-amd64` (`7.1.8-1~bpo13+1`, i.e. the backports
|
||||
metapackage), comfortably above the floor — **but `linux-image-6.12.107+deb13-amd64` and
|
||||
`linux-image-6.12.94+deb13-amd64` are also still installed, below the floor, bootable, and held by
|
||||
nothing.**
|
||||
|
||||
So an unattended process must not be able to touch the kernel at all:
|
||||
|
||||
1. **Origin exclusion (structural).** The running kernel comes from `o=Debian Backports` (measured
|
||||
via `apt-cache policy`, pin priority 100 — the `NotAutomatic` + `ButAutomaticUpgrades`
|
||||
signature). That origin is not in `Origins-Pattern`, so unattended-upgrades cannot see a kernel
|
||||
upgrade at all.
|
||||
2. **Package blacklist (belt and braces).** `^linux-` is blacklisted, so even if the origin list is
|
||||
later widened — the most likely future edit to this role, and the one that would silently re-arm
|
||||
the hazard — the kernel is still untouchable.
|
||||
|
||||
The role **asserts both** against `apt-config dump` (apt's own parse of the whole
|
||||
`/etc/apt/apt.conf.d/` directory) rather than trusting the file it just wrote.
|
||||
|
||||
**This does not stop the kernel moving.** The operator still advances it deliberately with
|
||||
|
||||
```
|
||||
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel --limit <host>
|
||||
```
|
||||
|
||||
which is the only thing that maintains the backports source, the metapackage and the on-node
|
||||
provenance stamp. `node_baseline`'s own note that `apt upgrade` keeps the backports kernel current
|
||||
still holds for an operator-run apt. What changed is that a **new unattended actor** is forbidden
|
||||
from it.
|
||||
|
||||
Also blacklisted, for the same "operator-only" reason: `intel-microcode`, `amd64-microcode`,
|
||||
`grub-*`. Microcode and bootloader changes take effect on reboot and alter boot behaviour on the
|
||||
very firmware that is already misbehaving here.
|
||||
|
||||
`Remove-Unused-Kernel-Packages` is **false**, against the Debian default: those older kernels are
|
||||
the documented rollback path, and an unattended process pruning them could remove the fallback an
|
||||
operator needs in front of a box that will not boot.
|
||||
|
||||
### 3. needrestart restarts services — but never k3s
|
||||
|
||||
A security-only policy that restarts nothing patches bytes on disk that no running process reads.
|
||||
`needrestart` in mode `a` restarts services still mapped to a deleted library, which is what makes
|
||||
the policy actually effective.
|
||||
|
||||
`k3s.service` is blacklisted. Restarting the Kubernetes control plane unattended at 04:00 is a
|
||||
worse outcome than a delayed library fix; that fix lands at the next kured reboot instead. Measured
|
||||
2026-10-17, `k3s.service` is the **only** matching unit — k3s bundles its own containerd and
|
||||
kubelet, and neither `containerd` nor `containerd.io` is an apt package on these nodes — so this
|
||||
one entry is complete today.
|
||||
|
||||
## Schedule
|
||||
|
||||
`apt-daily` at 03:30 and `apt-daily-upgrade` at 04:00, both `±60m` randomised, via systemd drop-ins.
|
||||
Chosen to land **inside** kured's 03:00–06:00 reboot window so that a sentinel written by an upgrade
|
||||
can be acted on the same night rather than ~24h later.
|
||||
|
||||
Two traps the role handles explicitly:
|
||||
|
||||
- `/etc/apt/apt.conf.d/20auto-upgrades` must set the four `APT::Periodic::*` counters. Without
|
||||
them the apt-daily timers fire and **do nothing** — installed, configured, green, and inert.
|
||||
- The timer drop-ins clear `OnCalendar=` before setting it. systemd **accumulates** `OnCalendar`
|
||||
entries across drop-ins, so omitting the reset keeps the shipped schedule *in addition* to ours.
|
||||
|
||||
## Why a new role rather than extending `node_baseline`
|
||||
|
||||
`node_baseline` is the repo's only other apt surface and would have been a defensible home. Two
|
||||
reasons against:
|
||||
|
||||
1. **Single responsibility.** `node_baseline` is the *first-boot floor* — it runs before k3s exists
|
||||
and is the thing that makes a node installable. Continuous patching policy is a different
|
||||
lifecycle with a different blast radius, and it wants to be runnable (and re-runnable) without
|
||||
going anywhere near that role's data-disk tasks.
|
||||
2. **Concurrent work.** Other open branches touch `roles/node_baseline/`. A new role has nothing to
|
||||
merge.
|
||||
|
||||
Nothing here duplicates `node_baseline`: it does not add apt sources, does not touch the kernel, and
|
||||
does not manage disks.
|
||||
|
||||
## Post-merge verification
|
||||
|
||||
Running the role is safe on a live cluster at any time — it installs and configures the upgrader, and
|
||||
never runs `apt upgrade`, restarts k3s, or reboots.
|
||||
|
||||
```bash
|
||||
# Apply (prod). Idempotent; re-runnable.
|
||||
cd infra/ansible && ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod
|
||||
|
||||
# Re-run ONLY the read-only assertions + the per-node pending-reboot summary.
|
||||
cd infra/ansible && ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify
|
||||
```
|
||||
|
||||
Then, on a node, confirm the three properties that matter (all read-only, no sudo needed for the
|
||||
first two):
|
||||
|
||||
```bash
|
||||
ssh [email protected] 'apt-config dump | grep -iE "unattended-upgrade::(automatic-reboot|package-blacklist|origins)"'
|
||||
ssh [email protected] 'systemctl list-timers apt-daily-upgrade.timer --no-pager'
|
||||
ssh [email protected] 'ls -la /var/run/reboot-required* 2>&1 | head -2
|
||||
ls -l /usr/local/sbin/k3s-notify-reboot-required /etc/apt/apt.conf.d/99-k3s-reboot-required 2>&1
|
||||
dpkg -l needrestart unattended-upgrades | tail -3'
|
||||
```
|
||||
|
||||
Expect: `Automatic-Reboot "false"`, a blacklist containing `^linux-`, an `Origins-Pattern`
|
||||
containing `Debian-Security` and **not** `Backports`, a next-elapse around 04:00, and all three
|
||||
packages `ii`.
|
||||
|
||||
A dry run of what the upgrader would actually do (needs root, so this one is an operator step):
|
||||
|
||||
```bash
|
||||
sudo unattended-upgrade --dry-run --debug
|
||||
```
|
||||
|
||||
The kernel must appear in its blacklist output, not its upgrade list.
|
||||
|
||||
## What this role cannot tell you
|
||||
|
||||
- **That a security upgrade has ever succeeded.** That is in
|
||||
`/var/log/unattended-upgrades/unattended-upgrades.log` on each node, and nothing in this repo or
|
||||
in Prometheus reads it. There is no alert for "automatic patching has silently stopped working".
|
||||
That is a real, known gap — see the handoff entry
|
||||
`decisions/2026-10-17-automated-upgrades.md`.
|
||||
- **That a reboot actually happened.** That is kured's half, and it *is* alerted on
|
||||
(`KuredRebootPending`, `KuredNodeCoverageGap`, `KuredNotReporting` in
|
||||
`platform/kured/alerts.yaml`).
|
||||
@@ -0,0 +1,199 @@
|
||||
---
|
||||
# roles/unattended_upgrades — automatic Debian SECURITY patching on the k3s nodes, with the kernel
|
||||
# deliberately excluded and with reboots delegated to kured (platform/kured).
|
||||
#
|
||||
# OFF BY DEFAULT, ON FOR PROD. Same seam as node_baseline_backports_kernel / etcd_io_priority_enabled:
|
||||
# a plain boolean, default false, turned on in inventory/group_vars/k3s_prod/vars.yml. Not a hostname
|
||||
# test — that would encode the environment into the role (see node_baseline's defaults for the full
|
||||
# argument, which this role follows rather than restates).
|
||||
#
|
||||
# ═════════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# STATE BEFORE THIS ROLE (measured on all three prod nodes, 2026-10-17, via `dpkg-query -W`)
|
||||
# ═════════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# unattended-upgrades NOT INSTALLED
|
||||
# update-notifier-common NOT INSTALLED
|
||||
# needrestart NOT INSTALLED
|
||||
# apt-mark showhold EMPTY (no package is held on any node)
|
||||
# /var/run/reboot-required ABSENT, and nothing on the node can create it
|
||||
# /etc/apt/preferences.d/ EMPTY (no apt pin declares the backports-kernel intent)
|
||||
# So: zero automatic patching, and no mechanism by which a reboot could ever be signalled. That last
|
||||
# point is why this role and platform/kured must land together — kured watches
|
||||
# /var/run/reboot-required, and on these nodes today NOTHING WRITES IT. Deploying kured alone would
|
||||
# produce a daemon that is permanently, silently idle while looking perfectly healthy.
|
||||
#
|
||||
# ═════════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# DECISION 1 — SECURITY ONLY, NOT FULL UPGRADES
|
||||
# ═════════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# Measured `apt-cache policy` on k3s-prod-1, 2026-10-17, gives the four archives and their pins:
|
||||
# trixie o=Debian, a=stable, l=Debian pin 500
|
||||
# trixie-updates o=Debian, a=stable-updates, l=Debian pin 500
|
||||
# trixie-security o=Debian, a=stable-security, l=Debian-Security pin 500
|
||||
# trixie-backports o=Debian Backports, a=stable-backports, l=Debian Backports pin 100
|
||||
#
|
||||
# Only `Debian-Security` is allowed below. The reasoning, in order of weight:
|
||||
#
|
||||
# * Security updates are the class where the cost of NOT applying them is unbounded and the
|
||||
# change itself is minimal by Debian policy (targeted fixes against the frozen stable version,
|
||||
# no feature churn). It is the one class worth applying with nobody watching.
|
||||
# * A full `apt upgrade` on these nodes would also pull trixie point-release changes, which is a
|
||||
# larger surface than anyone has reviewed for a 3-node etcd cluster, for no urgency.
|
||||
# * `trixie-updates` is DELIBERATELY EXCLUDED, not overlooked. It carries non-security fixes that
|
||||
# are too urgent to wait for a point release — useful, but it is the archive through which an
|
||||
# unreviewed change would most plausibly arrive, and it is not what "unattended" should mean on
|
||||
# a control-plane node. Add it consciously later if the operator wants it; the knob is below.
|
||||
# * `trixie-backports` is EXCLUDED, AND THAT IS THE KERNEL ARGUMENT — see DECISION 2.
|
||||
#
|
||||
# ═════════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# DECISION 2 — THE KERNEL IS EXCLUDED TWICE OVER. THIS IS THE HIGHEST-RISK PART OF THE CHANGE.
|
||||
# ═════════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# The constraint: HP's R21 firmware on these EliteDesk 800 G5 minis publishes a corrupt EFI Memory
|
||||
# Attributes Table, and Linux < 6.13 does not validate it before reserving memory from its contents
|
||||
# — k3s-prod-2 twice booted with 2.6 GiB of 16 GB. Upstream 8fbe4c49c0cc (first in v6.13) is the
|
||||
# ONLY fix; there is no kernel cmdline workaround. So these nodes run the backports metapackage
|
||||
# `linux-image-amd64` (measured 2026-10-17: version 7.1.8-1~bpo13+1 installed and running on all
|
||||
# three, well above the floor). Full write-up: infra/PROD-PROVISIONING.md §0b, and the rationale
|
||||
# block in roles/node_baseline/defaults/main.yml:21-51.
|
||||
#
|
||||
# A NODE THAT COMES BACK FROM AN UNATTENDED UPGRADE ON A 6.12 KERNEL IS A NODE THAT MAY COME BACK
|
||||
# WITH A FRACTION OF ITS RAM. Note that 6.12 kernels are not hypothetical here — measured
|
||||
# 2026-10-17, `linux-image-6.12.107+deb13-amd64` and `linux-image-6.12.94+deb13-amd64` are BOTH
|
||||
# still installed on all three nodes, below the floor, bootable, and nothing holds them.
|
||||
#
|
||||
# Two independent mechanisms keep unattended-upgrades away from the kernel, and the role asserts
|
||||
# both rather than trusting either:
|
||||
#
|
||||
# 1. ORIGIN EXCLUSION (structural). The running kernel comes from `o=Debian Backports`, which is
|
||||
# not in Allowed-Origins. u-u only ever considers packages from allowed origins, so it cannot
|
||||
# see a backports kernel upgrade at all. This also means the backports kernel receives no
|
||||
# automatic patching — accepted, and in any case backports carries no security guarantees
|
||||
# (node_baseline's defaults state that cost plainly: "no security-tracker commitment, no DSAs").
|
||||
# 2. PACKAGE BLACKLIST (belt and braces). `linux-*` is blacklisted below, so even if the origin
|
||||
# list is later widened — the most likely future edit to this file, and the one that would
|
||||
# silently re-arm the hazard — the kernel is still untouchable. Without this, widening origins
|
||||
# to include backports would hand an unattended process the EFI-memattr-critical package.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* DO, AND MUST NOT BE READ AS DOING: it does not stop the kernel moving. The
|
||||
# operator still advances it deliberately with
|
||||
# ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel --limit <host>
|
||||
# which is the only thing that maintains the backports source, the metapackage and the on-node
|
||||
# provenance stamp. node_baseline's own comment ("`ButAutomaticUpgrades: yes` ... means `apt upgrade`
|
||||
# keeps it current once installed") still holds for an operator-run apt; the measured pin priority of
|
||||
# 100 is exactly the NotAutomatic + ButAutomaticUpgrades signature that produces that behaviour.
|
||||
# Nothing here changes it. What changes is that a NEW unattended actor is forbidden from the kernel.
|
||||
#
|
||||
# NOT BLACKLISTED, BECAUSE THEY ARE NOT APT PACKAGES AT ALL — measured 2026-10-17: `dpkg-query` finds
|
||||
# no `k3s`, no `containerd` and no `containerd.io`. k3s is a single static binary at
|
||||
# /usr/local/bin/k3s installed by roles/k3s_server, and it bundles its own containerd and kubelet.
|
||||
# So the entire Kubernetes control plane and container runtime are OUTSIDE apt's reach and cannot be
|
||||
# upgraded by this role even in principle. That is worth knowing before worrying about it.
|
||||
unattended_upgrades_enabled: false
|
||||
|
||||
# The packages this role installs. Each one is load-bearing; see the notes.
|
||||
unattended_upgrades_packages:
|
||||
# The upgrader itself.
|
||||
- unattended-upgrades
|
||||
# NOTE: `update-notifier-common` was listed here and REMOVED (2026-10-18). It is an UBUNTU
|
||||
# package and does not exist in Debian at all -- `apt-cache policy update-notifier-common` on
|
||||
# trixie returns `Candidate: (none)`, so the play failed outright with "No package matching
|
||||
# 'update-notifier-common' is available" on all three nodes. The sentinel writer is now provided
|
||||
# by this role itself; see unattended_upgrades_sentinel_* below and tasks/main.yml.
|
||||
# Restarts services still mapped to a deleted library after a security upgrade. Without it a
|
||||
# security-only policy patches bytes on disk that no running process reads — see DECISION 3.
|
||||
- needrestart
|
||||
|
||||
# --- Allowed origins -----------------------------------------------------------------------------
|
||||
# Verbatim apt origin expressions, substituted into 50unattended-upgrades. The ${distro_*} tokens
|
||||
# here are UNATTENDED-UPGRADES' OWN placeholders, expanded by it at runtime — they are not Ansible
|
||||
# or Flux substitutions, which is why they are single-quoted and why this role's files never travel
|
||||
# through Flux envsubst.
|
||||
#
|
||||
# `l=Debian-Security` plus the `-security` codename is the form Debian's own shipped config uses, and
|
||||
# both halves were confirmed against the measured archive metadata above (l=Debian-Security,
|
||||
# n=trixie-security). Matching on the LABEL rather than the suite name is what makes this survive the
|
||||
# next Debian release.
|
||||
unattended_upgrades_allowed_origins:
|
||||
- 'origin=Debian,codename=${distro_codename},label=Debian-Security'
|
||||
- 'origin=Debian,codename=${distro_codename}-security,label=Debian-Security'
|
||||
# Set this to add 'origin=Debian,codename=${distro_codename}-updates,label=Debian' — a deliberate
|
||||
# widening, kept as a knob so that turning it on is a reviewable one-line change rather than an edit
|
||||
# to a heredoc. See DECISION 1 for why it is off.
|
||||
unattended_upgrades_include_updates: false
|
||||
|
||||
# --- Package blacklist ---------------------------------------------------------------------------
|
||||
# Regular expressions matched against package names. See DECISION 2: `linux-` is the EFI-memattr
|
||||
# guard and must not be removed without re-reading PROD-PROVISIONING.md §0b.
|
||||
unattended_upgrades_blacklist:
|
||||
# Every kernel package: linux-image-*, linux-headers-*, linux-image-amd64 (the metapackage that
|
||||
# actually tracks backports), plus linux-kbuild / linux-libc-dev. Anchored at the start so it
|
||||
# cannot accidentally match an unrelated package containing the substring.
|
||||
- '^linux-'
|
||||
# Microcode changes take effect only on reboot and can alter boot behaviour on the very firmware
|
||||
# that is already misbehaving here. Operator-only, same argument as the kernel.
|
||||
- '^intel-microcode$'
|
||||
- '^amd64-microcode$'
|
||||
# The bootloader. A grub upgrade rewrites boot configuration on a machine with a documented
|
||||
# warm-reboot hang (k3s-prod-2, 2026-09-03). Never unattended.
|
||||
- '^grub-'
|
||||
- '^grub2-common$'
|
||||
|
||||
# --- Reboot policy -------------------------------------------------------------------------------
|
||||
# FALSE, AND THIS IS THE DIVISION OF LABOUR WITH kured, NOT AN OVERSIGHT.
|
||||
# unattended-upgrades can reboot a node itself (Automatic-Reboot "true"). It must not here: it has
|
||||
# no notion of a cluster, so it would not cordon, would not drain, would not evict the 25 attached
|
||||
# Longhorn volumes cleanly, would not check whether another node is already down, and — on three
|
||||
# control-plane nodes where etcd quorum is 2 of 3 — three nodes each deciding independently to
|
||||
# reboot at their own `Automatic-Reboot-Time` is precisely how you lose the cluster.
|
||||
# u-u therefore only ever leaves the sentinel behind; kured owns every reboot, one node at a time,
|
||||
# under a cluster-wide lock. See platform/kured/helmrelease.yaml.
|
||||
unattended_upgrades_automatic_reboot: false
|
||||
|
||||
# --- needrestart (DECISION 3) --------------------------------------------------------------------
|
||||
# See the role README for the full argument. Summary: 'a' (restart services automatically) is what
|
||||
# makes a security-only policy actually effective — a patched libssl on disk does nothing for a
|
||||
# process still mapping the old one — but k3s.service is EXCLUDED, because restarting the
|
||||
# Kubernetes control plane unattended at 06:00 is a worse outcome than a delayed library fix.
|
||||
# Measured 2026-10-17: the only matching unit on these nodes is `k3s.service` (no separate
|
||||
# containerd or kubelet unit exists; k3s bundles both), so this blacklist is complete today.
|
||||
unattended_upgrades_needrestart_mode: a
|
||||
unattended_upgrades_needrestart_blacklist:
|
||||
- '^k3s'
|
||||
|
||||
# --- Schedule ------------------------------------------------------------------------------------
|
||||
# apt's own systemd timers (apt-daily, apt-daily-upgrade) drive u-u; these set their windows.
|
||||
# 04:00 with a 1h randomised delay, so the upgrade lands INSIDE kured's 03:00-06:00 reboot window
|
||||
# (platform/kured/helmrelease.yaml) rather than hours away from it: if an upgrade ever does set the
|
||||
# sentinel, the reboot can be taken the same night instead of waiting ~24h. The randomised delay
|
||||
# keeps the three nodes from hitting the mirror simultaneously; it does NOT serialise anything, and
|
||||
# it does not need to — u-u never reboots, so three concurrent upgrades are harmless.
|
||||
unattended_upgrades_download_time: "03:30"
|
||||
unattended_upgrades_upgrade_time: "04:00"
|
||||
unattended_upgrades_randomised_delay: 60m
|
||||
|
||||
|
||||
# ── THE REBOOT SENTINEL, PROVIDED BY THIS ROLE ──────────────────────────────────────────────────
|
||||
# /var/run/reboot-required is the ONLY thing kured watches, and on Debian trixie **nothing creates
|
||||
# it**. Measured on all three nodes 2026-10-18:
|
||||
# * `update-notifier-common` (which owns that hook on Ubuntu) has `Candidate: (none)` — the
|
||||
# package does not exist in Debian. Installing it is what broke the first run of this play.
|
||||
# * `/usr/share/update-notifier/notify-reboot-required` is absent.
|
||||
# * NO `linux-image-*` postinst references `notify-reboot-required` — calling it is an Ubuntu
|
||||
# patch, not Debian behaviour, so merely dropping a script at that path would never be invoked.
|
||||
# * No hook under /etc/apt/apt.conf.d or /etc/kernel/ mentions reboot-required.
|
||||
# So the writer has to be ours. Without it kured is a permanent, INVISIBLE no-op: it stays 1/1
|
||||
# Running and reports kured_reboot_required=0 forever, which is the "failure mode is a green tick"
|
||||
# shape this repo keeps getting burned by.
|
||||
#
|
||||
# MECHANISM: a dpkg Post-Invoke hook runs a script that compares the RUNNING kernel against the
|
||||
# newest INSTALLED one and touches the sentinel when they differ. Deliberately kernel-only, which
|
||||
# matches the design: unattended-upgrades is security-only with `^linux-` blacklisted, so it will
|
||||
# essentially never set this — the sentinel appears when the OPERATOR deliberately runs the
|
||||
# node_baseline kernel play, and kured then automates only the mechanical reboot.
|
||||
#
|
||||
# It uses `linux-version` (from `linux-base`, a dependency of every linux-image package, so
|
||||
# guaranteed present — measured `linux-base 4.12.1` installed) rather than parsing `dpkg -l`
|
||||
# output, because kernel version ordering is not string ordering.
|
||||
#
|
||||
# /run is a tmpfs (measured), so the sentinel self-clears on boot: after a reboot the running
|
||||
# kernel IS the newest and the script writes nothing. No cleanup task is needed, and none exists.
|
||||
unattended_upgrades_sentinel_script: /usr/local/sbin/k3s-notify-reboot-required
|
||||
unattended_upgrades_sentinel_hook: /etc/apt/apt.conf.d/99-k3s-reboot-required
|
||||
@@ -0,0 +1,29 @@
|
||||
---
|
||||
# Re-arm apt's periodic timers after a config or drop-in change.
|
||||
#
|
||||
# WHY A daemon_reload IS REQUIRED AND NOT OPTIONAL. The timer windows are set by drop-ins under
|
||||
# /etc/systemd/system/apt-daily{,-upgrade}.timer.d/. systemd does not re-read unit files on its own,
|
||||
# so until it is told to, the RUNNING manager keeps the shipped schedule and the new window is
|
||||
# ignored — the files look right in git and on disk, `systemctl cat` shows them, and the upgrade
|
||||
# still fires at the old time. That is a silent config no-op, which is the failure mode this whole
|
||||
# role is written against.
|
||||
#
|
||||
# `daemon_reload` re-parses unit files and re-runs generators. It starts and stops nothing, so it
|
||||
# cannot disturb k3s, containerd or Longhorn — the same reasoning node_baseline's "Reload systemd
|
||||
# unit files" handler records.
|
||||
#
|
||||
# RESTARTING THE TIMERS IS ALSO SAFE, and is not the same as running the upgrade. `*.timer` units
|
||||
# only hold the schedule; restarting one recomputes its next elapse and does NOT trigger its
|
||||
# `*.service`. With `Persistent=true` set in the drop-in, a window missed while the node was down is
|
||||
# caught up on the next boot rather than skipped.
|
||||
#
|
||||
# Both fire only when a task actually changed, so a re-run of the role is silent.
|
||||
- name: Restart unattended-upgrades timers
|
||||
ansible.builtin.systemd_service:
|
||||
name: "{{ item }}"
|
||||
state: restarted
|
||||
enabled: true
|
||||
daemon_reload: true
|
||||
loop:
|
||||
- apt-daily.timer
|
||||
- apt-daily-upgrade.timer
|
||||
@@ -0,0 +1,369 @@
|
||||
---
|
||||
# roles/unattended_upgrades — automatic Debian security patching, kernel excluded, reboots delegated
|
||||
# to kured. Every decision and its reasoning lives in defaults/main.yml; this file is the mechanism.
|
||||
#
|
||||
# TAGS — following roles/node_baseline, every task carries one of `packages`, `config` or `verify`.
|
||||
# An untagged run does all three in that order. The split exists because `verify` is the half an
|
||||
# operator will want to re-run on its own to answer "is automatic patching actually working on this
|
||||
# box", without re-writing any configuration:
|
||||
# ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify
|
||||
#
|
||||
# IT DOES NOT UPGRADE ANYTHING, AND IT DOES NOT REBOOT. This role installs and CONFIGURES the
|
||||
# upgrader; it never runs `apt upgrade` itself and never takes a reboot. Both of those are deliberate
|
||||
# and both follow roles/node_baseline/tasks/main.yml:61-67 ("IT DOES NOT REBOOT", after k3s-prod-2
|
||||
# hung on a warm reboot on 2026-09-03 and needed a physical power button). Applying this role is
|
||||
# therefore a SAFE operation on a live cluster at any time of day: the first unattended upgrade
|
||||
# happens at the next timer firing, and any reboot it implies is kured's to schedule inside the
|
||||
# 03:00-06:00 window.
|
||||
- name: Automatic security upgrades (unattended-upgrades; kernel excluded, reboots via kured)
|
||||
when: unattended_upgrades_enabled | bool
|
||||
tags: [packages, config, verify]
|
||||
block:
|
||||
|
||||
# ── PREFLIGHT ───────────────────────────────────────────────────────────────────────────────
|
||||
- name: Assert this is a Debian-family host
|
||||
# The config file layout, the origin expressions and the package names below are all
|
||||
# Debian-specific. On anything else this role would write files that are silently ignored,
|
||||
# which is the failure mode this whole PR exists to avoid.
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- ansible_facts['os_family'] == 'Debian'
|
||||
fail_msg: >-
|
||||
REFUSING TO CONFIGURE unattended-upgrades on {{ inventory_hostname }}: os_family is
|
||||
'{{ ansible_facts["os_family"] }}', not 'Debian'. The allowed-origin expressions and
|
||||
config paths in this role are Debian-specific.
|
||||
tags: [packages, config, verify]
|
||||
|
||||
# ── PACKAGES ────────────────────────────────────────────────────────────────────────────────
|
||||
- name: Install unattended-upgrades and needrestart
|
||||
# The reboot-sentinel writer is NOT a package here. On Debian nothing ships one -- see
|
||||
# defaults/main.yml -- so this role installs its own script and dpkg hook, two tasks below.
|
||||
ansible.builtin.apt:
|
||||
name: "{{ unattended_upgrades_packages }}"
|
||||
state: present
|
||||
update_cache: true
|
||||
cache_valid_time: 3600
|
||||
tags: [packages]
|
||||
|
||||
# ── THE REBOOT SENTINEL WRITER (ours, because Debian ships none) ────────────────────────────
|
||||
# Full rationale and the four measurements behind it: defaults/main.yml. Short version: kured
|
||||
# watches /var/run/reboot-required and nothing on Debian creates it, so without these two files
|
||||
# the entire reboot half is an invisible no-op.
|
||||
- name: Install the reboot-sentinel writer
|
||||
ansible.builtin.copy:
|
||||
dest: "{{ unattended_upgrades_sentinel_script }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
content: |
|
||||
#!/bin/sh
|
||||
# MANAGED BY ANSIBLE -- infra/ansible/roles/unattended_upgrades. Local edits are reverted.
|
||||
#
|
||||
# Touch /var/run/reboot-required when the RUNNING kernel is not the newest INSTALLED one.
|
||||
# kured watches that path; see the role's defaults/main.yml for why this is ours to ship.
|
||||
#
|
||||
# `linux-version` comes from linux-base (a dependency of every linux-image package), and
|
||||
# is used rather than sorting `dpkg -l` output because kernel version ordering is not
|
||||
# string ordering. If it is somehow missing, exit 0 and write nothing: a dpkg hook must
|
||||
# never fail an apt transaction, and a missing sentinel is caught by the verify assert.
|
||||
set -eu
|
||||
command -v linux-version >/dev/null 2>&1 || exit 0
|
||||
|
||||
running="$(uname -r)"
|
||||
newest="$(linux-version list 2>/dev/null | linux-version sort --reverse 2>/dev/null | head -n1)"
|
||||
|
||||
# No newest (nothing installed we can read) -> nothing to say.
|
||||
[ -n "$newest" ] || exit 0
|
||||
|
||||
if [ "$running" != "$newest" ]; then
|
||||
touch /var/run/reboot-required 2>/dev/null || exit 0
|
||||
# kured only needs the file, but the .pkgs companion is what an operator reads to find
|
||||
# out WHY a node is waiting, so write a reason rather than an empty marker.
|
||||
printf 'linux-image-%s\n' "$newest" > /var/run/reboot-required.pkgs 2>/dev/null || true
|
||||
fi
|
||||
exit 0
|
||||
tags: [packages, config]
|
||||
|
||||
- name: Install the dpkg hook that runs the sentinel writer
|
||||
# Post-Invoke, so it runs after dpkg has finished and a newly installed kernel is already
|
||||
# visible to `linux-version list`. `|| true` so a failure here can never abort an apt
|
||||
# transaction -- an unwritten sentinel is a missed reboot, but a broken apt is a broken node.
|
||||
ansible.builtin.copy:
|
||||
dest: "{{ unattended_upgrades_sentinel_hook }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: |
|
||||
// MANAGED BY ANSIBLE -- infra/ansible/roles/unattended_upgrades. Local edits reverted.
|
||||
// Writes /var/run/reboot-required when the running kernel is not the newest installed.
|
||||
DPkg::Post-Invoke { "{{ unattended_upgrades_sentinel_script }} || true"; };
|
||||
tags: [packages, config]
|
||||
|
||||
# ── CONFIG ──────────────────────────────────────────────────────────────────────────────────
|
||||
# Written as a whole file rather than patched in place, for the reason node_baseline gives for
|
||||
# its own apt source: an idempotent full-file write with a managed-by header is reviewable and
|
||||
# re-runnable, whereas line-editing a conffile fights dpkg on every package upgrade.
|
||||
#
|
||||
# 52- RATHER THAN 50-: Debian ships /etc/apt/apt.conf.d/50unattended-upgrades as a CONFFILE of
|
||||
# the unattended-upgrades package. Overwriting it would make every future apt upgrade of that
|
||||
# package prompt about a locally-modified conffile (or, under the non-interactive defaults a
|
||||
# scripted run uses, silently keep one side and discard the other). apt reads the directory in
|
||||
# lexical order and later files win, so a separate 52- file overrides the shipped defaults
|
||||
# without ever touching them, and `apt-config dump` in the verify block below proves which value
|
||||
# actually won rather than assuming this ordering holds.
|
||||
- name: Configure unattended-upgrades (52-k3s-unattended-upgrades)
|
||||
ansible.builtin.copy:
|
||||
dest: /etc/apt/apt.conf.d/52-k3s-unattended-upgrades
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: |
|
||||
// MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades. Local edits are reverted
|
||||
// on the next run. Rationale for every value: that role's defaults/main.yml.
|
||||
//
|
||||
// Overrides the shipped 50unattended-upgrades by lexical ordering (52 > 50). The shipped
|
||||
// file is a dpkg conffile and is deliberately left untouched.
|
||||
|
||||
// SECURITY ONLY. Deliberately excludes trixie-updates (non-security fixes) and
|
||||
// trixie-backports (o=Debian Backports) -- and excluding backports is what makes it
|
||||
// structurally impossible for this upgrader to move the EFI-memattr-critical kernel,
|
||||
// which is installed from there. See defaults/main.yml DECISION 2.
|
||||
Unattended-Upgrade::Origins-Pattern {
|
||||
{% for origin in unattended_upgrades_allowed_origins %}
|
||||
"{{ origin }}";
|
||||
{% endfor %}
|
||||
{% if unattended_upgrades_include_updates | bool %}
|
||||
"origin=Debian,codename=${distro_codename}-updates,label=Debian";
|
||||
{% endif %}
|
||||
};
|
||||
|
||||
// NEVER AUTOMATICALLY UPGRADED, whatever the origin list later says. `^linux-` is the
|
||||
// EFI-memattr guard: these HP minis need kernel >= 6.13 or they can boot with a fraction
|
||||
// of their RAM, and 6.12 kernels are still installed and bootable on all three nodes.
|
||||
// Do not remove without re-reading infra/PROD-PROVISIONING.md §0b.
|
||||
Unattended-Upgrade::Package-Blacklist {
|
||||
{% for pkg in unattended_upgrades_blacklist %}
|
||||
"{{ pkg }}";
|
||||
{% endfor %}
|
||||
};
|
||||
|
||||
// REBOOTS BELONG TO kured (platform/kured), NOT TO apt. All three nodes are
|
||||
// control-plane + etcd with quorum 2 of 3, so three hosts independently deciding to
|
||||
// reboot at their own Automatic-Reboot-Time is how the cluster is lost. u-u leaves
|
||||
// /var/run/reboot-required behind; kured takes a cluster-wide lock and reboots exactly
|
||||
// one node at a time, after cordon + drain, inside 03:00-06:00, gated on Longhorn volume
|
||||
// health.
|
||||
Unattended-Upgrade::Automatic-Reboot "{{ unattended_upgrades_automatic_reboot | bool | lower }}";
|
||||
Unattended-Upgrade::Automatic-Reboot-WithUsers "false";
|
||||
|
||||
// Keep the node's disk from filling with superseded .debs, and keep a log an operator can
|
||||
// read after the fact (/var/log/unattended-upgrades/). MinAge guards against removing a
|
||||
// package still needed by a rollback.
|
||||
Unattended-Upgrade::Remove-Unused-Kernel-Packages "false";
|
||||
Unattended-Upgrade::Remove-New-Unused-Dependencies "true";
|
||||
Unattended-Upgrade::Remove-Unused-Dependencies "false";
|
||||
Unattended-Upgrade::AutoFixInterruptedDpkg "true";
|
||||
Unattended-Upgrade::MinimalSteps "true";
|
||||
Unattended-Upgrade::SyslogEnable "true";
|
||||
|
||||
// Remove-Unused-Kernel-Packages is FALSE on purpose, against the Debian default. These
|
||||
// nodes deliberately carry more than one kernel: the backports kernel they boot and the
|
||||
// distro kernels that are the documented rollback path (node_baseline writes the rollback
|
||||
// procedure onto the node itself). An unattended process pruning kernels could remove the
|
||||
// fallback an operator needs at 3am in front of a box that will not boot.
|
||||
notify: Restart unattended-upgrades timers
|
||||
tags: [config]
|
||||
|
||||
- name: Enable the apt periodic timers that actually run unattended-upgrades
|
||||
# Installing the package is NOT sufficient to make it run on a schedule: without these
|
||||
# counters apt's daily units do nothing, and the whole mechanism is inert while looking
|
||||
# installed. `1` means "every day".
|
||||
ansible.builtin.copy:
|
||||
dest: /etc/apt/apt.conf.d/20auto-upgrades
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: |
|
||||
// MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades.
|
||||
// Without these four counters the apt-daily timers fire and do nothing, so
|
||||
// unattended-upgrades would be installed, configured, and never once invoked.
|
||||
APT::Periodic::Update-Package-Lists "1";
|
||||
APT::Periodic::Download-Upgradeable-Packages "1";
|
||||
APT::Periodic::Unattended-Upgrade "1";
|
||||
APT::Periodic::AutocleanInterval "7";
|
||||
tags: [config]
|
||||
|
||||
- name: Ensure the systemd timer drop-in directories exist
|
||||
# `copy` does not create parent directories, and these two .timer.d/ dirs do not exist on a
|
||||
# stock Debian node — without this task the next task fails with "Destination directory does
|
||||
# not exist" on a first run.
|
||||
ansible.builtin.file:
|
||||
path: "/etc/systemd/system/{{ item }}.timer.d"
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
loop:
|
||||
- apt-daily
|
||||
- apt-daily-upgrade
|
||||
tags: [config]
|
||||
|
||||
- name: Pin the apt timer windows into kured's reboot window
|
||||
# Overriding the shipped timers rather than editing them: a drop-in survives package upgrades.
|
||||
# OnCalendar is cleared first (`OnCalendar=` with no value) because systemd ACCUMULATES
|
||||
# OnCalendar entries across drop-ins — omit the reset and the unit keeps its shipped schedule
|
||||
# IN ADDITION to ours, which would run the upgrade at both times and look like it worked.
|
||||
ansible.builtin.copy:
|
||||
dest: "/etc/systemd/system/{{ item.unit }}.timer.d/override.conf"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: |
|
||||
# MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades.
|
||||
# Lands the upgrade inside kured's 03:00-06:00 reboot window so a sentinel written by an
|
||||
# upgrade can be acted on the same night rather than ~24h later.
|
||||
[Timer]
|
||||
OnCalendar=
|
||||
OnCalendar={{ item.time }}
|
||||
RandomizedDelaySec={{ unattended_upgrades_randomised_delay }}
|
||||
Persistent=true
|
||||
force: true
|
||||
loop:
|
||||
- { unit: apt-daily, time: "{{ unattended_upgrades_download_time }}" }
|
||||
- { unit: apt-daily-upgrade, time: "{{ unattended_upgrades_upgrade_time }}" }
|
||||
loop_control:
|
||||
label: "{{ item.unit }} at {{ item.time }}"
|
||||
notify: Restart unattended-upgrades timers
|
||||
tags: [config]
|
||||
|
||||
- name: Configure needrestart (restart services, but never k3s)
|
||||
# See defaults/main.yml DECISION 3. Auto-restart is what makes a security-only policy real --
|
||||
# a patched library on disk does nothing for a process still mapping the deleted one -- but
|
||||
# k3s.service is excluded because restarting the control plane unattended is worse than a
|
||||
# delayed library fix. `$nrconf{restart}` is Perl config read by needrestart itself.
|
||||
ansible.builtin.copy:
|
||||
dest: /etc/needrestart/conf.d/50-k3s.conf
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: |
|
||||
# MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades.
|
||||
# Restart mode: '{{ unattended_upgrades_needrestart_mode }}' (a = automatically, l = list only, i = interactive)
|
||||
$nrconf{restart} = '{{ unattended_upgrades_needrestart_mode }}';
|
||||
# Units matching these patterns are NEVER restarted automatically. k3s bundles containerd
|
||||
# and the kubelet, so this one entry covers the entire control plane and runtime on these
|
||||
# nodes (measured 2026-10-17: k3s.service is the only matching unit). A library fix that
|
||||
# needs k3s restarted therefore waits for the reboot kured will take.
|
||||
$nrconf{blacklist_rc} = [
|
||||
{% for pat in unattended_upgrades_needrestart_blacklist %}
|
||||
qr({{ pat }}),
|
||||
{% endfor %}
|
||||
];
|
||||
tags: [config]
|
||||
|
||||
# ── VERIFY ──────────────────────────────────────────────────────────────────────────────────
|
||||
# These tasks exist because every component of this mechanism fails SILENTLY. An installed
|
||||
# package, a written config file and a green play are all compatible with nothing ever being
|
||||
# upgraded and no reboot ever being signalled.
|
||||
|
||||
- name: Ask apt-config what the effective settings actually are
|
||||
# `apt-config dump` is apt's OWN parse of the whole /etc/apt/apt.conf.d/ directory, so it
|
||||
# proves what WON after lexical ordering rather than what this role wrote. If the shipped
|
||||
# 50unattended-upgrades were ever renamed above 52-, this is the task that would catch it.
|
||||
ansible.builtin.command: apt-config dump
|
||||
register: unattended_upgrades_aptconfig
|
||||
changed_when: false
|
||||
check_mode: false
|
||||
tags: [verify]
|
||||
|
||||
- name: Assert the kernel is blacklisted and automatic reboots are off
|
||||
# The two properties that must never silently regress. Asserted against apt's own view.
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
# apt-config renders each list entry as `Unattended-Upgrade::Package-Blacklist:: "^linux-";`
|
||||
# so the anchored pattern itself is the thing to look for. Checking for the bare string
|
||||
# `linux-` would pass on almost any dump and prove nothing.
|
||||
- "'\"^linux-\"' in unattended_upgrades_aptconfig.stdout"
|
||||
- "'Unattended-Upgrade::Automatic-Reboot \"false\"' in unattended_upgrades_aptconfig.stdout"
|
||||
# The origin list must be non-empty, or u-u runs nightly and upgrades nothing — installed,
|
||||
# scheduled, green, and inert.
|
||||
- "'Debian-Security' in unattended_upgrades_aptconfig.stdout"
|
||||
fail_msg: >-
|
||||
EFFECTIVE apt CONFIG ON {{ inventory_hostname }} IS NOT WHAT THIS ROLE INTENDED.
|
||||
Either the `^linux-` kernel blacklist or `Automatic-Reboot "false"` is not winning.
|
||||
Both are safety properties, not preferences: without the blacklist an unattended process
|
||||
can move a kernel these HP minis must keep >= 6.13 (PROD-PROVISIONING.md §0b), and
|
||||
without Automatic-Reboot=false three control-plane nodes can each reboot themselves
|
||||
independently and take etcd below quorum. Inspect with `apt-config dump | grep -i
|
||||
unattended` and check for a file sorting after 52- in /etc/apt/apt.conf.d/.
|
||||
success_msg: >-
|
||||
Effective apt config confirmed: kernel blacklisted, automatic reboot disabled.
|
||||
tags: [verify]
|
||||
|
||||
# THE SENTINEL IS THE SINGLE POINT OF FAILURE FOR THE ENTIRE REBOOT HALF OF THIS DESIGN.
|
||||
# This was originally a `find` over /etc/kernel/postinst.d and /etc/apt/apt.conf.d, on the
|
||||
# theory that the writer came from a package and the role could only discover whatever that
|
||||
# package happened to ship. That theory was wrong: no such package exists on Debian (see
|
||||
# defaults/main.yml), so the writer is now OURS and lives at two known paths. Checking two
|
||||
# exact paths is strictly stronger than a glob -- a glob would have matched Debian's own
|
||||
# unattended-upgrades apt conffile and passed while no sentinel writer existed at all.
|
||||
- name: Stat the sentinel writer script
|
||||
ansible.builtin.stat:
|
||||
path: "{{ unattended_upgrades_sentinel_script }}"
|
||||
register: unattended_upgrades_sentinel_script_stat
|
||||
changed_when: false
|
||||
tags: [verify]
|
||||
|
||||
- name: Stat the sentinel dpkg hook
|
||||
ansible.builtin.stat:
|
||||
path: "{{ unattended_upgrades_sentinel_hook }}"
|
||||
register: unattended_upgrades_sentinel_hook_stat
|
||||
changed_when: false
|
||||
tags: [verify]
|
||||
|
||||
- name: Assert something on this node can write the reboot sentinel
|
||||
# If this fails, kured is a no-op: it watches /var/run/reboot-required forever, the file is
|
||||
# never created, no reboot is ever taken, and NOTHING GOES RED -- the DaemonSet stays 1/1
|
||||
# Running and `kured_reboot_required` reports a steady 0 on every node. That is the exact
|
||||
# shape of failure (a check whose failure mode is a green tick) this repo has been burned by
|
||||
# before, so it is an assert and not a debug message.
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- unattended_upgrades_sentinel_script_stat.stat.exists
|
||||
- unattended_upgrades_sentinel_script_stat.stat.executable | default(false)
|
||||
- unattended_upgrades_sentinel_hook_stat.stat.exists
|
||||
fail_msg: >-
|
||||
NOTHING ON {{ inventory_hostname }} CAN CREATE /var/run/reboot-required.
|
||||
{{ unattended_upgrades_sentinel_script }} (executable) and
|
||||
{{ unattended_upgrades_sentinel_hook }} must both exist -- this role ships both, because
|
||||
Debian provides no sentinel writer of its own (update-notifier-common is an UBUNTU
|
||||
package and is not installable here). kured watches that path and reboots only when it
|
||||
exists, so with no writer the reboot automation is permanently and INVISIBLY idle --
|
||||
kured stays 1/1 Running and reports kured_reboot_required=0 on every node forever.
|
||||
Re-run this role without `--tags verify` to install them.
|
||||
success_msg: >-
|
||||
Reboot sentinel writer present ({{ unattended_upgrades_sentinel_script }} + dpkg hook)
|
||||
-- a kernel change will signal kured on the next apt transaction.
|
||||
tags: [verify]
|
||||
|
||||
- name: Report the pending-reboot state of this node
|
||||
# Read-only. Tells the operator whether this node is currently waiting on kured for a reboot,
|
||||
# which is the question `--tags verify` is usually being run to answer.
|
||||
ansible.builtin.stat:
|
||||
path: /var/run/reboot-required
|
||||
register: unattended_upgrades_reboot_required
|
||||
changed_when: false
|
||||
tags: [verify]
|
||||
|
||||
- name: Summary
|
||||
ansible.builtin.debug:
|
||||
msg: >-
|
||||
{{ inventory_hostname }}: security-only unattended upgrades ACTIVE
|
||||
(upgrade timer {{ unattended_upgrades_upgrade_time }} +/-{{ unattended_upgrades_randomised_delay }}),
|
||||
kernel and microcode blacklisted, automatic reboot DISABLED.
|
||||
Reboot currently pending: {{ unattended_upgrades_reboot_required.stat.exists }}.
|
||||
{% if unattended_upgrades_reboot_required.stat.exists %}
|
||||
This node is waiting for kured to reboot it in the next 03:00-06:00 window; it will be
|
||||
cordoned and drained first, and no other node will be rebooted at the same time.
|
||||
{% endif %}
|
||||
tags: [verify]
|
||||
@@ -1,112 +0,0 @@
|
||||
---
|
||||
# =================================================================================================
|
||||
# coredns HA floor — hold in-cluster DNS at >= 2 replicas, which k3s does not.
|
||||
# =================================================================================================
|
||||
# MEASURED on prod 2026-10-18: `kube-system/coredns` ran with **replicas: 1**. In-cluster DNS is a
|
||||
# single pod on a single node, so **draining or rebooting that one node is a cluster-wide
|
||||
# in-cluster DNS outage** for as long as the pod takes to reschedule and become ready — every pod
|
||||
# that looks up a Service, every CNPG replica finding its primary, every operator talking to the
|
||||
# API by name. It is the single cheapest availability defect in the cluster, and it was invisible
|
||||
# because `1/1 Ready` looks perfectly healthy.
|
||||
#
|
||||
# This matters more now than it did: the reboot automation (kured + unattended-upgrades) drains and
|
||||
# reboots one node at a time on a schedule, which turns a latent single point of failure into a
|
||||
# recurring one, unattended, in the middle of the night.
|
||||
#
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# WHY AN HPA, AND NOT JUST `replicas: 2` — this is the whole decision
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# **k3s owns the coredns Deployment and we cannot durably edit it.** It is applied by k3s's own
|
||||
# deploy controller from a bundled manifest, not by Flux:
|
||||
#
|
||||
# $ kubectl -n kube-system get addon coredns
|
||||
# NAME SOURCE CHECKSUM
|
||||
# coredns /data/k3s/server/manifests/coredns.yaml 8cd91595...
|
||||
# $ kubectl -n kube-system get deploy coredns -o jsonpath='{.metadata.managedFields[*].manager}'
|
||||
# deploy@k3s-prod-1 k3s
|
||||
#
|
||||
# and the Deployment carries an `objectset.rio.cattle.io/hash` label — a wrangler objectset. k3s
|
||||
# RE-STAGES its bundled manifests on startup, so a `replicas: 2` written into that file (or a
|
||||
# `kubectl scale`) is reverted the next time k3s restarts or is upgraded. MEASURED the same day: a
|
||||
# hand `kubectl scale` to 2 was NOT actively reverted (it held, and the AddOn checksum did not
|
||||
# change) — the deploy controller re-applies on manifest change and on restart, not on a drift
|
||||
# timer. So the hand fix survives until exactly the event we are protecting against: a reboot.
|
||||
#
|
||||
# The two durable alternatives were weighed and rejected:
|
||||
# * **`coredns.yaml.skip` + our own templated copy** (the pattern roles/kube_vip already uses for
|
||||
# a manifest in server/manifests). Fully deterministic and never flaps — but it takes OWNERSHIP
|
||||
# of coredns away from k3s permanently, so k3s upgrades would stop updating it and the version
|
||||
# would have to be refreshed by hand forever. Too high a price for a replica count.
|
||||
# * **A Flux-managed patch of the Deployment itself.** Rejected outright: Flux and k3s's deploy
|
||||
# controller would both own one object and fight continuously. That is the shape of the
|
||||
# `default-deny` NetworkPolicy drift loop this cluster already suffered (Kyverno generate + Flux
|
||||
# both owning one resource, ~2500 events per 10 minutes).
|
||||
#
|
||||
# An HPA avoids both. It is a SEPARATE object that k3s does not own and will never reconcile away,
|
||||
# and it drives the Deployment's `/scale` subresource rather than its spec — so there is no
|
||||
# shared-ownership fight. If k3s re-stages `replicas: 1`, the HPA observes it under `minReplicas`
|
||||
# and scales back to 2 within one reconcile (~15s by default).
|
||||
#
|
||||
# THE COST, STATED PLAINLY: there is a brief window — seconds, after a k3s restart — where coredns
|
||||
# is back at 1 replica until the HPA notices. That is strictly better than the status quo, where it
|
||||
# is at 1 replica permanently, but it is not zero. Accepting that window is the trade being made
|
||||
# here; the `.skip` option above is what to reach for if it ever proves unacceptable.
|
||||
#
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# THIS IS A FLOOR-HOLDER, NOT A REAL AUTOSCALER
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# `minReplicas: 2` is the entire point. The CPU metric exists because an autoscaling/v2 HPA needs a
|
||||
# metric to be a valid object, not because coredns is expected to scale on load: measured via
|
||||
# metrics-server, the two pods use **5m and 4m** against a **100m** request — ~5% utilisation, so
|
||||
# the 80% target will never be approached and the HPA will sit at `minReplicas` indefinitely.
|
||||
#
|
||||
# The CPU request is what makes this legal: coredns ships `requests: {cpu: 100m, memory: 70Mi}`
|
||||
# (measured), and a Resource/Utilization metric is meaningless without a request. metrics-server is
|
||||
# installed as a k3s addon and verified serving (`kubectl top pod -l k8s-app=kube-dns` returns
|
||||
# figures), so the metric resolves rather than leaving the HPA with `ScalingActive=False`.
|
||||
#
|
||||
# `maxReplicas: 3` = the node count. coredns' pod anti-affinity prefers one pod per node, so a
|
||||
# fourth replica would add no failure-domain coverage; the ceiling exists so a metrics glitch can
|
||||
# never balloon DNS pods across the cluster.
|
||||
apiVersion: autoscaling/v2
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata:
|
||||
name: coredns
|
||||
# kube-system, because that is where k3s puts coredns. This is the only object this component
|
||||
# places outside a namespace it owns, and it is additive: it creates nothing in kube-system and
|
||||
# mutates only the coredns Deployment's scale subresource.
|
||||
namespace: kube-system
|
||||
labels:
|
||||
app.kubernetes.io/name: coredns
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: coredns
|
||||
minReplicas: 2
|
||||
maxReplicas: 3
|
||||
metrics:
|
||||
- type: Resource
|
||||
resource:
|
||||
name: cpu
|
||||
target:
|
||||
type: Utilization
|
||||
# Deliberately high. See "floor-holder" above: this must not scale on normal DNS load.
|
||||
averageUtilization: 80
|
||||
behavior:
|
||||
scaleDown:
|
||||
# 10 minutes. DNS is the worst thing in the cluster to flap: a scale-down races in-flight
|
||||
# queries and every client's negative cache. The HPA should be reluctant in this direction.
|
||||
stabilizationWindowSeconds: 600
|
||||
policies:
|
||||
- type: Pods
|
||||
value: 1
|
||||
periodSeconds: 600
|
||||
scaleUp:
|
||||
# Prompt in the direction that restores redundancy — this is the path taken after a k3s
|
||||
# restart has re-staged `replicas: 1`.
|
||||
stabilizationWindowSeconds: 0
|
||||
policies:
|
||||
- type: Pods
|
||||
value: 2
|
||||
periodSeconds: 60
|
||||
@@ -7,7 +7,3 @@ apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- coredns-custom.yaml
|
||||
# HA floor for the coredns Deployment itself — a separate concern from the split-horizon server
|
||||
# block above, and deliberately an HPA rather than a replica edit: k3s owns that Deployment via
|
||||
# its own deploy controller and re-stages `replicas: 1` on restart. hpa.yaml has the reasoning.
|
||||
- hpa.yaml
|
||||
@@ -0,0 +1,127 @@
|
||||
# platform/kured
|
||||
|
||||
**kured (Kubernetes Reboot Daemon)** — coordinated, one-node-at-a-time OS reboots of the three prod
|
||||
control-plane nodes.
|
||||
|
||||
- **Flux tier:** `clusters/base/kured.yaml` (its own Kustomization, `dependsOn: [platform]`,
|
||||
`wait: true`)
|
||||
- **Chart:** `kured` 6.1.0 → appVersion 1.23.0, from `https://kubereboot.github.io/charts`
|
||||
- **The other half:** `infra/ansible/roles/unattended_upgrades` — the package upgrades. **Neither
|
||||
half is useful alone.**
|
||||
- **Runbook:** `/runbooks/node-reboot-automation`
|
||||
- **Every tuned value and its reasoning:** `helmrelease.yaml`. That file is the argument; this one is
|
||||
the orientation.
|
||||
- **Design record:** `decisions/2026-10-17-automated-upgrades.md`
|
||||
|
||||
## What this component owns
|
||||
|
||||
The *mechanics* of a reboot: take a cluster-wide lock, cordon, drain, reboot, uncordon, hold the
|
||||
lock a while longer. Nothing else.
|
||||
|
||||
**It does not decide whether to reboot.** kured reboots a node if and only if
|
||||
`/var/run/reboot-required` exists there. On these nodes that file is written by a deliberate
|
||||
operator kernel upgrade (`playbooks/node-baseline.yml --tags kernel`) — unattended-upgrades is
|
||||
security-only with the kernel blacklisted, so it essentially never produces one. **The human gate is
|
||||
the sentinel, not the daemon.**
|
||||
|
||||
## The problem this fixes
|
||||
|
||||
Rebooting these three nodes was a manual, serialized, supervised procedure, and
|
||||
`roles/node_baseline` has to end its kernel run by shouting the rules at the operator: *"Reboot prod
|
||||
nodes ONE AT A TIME, verify each returns before touching the next, and NEVER node 2 first"*. That is
|
||||
a correct procedure and an error-prone one, and nothing enforced it.
|
||||
|
||||
What it has to get right, measured on prod 2026-10-17:
|
||||
|
||||
- **etcd quorum is 2 of 3** — all three nodes are control-plane+etcd. Two down is a lost cluster.
|
||||
- **Every drain degrades all 25 Longhorn volumes to 2/3**, because 3 nodes × 3 replicas ×
|
||||
`replica-soft-anti-affinity: false` leaves nowhere to place a replacement, and rebuild is
|
||||
serialized behind a 600s wait. Reboot the next node too early and volumes hit 1/3, where
|
||||
`block-if-contains-last-replica` can block a drain forever.
|
||||
- **Six PodDisruptionBudgets allow zero voluntary disruptions** — 3 Longhorn `instance-manager-*`
|
||||
(structurally 0 by design; this is how Longhorn gates the drain) and 3 CNPG `*-primary`.
|
||||
|
||||
## Placement — why its own Kustomization
|
||||
|
||||
`platform` is `wait: true` over twelve components and gates `platform-config` → every apps wave. A
|
||||
reboot daemon that failed to go healthy there would stall **every application in the cluster** on
|
||||
"dependency not ready". A component added to make maintenance safer must not be able to cause an
|
||||
outage. `wait: true` here is a health **signal** for something nothing depends on — and nothing
|
||||
should ever `dependsOn: [kured]`.
|
||||
|
||||
The `dependsOn: [platform]` edge is load-bearing, not conventional: it supplies the
|
||||
`monitoring.coreos.com` CRDs this component's `PrometheusRule` needs (F-MON1 — a pass containing an
|
||||
unknown kind aborts), plus the Prometheus that kured **queries** before every reboot.
|
||||
|
||||
## How concurrent reboots are made impossible
|
||||
|
||||
| mechanism | value | why |
|
||||
|---|---|---|
|
||||
| cluster-wide lock | one annotation on kured's own DaemonSet | a node reboots only while holding it; enforced by API-server optimistic concurrency, not timing |
|
||||
| `concurrency` | `1` | one lock holder at a time |
|
||||
| **`lockTtl`** | **`0` (disabled)** | a TTL lets reboots continue after the holder dies — on 3 etcd nodes that is the catastrophe. A hung node holds the lock **forever** and nothing cascades. |
|
||||
| `lockReleaseDelay` | `30m` | the next node waits out Longhorn's 600s replenishment even if no alert latched |
|
||||
| alert gate | `alertFilterMatchOnly: true` + anchored allow-list | refuses to proceed while a volume is degraded/faulted or a node is down |
|
||||
| window | 03:00–06:00 daily, America/New_York | matches `renovate.json`'s window; daily (not weekly) is what makes `KuredRebootPending`'s `for: 36h` meaningful |
|
||||
|
||||
## Deliberate non-choices
|
||||
|
||||
- **No `--ignore-daemonsets` / `--delete-emptydir-data` / `--force`.** Those flags **do not exist**
|
||||
in kured 1.23.0 — it hardcodes all three in the drain helper (`cmd/kured/main.go:438-449`).
|
||||
Passing them would CrashLoop the DaemonSet. All three are genuinely required here (12 DaemonSets,
|
||||
135 emptyDir pods).
|
||||
- **`rebootMethod: command`, not `signal`.** `signal` would allow `privileged: false` + `CAP_KILL`,
|
||||
but needs a host-side `kured-reboot.service` that does not exist on these nodes. A real future
|
||||
hardening step; explicitly not taken here.
|
||||
- **No tolerations.** No prod node is tainted (measured). A future tainted node would be silently
|
||||
uncovered — hence `KuredNodeCoverageGap`.
|
||||
- **`NodeEtcdRequestsSlow` / `NodeIOPressure*` are NOT in the alert gate**, despite being exactly
|
||||
what you would want to pause for. prod-1 has a documented etcd WAL I/O starvation history and
|
||||
including them risks a permanent wedge that looks like safety.
|
||||
- **No Slack/shoutrrr notifications.** Would require a SOPS Secret and a decryption patch; the three
|
||||
alerts cover the same ground through Alertmanager.
|
||||
|
||||
## Versioning
|
||||
|
||||
Chart pinned `>=6.1.0 <6.2.0`. The minor ceiling is **deliberate here**, unlike elsewhere in this
|
||||
repo where it is a main cause of pin rot: a kured minor may change drain or lock semantics, and this
|
||||
is the one component where "a human read the changelog" should be a precondition. `renovate.json`'s
|
||||
`blast-radius` rule additionally forbids auto-merging it, so the ceiling is a review prompt, not a
|
||||
wall.
|
||||
|
||||
## What breaks silently, and what catches it
|
||||
|
||||
Every failure mode of this component leaves a cluster that looks healthy — kured stays 1/1 Running
|
||||
while rebooting nothing. `alerts.yaml` ships three rules over kured's single metric, with a
|
||||
companion promtool suite:
|
||||
|
||||
| alert | catches |
|
||||
|---|---|
|
||||
| `KuredNotReporting` | the scrape pipeline is dead (chart defaults `service.create`/`metrics.create` to **false**; a NetworkPolicy can time the scrape out) — and the other two are inert while this is true |
|
||||
| `KuredNodeCoverageGap` | kured missing on *some* node, which `absent()` cannot see |
|
||||
| `KuredRebootPending` | a reboot was offered and blocked for >36h |
|
||||
|
||||
## Post-merge verification
|
||||
|
||||
```bash
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds,pod,svc,servicemonitor -o wide'
|
||||
infra/devcontainer/dev.sh run bash -c 'flux --context k3s-prod get kustomization kured'
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured logs ds/kured --tail=40'
|
||||
```
|
||||
|
||||
Expect `3/3` ready, the Kustomization `Ready=True`, and kured logging a reboot-check decision each
|
||||
period. Then confirm the scrape is real — this is the step that proves the alerts are not inert:
|
||||
|
||||
```bash
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n monitoring exec \
|
||||
sts/prometheus-kube-prometheus-stack-prometheus -c prometheus -- \
|
||||
wget -qO- "http://localhost:9090/api/v1/query?query=count(kured_reboot_required)"'
|
||||
```
|
||||
|
||||
**Expect `3`.** Anything else (and especially an empty result) means the Service, ServiceMonitor or
|
||||
NetworkPolicy is wrong and all three alerts are silently dead.
|
||||
|
||||
**Before trusting the timer with a real kernel upgrade**, do one watched drain to settle the open
|
||||
question the design record flags: whether CNPG actually switches its primary away in response to a
|
||||
drain-induced eviction. If it does not, draining `k3s-prod-1` blocks on three 0-disruption PDBs at
|
||||
once.
|
||||
@@ -0,0 +1,282 @@
|
||||
---
|
||||
# promtool unit tests for the three rules in platform/kured/alerts.yaml.
|
||||
#
|
||||
# WHY THIS FILE EXISTS. The rules it covers are the ONLY thing that can tell an operator that the
|
||||
# reboot automation has stopped working, and every failure mode they describe is silent — kured stays
|
||||
# 1/1 Running while rebooting nothing. An alert that cannot fire is therefore not a cosmetic defect
|
||||
# here, it is the removal of the only signal. This repo has shipped that bug in both directions
|
||||
# already (a permanently-firing VPNTunnelDown; a structurally inert VeleroNoRecentBackup over an
|
||||
# absent series), which is why the harness exists at all. Each rule below is pinned BOTH ways: quiet
|
||||
# on the healthy state, firing on the constructed bad one, with the `for:` boundary asserted rather
|
||||
# than assumed.
|
||||
#
|
||||
# HOW TO RUN. `platform/monitoring/tests/run.sh` — the one repo-wide rule-unit harness. It finds this
|
||||
# file by the COMPANION convention (`<manifest>.promtool-test.yaml` beside `<manifest>.yaml`), so the
|
||||
# manifest under test is exactly platform/kured/alerts.yaml and nothing else. Run just this file with
|
||||
# platform/monitoring/tests/run.sh platform/kured/alerts.promtool-test.yaml
|
||||
# The harness flattens the PrometheusRule CR's `.spec.groups` into the bare rules file promtool wants,
|
||||
# `promtool check rules` it first so a non-parsing file cannot pass vacuously, and runs it all in a
|
||||
# temp dir. It leaves Flux's `${BASE_DOMAIN}` in the runbook_url annotations verbatim, which is safe
|
||||
# because the assertions below query the `ALERTS` series and never compare annotations.
|
||||
#
|
||||
# WHY THE ASSERTIONS QUERY `ALERTS` RATHER THAN USING `exp_alerts`. promtool's alert_rule_test
|
||||
# compares annotations VERBATIM, which would mean pasting each rule's description prose in here and
|
||||
# re-pasting it on every wording change. Querying the synthetic `ALERTS` series proves what actually
|
||||
# matters — the alert reaches `pending` and then `firing` across its `for:` boundary, with the right
|
||||
# labels — and stays decoupled from annotation text. Same choice as
|
||||
# platform/cnpg/alerts.promtool-test.yaml.
|
||||
#
|
||||
# THE SERIES SHAPES ARE REAL. `kured_reboot_required{node=...}` is a GaugeVec with exactly one label,
|
||||
# `node` — read from kured's cmd/kured/main.go:92-96 (the only registered metric in the project).
|
||||
# `kube_node_info{node=...}` was measured on prod Prometheus on 2026-10-17 returning
|
||||
# `count(kube_node_info) == 3`, matching the three nodes. The extra kube_node_info labels prod
|
||||
# carries are omitted deliberately: every rule aggregates with `count()`, so they cannot affect the
|
||||
# result, and inventing plausible-but-wrong values would be worse than leaving them out.
|
||||
#
|
||||
# evaluation_interval is 5m, not the usual 1m: KuredRebootPending has `for: 36h`, and at a 1m
|
||||
# interval promtool would evaluate that test over 2000+ steps for no added coverage.
|
||||
rule_files:
|
||||
- alerts.rules.yaml
|
||||
evaluation_interval: 5m
|
||||
|
||||
tests:
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
# 1. THE HEALTHY STEADY STATE. Three nodes, kured running on all three, nothing pending.
|
||||
# ALL THREE RULES MUST BE SILENT. This is the case that a badly-written `absent()` or a
|
||||
# `count() <` comparison with an off-by-one gets wrong, and it is also the state the cluster
|
||||
# is in ~100% of the time, so a false positive here would train the operator to ignore the
|
||||
# whole group.
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
- interval: 5m
|
||||
name: healthy cluster - no reboots pending, kured on every node, all three rules quiet
|
||||
input_series:
|
||||
- series: 'kured_reboot_required{node="k3s-prod-1"}'
|
||||
values: '0x60'
|
||||
- series: 'kured_reboot_required{node="k3s-prod-2"}'
|
||||
values: '0x60'
|
||||
- series: 'kured_reboot_required{node="k3s-prod-3"}'
|
||||
values: '0x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-1"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-2"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-3"}'
|
||||
values: '1x60'
|
||||
promql_expr_test:
|
||||
# Nothing from this group may be pending or firing, at any point in the window.
|
||||
- expr: 'count(ALERTS{alertname=~"Kured.*"})'
|
||||
eval_time: 4h
|
||||
exp_samples: []
|
||||
# And the underlying expressions must each be genuinely false, not merely unevaluated:
|
||||
# absent() yields no series because the vector is populated...
|
||||
- expr: 'absent(kured_reboot_required)'
|
||||
eval_time: 4h
|
||||
exp_samples: []
|
||||
# ...coverage is exact (3 == 3), so the `<` comparison drops the sample...
|
||||
- expr: 'count(kured_reboot_required) < count(kube_node_info)'
|
||||
eval_time: 4h
|
||||
exp_samples: []
|
||||
# ...and no node reports a pending reboot.
|
||||
- expr: 'max by (node) (kured_reboot_required) == 1'
|
||||
eval_time: 4h
|
||||
exp_samples: []
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
# 2. KuredNotReporting — THE WHOLE PIPELINE IS DEAD. No kured series at all, which is what the
|
||||
# cluster looks like if the chart's `service.create`/`metrics.create` regress to their FALSE
|
||||
# defaults, if the ServiceMonitor is dropped, or if a NetworkPolicy silently times out the
|
||||
# scrape (the Longhorn precedent). The nodes still exist, so kube_node_info is present —
|
||||
# proving the rule keys on kured's absence and not on a dead cluster.
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
- interval: 5m
|
||||
name: no kured series anywhere - KuredNotReporting fires after 30m
|
||||
input_series:
|
||||
- series: 'kube_node_info{node="k3s-prod-1"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-2"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-3"}'
|
||||
values: '1x60'
|
||||
promql_expr_test:
|
||||
# The expression itself yields 1 immediately.
|
||||
- expr: 'absent(kured_reboot_required)'
|
||||
eval_time: 10m
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: 1
|
||||
# `for: 30m` is honoured — still only pending at 25m.
|
||||
- expr: 'ALERTS{alertname="KuredNotReporting"}'
|
||||
eval_time: 25m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="KuredNotReporting", alertstate="pending", severity="warning"}'
|
||||
value: 1
|
||||
# ...and firing once past it.
|
||||
- expr: 'ALERTS{alertname="KuredNotReporting", alertstate="firing"}'
|
||||
eval_time: 35m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="KuredNotReporting", alertstate="firing", severity="warning"}'
|
||||
value: 1
|
||||
# THE GUARD CLAUSE: with kured absent, the other two rules are inert — exactly the reason
|
||||
# this rule has to exist. KuredRebootPending cannot fire (no series to be 1) and
|
||||
# KuredNodeCoverageGap cannot either, because `count()` over an empty vector returns NO
|
||||
# SAMPLE rather than 0, so the `<` comparison has nothing to compare. Asserting that here
|
||||
# pins the dependency: if a future edit made coverage-gap fire in this state, the overlap
|
||||
# would be deliberate rather than accidental.
|
||||
- expr: 'ALERTS{alertname="KuredRebootPending"}'
|
||||
eval_time: 35m
|
||||
exp_samples: []
|
||||
- expr: 'ALERTS{alertname="KuredNodeCoverageGap"}'
|
||||
eval_time: 35m
|
||||
exp_samples: []
|
||||
- expr: 'count(kured_reboot_required) < count(kube_node_info)'
|
||||
eval_time: 35m
|
||||
exp_samples: []
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
# 3. KuredNodeCoverageGap — kured COVERS ONLY TWO OF THREE NODES. k3s-prod-3's series stops at
|
||||
# 1h and never returns (a node taint this DaemonSet does not tolerate, a node that never came
|
||||
# back from a reboot, a DaemonSet nodeSelector edit). This also pins the `for: 2h` choice,
|
||||
# which exists SO THAT AN ORDINARY REBOOT DOES NOT TRIP IT: the gap is real from ~1h05 but
|
||||
# the alert is only pending an hour later, and a healthy reboot is minutes.
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
- interval: 5m
|
||||
name: kured missing on one node - KuredNodeCoverageGap fires after 2h, not before
|
||||
input_series:
|
||||
- series: 'kured_reboot_required{node="k3s-prod-1"}'
|
||||
values: '0x60'
|
||||
- series: 'kured_reboot_required{node="k3s-prod-2"}'
|
||||
values: '0x60'
|
||||
# Ends after 1h (13 samples at 5m) and is stale thereafter.
|
||||
- series: 'kured_reboot_required{node="k3s-prod-3"}'
|
||||
values: '0x12'
|
||||
- series: 'kube_node_info{node="k3s-prod-1"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-2"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-3"}'
|
||||
values: '1x60'
|
||||
promql_expr_test:
|
||||
# Before the gap opens, coverage is exact and nothing is pending.
|
||||
- expr: 'ALERTS{alertname="KuredNodeCoverageGap"}'
|
||||
eval_time: 30m
|
||||
exp_samples: []
|
||||
# The shortfall is real well before the alert fires — 2 < 3.
|
||||
- expr: 'count(kured_reboot_required)'
|
||||
eval_time: 2h
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: 2
|
||||
# A SHORT OUTAGE DOES NOT ALERT: still only pending at 2h, an hour after the gap opened.
|
||||
- expr: 'ALERTS{alertname="KuredNodeCoverageGap"}'
|
||||
eval_time: 2h
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="KuredNodeCoverageGap", alertstate="pending", severity="warning"}'
|
||||
value: 1
|
||||
# Firing once the gap has persisted past `for: 2h`.
|
||||
- expr: 'ALERTS{alertname="KuredNodeCoverageGap", alertstate="firing"}'
|
||||
eval_time: 3h15m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="KuredNodeCoverageGap", alertstate="firing", severity="warning"}'
|
||||
value: 1
|
||||
# KuredNotReporting must stay QUIET: two nodes are still publishing, so the vector is not
|
||||
# absent. This is the distinction between "kured is gone" and "kured is incomplete", and
|
||||
# getting it wrong in either direction would collapse two different diagnoses into one.
|
||||
- expr: 'ALERTS{alertname="KuredNotReporting"}'
|
||||
eval_time: 3h15m
|
||||
exp_samples: []
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
# 4. KuredRebootPending — A REBOOT IS BLOCKED. k3s-prod-1 reports a pending reboot continuously:
|
||||
# the sentinel was written (an operator kernel upgrade) and the reboot has not happened across
|
||||
# more than one 03:00-06:00 window. Asserts the 36h boundary in both directions, which is the
|
||||
# assertion that makes the DAILY-window decision load-bearing rather than cosmetic: on a
|
||||
# weekly window this threshold could not distinguish "blocked" from "waiting for Sunday".
|
||||
# Series run to 38h20m (456 samples at 5m) so the post-boundary evaluation has real data.
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
- interval: 5m
|
||||
name: reboot pending on one node for over 36h - KuredRebootPending fires, per-node label kept
|
||||
input_series:
|
||||
- series: 'kured_reboot_required{node="k3s-prod-1"}'
|
||||
values: '1x456'
|
||||
- series: 'kured_reboot_required{node="k3s-prod-2"}'
|
||||
values: '0x456'
|
||||
- series: 'kured_reboot_required{node="k3s-prod-3"}'
|
||||
values: '0x456'
|
||||
- series: 'kube_node_info{node="k3s-prod-1"}'
|
||||
values: '1x456'
|
||||
- series: 'kube_node_info{node="k3s-prod-2"}'
|
||||
values: '1x456'
|
||||
- series: 'kube_node_info{node="k3s-prod-3"}'
|
||||
values: '1x456'
|
||||
promql_expr_test:
|
||||
# The expression selects ONLY the pending node — the operator needs to know which box.
|
||||
- expr: 'max by (node) (kured_reboot_required) == 1'
|
||||
eval_time: 1h
|
||||
exp_samples:
|
||||
- labels: '{node="k3s-prod-1"}'
|
||||
value: 1
|
||||
# A NORMAL WAIT IS NOT AN ALERT. At 24h the node is still inside its legitimate
|
||||
# once-per-day window wait, so the alert may be pending but must NOT be firing.
|
||||
- expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}'
|
||||
eval_time: 24h
|
||||
exp_samples: []
|
||||
# Still not firing just short of the boundary.
|
||||
- expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}'
|
||||
eval_time: 35h30m
|
||||
exp_samples: []
|
||||
# Firing past it, carrying the node label through to the notification.
|
||||
- expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}'
|
||||
eval_time: 36h30m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="KuredRebootPending", alertstate="firing", node="k3s-prod-1", severity="warning"}'
|
||||
value: 1
|
||||
# The two healthy nodes must NOT be alerted on. A rule written with `kured_reboot_required`
|
||||
# unaggregated, or compared with `>= 0`, would raise all three and bury the real one.
|
||||
- expr: 'count(ALERTS{alertname="KuredRebootPending", alertstate="firing"})'
|
||||
eval_time: 36h30m
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: 1
|
||||
# Liveness rules stay quiet: kured is present and covering all three nodes. A pending reboot
|
||||
# is NOT a wiring fault and must not be reported as one.
|
||||
- expr: 'ALERTS{alertname="KuredNotReporting"}'
|
||||
eval_time: 36h30m
|
||||
exp_samples: []
|
||||
- expr: 'ALERTS{alertname="KuredNodeCoverageGap"}'
|
||||
eval_time: 36h30m
|
||||
exp_samples: []
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
# 5. THE RULE MUST STOP. A reboot that SUCCEEDS clears the sentinel, and the alert must clear
|
||||
# with it — the other half of the failure space this harness exists for (a permanently-firing
|
||||
# rule is how VPNTunnelDown taught this repo the lesson). k3s-prod-1 is pending for ~40h,
|
||||
# long enough to fire, then the reboot happens and the gauge returns to 0.
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
- interval: 5m
|
||||
name: reboot completes after firing - KuredRebootPending resolves
|
||||
input_series:
|
||||
# 1 for 480 samples (40h), then 0 for the remainder.
|
||||
- series: 'kured_reboot_required{node="k3s-prod-1"}'
|
||||
values: '1x480 0x120'
|
||||
- series: 'kured_reboot_required{node="k3s-prod-2"}'
|
||||
values: '0x600'
|
||||
- series: 'kured_reboot_required{node="k3s-prod-3"}'
|
||||
values: '0x600'
|
||||
- series: 'kube_node_info{node="k3s-prod-1"}'
|
||||
values: '1x600'
|
||||
- series: 'kube_node_info{node="k3s-prod-2"}'
|
||||
values: '1x600'
|
||||
- series: 'kube_node_info{node="k3s-prod-3"}'
|
||||
values: '1x600'
|
||||
promql_expr_test:
|
||||
# Firing while the reboot is still outstanding.
|
||||
- expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}'
|
||||
eval_time: 39h
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="KuredRebootPending", alertstate="firing", node="k3s-prod-1", severity="warning"}'
|
||||
value: 1
|
||||
# Cleared once the sentinel is gone. If this assertion ever fails, the rule has become
|
||||
# self-sustaining and the operator will start ignoring it.
|
||||
- expr: 'ALERTS{alertname=~"Kured.*"}'
|
||||
eval_time: 45h
|
||||
exp_samples: []
|
||||
@@ -0,0 +1,124 @@
|
||||
---
|
||||
# Alert rules for the kured reboot automation.
|
||||
#
|
||||
# WHY THESE THREE, AND WHY THEY ARE NOT OPTIONAL. Reboot automation's characteristic failure is
|
||||
# SILENCE, not noise. Every way this component can break leaves a cluster that looks perfectly
|
||||
# healthy: a sentinel nothing writes, a Service the chart did not create, a NetworkPolicy that drops
|
||||
# the scrape, a taint that excludes a node, a drain wedged on a PodDisruptionBudget, a cluster lock
|
||||
# held forever by a node that hung on warm reboot. In none of those cases does anything turn red on
|
||||
# its own. kured itself stays 1/1 Running throughout.
|
||||
#
|
||||
# kured EXPOSES EXACTLY ONE METRIC. Read from cmd/kured/main.go:92-96 and :113 on 2026-10-17 — the
|
||||
# only `prometheus.MustRegister` in the tree is a GaugeVec `Subsystem: "kured"`, `Name:
|
||||
# "reboot_required"`, labelled `{node}`. So the full vocabulary available is
|
||||
# `kured_reboot_required{node="..."}`, 1 when the sentinel exists and 0 when it does not. Each rule
|
||||
# below is built from that one series plus `kube_node_info` (kube-state-metrics), and nothing else.
|
||||
# There is deliberately no rule over an invented metric name: this repo has shipped a
|
||||
# structurally-inert alert before (VeleroNoRecentBackup over an absent series, which is why
|
||||
# platform/monitoring/tests/run.sh exists), and an alert that cannot fire is worse than no alert
|
||||
# because it also removes the suspicion that would have caught it.
|
||||
#
|
||||
# THESE RULES LIVE HERE, NOT IN platform/monitoring/alerts.yaml, for two reasons. (1) The unit-test
|
||||
# harness treats a `<manifest>.promtool-test.yaml` COMPANION as covering just its sibling, whereas a
|
||||
# `tests/*.test.yaml` SUITE covers every PrometheusRule in its directory — so shipping them beside
|
||||
# the component keeps the test obligation local and legible (the platform/cnpg precedent). (2)
|
||||
# platform/monitoring/alerts.yaml is concurrently edited by other open branches; adding 3 rules to
|
||||
# its tail would create a merge conflict for no benefit.
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: kured
|
||||
namespace: kured
|
||||
labels:
|
||||
app.kubernetes.io/name: kured
|
||||
spec:
|
||||
groups:
|
||||
- name: platform-health.node-reboots
|
||||
rules:
|
||||
# ── 1. IS THE MECHANISM ALIVE AT ALL? ────────────────────────────────────────────────
|
||||
- alert: KuredNotReporting
|
||||
# `absent()` yields a series only when the ENTIRE vector is empty, so this fires when no
|
||||
# node at all is publishing — kured gone, or its Service/ServiceMonitor/NetworkPolicy
|
||||
# regressed so nothing is scraped. It stays quiet while a single node is mid-reboot
|
||||
# (the other two still report), which is what makes it specific to a wiring failure
|
||||
# rather than to a reboot in progress.
|
||||
#
|
||||
# THIS IS THE RULE THAT GUARDS THE OTHER TWO. Both of them read `kured_reboot_required`,
|
||||
# so if that series stops existing they go quiet and look like good news. This one turns
|
||||
# the absence itself into the signal. The specific regression it is aimed at: the kured
|
||||
# chart defaults `service.create` and `metrics.create` to FALSE, and Longhorn has already
|
||||
# demonstrated on this cluster (see the companion note in
|
||||
# conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh) that a
|
||||
# NetworkPolicy can let targets be DISCOVERED while every scrape times out and the TSDB
|
||||
# stays empty.
|
||||
expr: absent(kured_reboot_required)
|
||||
# 30m absorbs a HelmRelease upgrade rolling the DaemonSet and a Prometheus restart,
|
||||
# without tolerating a real wiring break for more than one quiet night's worth of time.
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: kured is publishing no reboot_required series on any node
|
||||
description: >-
|
||||
No node is reporting kured_reboot_required, so the reboot automation is either not
|
||||
running or not being scraped. While this is true, OS reboots are not happening and
|
||||
the two alerts that would say so (KuredRebootPending, KuredNodeCoverageGap) are
|
||||
themselves inert. Nodes will silently accumulate pending kernel upgrades.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-reboot-automation"
|
||||
|
||||
# ── 2. IS IT ALIVE ON *EVERY* NODE? ──────────────────────────────────────────────────
|
||||
- alert: KuredNodeCoverageGap
|
||||
# A per-node agent that silently covers only some nodes is the subtler half of the
|
||||
# liveness question, and `absent()` cannot see it. kured publishes one series per node it
|
||||
# runs on; kube-state-metrics publishes one `kube_node_info` per node that exists
|
||||
# (measured 2026-10-17: 3, matching the 3 nodes). A shortfall means at least one node has
|
||||
# no reboot automation — most likely a new or re-provisioned node carrying a taint this
|
||||
# DaemonSet does not tolerate (it declares none, because no prod node is tainted today).
|
||||
# That node would then never reboot, with nothing else to indicate it.
|
||||
expr: count(kured_reboot_required) < count(kube_node_info)
|
||||
# 2h, chosen to sit comfortably ABOVE the duration of a healthy reboot. A node being
|
||||
# rebooted stops publishing for a few minutes and legitimately creates a shortfall; a
|
||||
# node that is still missing after two hours is either genuinely uncovered or has hung on
|
||||
# reboot, and both of those want a human.
|
||||
for: 2h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: kured is not running on every node
|
||||
description: >-
|
||||
Fewer nodes are reporting kured_reboot_required than exist in the cluster, so at
|
||||
least one node has no reboot automation and will never be rebooted by it. Most
|
||||
likely a node carries a taint this DaemonSet does not tolerate, or has hung during
|
||||
a reboot.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-reboot-automation"
|
||||
|
||||
# ── 3. IS A REBOOT STUCK? ────────────────────────────────────────────────────────────
|
||||
- alert: KuredRebootPending
|
||||
# A node has wanted a reboot for longer than a full maintenance window plus margin, so
|
||||
# the reboot was OFFERED and did not happen. The causes are all real and all invisible
|
||||
# otherwise: the Prometheus alert gate is blocking on a degraded Longhorn volume that is
|
||||
# not recovering; the drain hit `drainTimeout` against a 0-disruption PDB; another node
|
||||
# holds the cluster lock and never released it (`lockTtl: 0` means a lock held by a hung
|
||||
# node is held FOREVER — that is deliberate, see helmrelease.yaml, and this is the alert
|
||||
# that makes the resulting stall visible).
|
||||
expr: max by (node) (kured_reboot_required) == 1
|
||||
# 36h, and the window being DAILY is what makes this number meaningful — see the window
|
||||
# block in helmrelease.yaml. With a 03:00-06:00 window every day, a pending reboot waits
|
||||
# at most ~24h legitimately, so 36h means at least one full window elapsed without the
|
||||
# reboot completing. On a weekly window this threshold would have to exceed 7 days, which
|
||||
# is longer than this cluster's 7d Prometheus retention and is reset by any Prometheus
|
||||
# restart past the 1h for-outage-tolerance — i.e. unreliable in the one direction that
|
||||
# matters. Note this alert can still be LATE for that reason, but it cannot be spurious.
|
||||
for: 36h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: Node {{ $labels.node }} has needed a reboot for over 36h
|
||||
description: >-
|
||||
kured reports a pending reboot on {{ $labels.node }} that has survived at least one
|
||||
full 03:00-06:00 maintenance window. The reboot is being blocked, not merely
|
||||
waiting: likely a degraded Longhorn volume holding the Prometheus alert gate, a
|
||||
drain timing out against a PodDisruptionBudget, or a stale cluster lock held by a
|
||||
node that did not come back. The kernel or security update that needed the reboot is
|
||||
installed but NOT running.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-reboot-automation"
|
||||
@@ -0,0 +1,289 @@
|
||||
---
|
||||
# kured (Kubernetes Reboot Daemon) — coordinated, one-node-at-a-time reboots of the three prod
|
||||
# control-plane nodes.
|
||||
#
|
||||
# ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# READ THIS FIRST: kured DOES NOT DECIDE *WHETHER* TO REBOOT. IT ONLY DECIDES *WHEN* AND *HOW*.
|
||||
# ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# kured reboots a node if and only if the file `/var/run/reboot-required` exists on it. On these
|
||||
# nodes exactly one thing creates that file: a deliberate operator run of
|
||||
# `playbooks/node-baseline.yml --tags kernel` (which advances the backports kernel). The
|
||||
# unattended-upgrades role this PR also adds installs SECURITY updates only and BLACKLISTS
|
||||
# `linux-image-*`/`linux-headers-*`, so it can never produce a reboot-required sentinel on its own —
|
||||
# see infra/ansible/roles/unattended_upgrades/defaults/main.yml for that argument in full.
|
||||
#
|
||||
# That is the whole answer to this repo's two standing objections to reboot automation:
|
||||
#
|
||||
# * infra/ansible/roles/node_baseline/tasks/main.yml:61-67 — "IT DOES NOT REBOOT." k3s-prod-2 hung
|
||||
# on a warm reboot on 2026-09-03 (powered, off the LAN ~38 minutes, physical power button
|
||||
# required), so rebooting was made "a SERIALIZED, SUPERVISED operator step ... and a role cannot
|
||||
# enforce that." STILL TRUE, and nothing here changes it: node_baseline continues not to reboot.
|
||||
# * docs/plans/cluster-upgrade-plan.md:291-299 — system-upgrade-controller was rejected because it
|
||||
# "automates away the human gate between nodes", and the operator's hard requirement is an
|
||||
# explicit PASS/STOP verification BETWEEN every node.
|
||||
#
|
||||
# THE HUMAN GATE IS PRESERVED, AND IT IS THE SENTINEL. The operator decides whether a kernel moves
|
||||
# and on which hosts (`--limit k3s-prod-1`), which is what writes the sentinel on those hosts and
|
||||
# only those hosts. kured then performs the cordon/drain/reboot/uncordon that the operator would
|
||||
# otherwise have typed by hand — the error-prone mechanical half, including the ordering rule
|
||||
# node_baseline's own message has to shout ("NEVER node 2 first"). Run the kernel play one host at a
|
||||
# time and the operator keeps per-node PASS/STOP control exactly as before; run it on all three and
|
||||
# kured still does them strictly one at a time, just in an order it chooses. Both are safe for
|
||||
# quorum; only the first preserves the ordering preference. See runbooks/node-reboot-automation.md.
|
||||
#
|
||||
# cluster-upgrade-plan.md:329-330 and :545-548 name the condition under which its own rejection
|
||||
# reverses — "SUC becomes viable once the cluster is on fast disks ... where drain-driven rebuild is
|
||||
# cheap" — and prod is that cluster. This component takes that clause up, but NOT its drain
|
||||
# assumption: see the Longhorn section below, where drain-driven rebuild is measurably NOT free even
|
||||
# on SSD, and is gated on explicitly rather than waved through.
|
||||
#
|
||||
# WHAT IS STILL UNMITIGATED, STATED PLAINLY: kured cannot press a power button. If a node hangs on
|
||||
# warm reboot the way k3s-prod-2 did in September, kured will not recover it. The blast radius is
|
||||
# bounded by design rather than by luck — `concurrency: 1` plus `lockTtl: 0` means the hung node
|
||||
# keeps the cluster lock forever, so NO second node is ever rebooted behind it; etcd keeps 2/3 quorum
|
||||
# and the cluster stays up, degraded, until a human attends. The KuredRebootPending alert in
|
||||
# alerts.yaml is what tells them. That is strictly better than the previous automation story (none)
|
||||
# but it is NOT the same as "safe to ignore at 3am on a Sunday".
|
||||
apiVersion: helm.toolkit.fluxcd.io/v2
|
||||
kind: HelmRelease
|
||||
metadata:
|
||||
name: kured
|
||||
namespace: kured
|
||||
spec:
|
||||
interval: 30m
|
||||
chart:
|
||||
spec:
|
||||
chart: kured
|
||||
# Chart 6.1.0 -> appVersion 1.23.0 (published 2026-06-30; newest of the 80 entries in
|
||||
# https://kubereboot.github.io/charts/index.yaml, read 2026-10-17).
|
||||
#
|
||||
# THE `<6.2.0` CEILING IS A DELIBERATE CHOICE HERE, NOT THE REFLEX IT IS ELSEWHERE. This PR's
|
||||
# own audit found that 9 of 12 HelmReleases in this repo pin a minor-locking range, and that
|
||||
# this is a MAIN CAUSE of pin rot: Renovate can then only ever offer patches inside a line it
|
||||
# has been told not to leave (kube-prometheus-stack is stuck 4 chart majors back that way).
|
||||
# For a component whose job is to reboot control-plane nodes, that ceiling is worth paying on
|
||||
# purpose: a kured MINOR may change drain or lock semantics, and this is the one component
|
||||
# where "a human looked at the changelog" should be a precondition, not a nice-to-have.
|
||||
# Renovate will raise the minor as a non-automerged PR (see the kured packageRule in
|
||||
# renovate.json), so the ceiling is a review prompt, not a wall.
|
||||
version: ">=6.1.0 <6.2.0"
|
||||
sourceRef:
|
||||
kind: HelmRepository
|
||||
name: kured
|
||||
namespace: kured
|
||||
install:
|
||||
remediation:
|
||||
retries: 3
|
||||
upgrade:
|
||||
remediation:
|
||||
retries: 3
|
||||
values:
|
||||
# ── IDENTITY / PLACEMENT ───────────────────────────────────────────────────────────────────
|
||||
# No tolerations. Measured 2026-10-17: all three prod nodes carry `spec.taints` EMPTY (k3s does
|
||||
# not taint its servers), so a DaemonSet reaches every node untainted. If a future node IS
|
||||
# tainted, kured silently stops covering it and that node then never reboots — invisible,
|
||||
# because the other two keep reporting. KuredNodeCoverageGap in alerts.yaml is the detector: it
|
||||
# compares kured's per-node series count against the node count rather than merely asking
|
||||
# whether kured exists.
|
||||
priorityClassName: system-node-critical
|
||||
# The chart's defaults are kept: `hostPID: true` and `containerSecurityContext.privileged: true`
|
||||
# with `readOnlyRootFilesystem: true`. Both are load-bearing and neither is reducible:
|
||||
# `rebootMethod: command` runs the reboot by nsenter'ing /proc/1/ns/mnt, which needs the host PID
|
||||
# namespace to see PID 1 and the privilege to enter its mount namespace. The documented
|
||||
# lower-privilege alternative (privileged:false + CAP_KILL) applies ONLY to
|
||||
# `rebootMethod: signal`, which signals a host-side kured-reboot.service that does not exist on
|
||||
# these nodes. Choosing `signal` would mean shipping and maintaining that unit via Ansible for a
|
||||
# capability reduction on a DaemonSet that is already cluster-admin-adjacent by function; it is a
|
||||
# reasonable future hardening step and is explicitly NOT taken here.
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
# No CPU limit on purpose (the repo's node-pressure work treats CPU limits on tiny system
|
||||
# agents as a throttling hazard). Memory is capped: kured holds no state beyond one
|
||||
# annotation and a gauge, and 128Mi is ~4x its idle RSS upstream reports.
|
||||
memory: 128Mi
|
||||
|
||||
configuration:
|
||||
# ── THE QUORUM CONSTRAINT: TWO NODES MUST NEVER BE DOWN AT ONCE ──────────────────────────
|
||||
# All three prod nodes are control-plane + etcd (measured 2026-10-17: every node reports
|
||||
# `control-plane,etcd`). etcd quorum is 2 of 3, so one node down is degraded-but-up and two
|
||||
# nodes down is a LOST CLUSTER that does not self-heal. Everything in this block exists to
|
||||
# make the second case structurally unreachable rather than merely unlikely.
|
||||
#
|
||||
# concurrency: 1 — kured's cluster-wide lock is a single annotation
|
||||
# (`weave.works/kured-node-lock`) on kured's OWN DaemonSet object, and a node reboots only
|
||||
# while it holds that lock. One lock + concurrency 1 = one reboot, enforced by the API server's
|
||||
# optimistic concurrency rather than by timing. This is the chart default; it is written out
|
||||
# explicitly because it is the single most important value in this file.
|
||||
concurrency: 1
|
||||
# lockTtl: 0 (DISABLED) — THIS IS A SAFETY DECISION, NOT A DEFAULT WE FORGOT TO TUNE.
|
||||
# A TTL makes the lock EXPIRE, which exists for the case "the lock holder died and we want
|
||||
# reboots to continue". On a 3-node etcd cluster that is precisely the catastrophe: if node A
|
||||
# has hung on a warm reboot (which has happened here — see the header) and its lock expires,
|
||||
# node B becomes eligible and takes the cluster from 2/3 to 1/3. We would rather the reboot
|
||||
# cycle STOP FOREVER and page a human than have it step over a corpse. The cost is that a
|
||||
# genuinely stale lock needs manual clearing; runbooks/node-reboot-automation.md has the
|
||||
# procedure, and KuredRebootPending is the alert that surfaces it.
|
||||
lockTtl: 0
|
||||
# lockReleaseDelay: 30m — hold the lock for 30 minutes AFTER this node is back and uncordoned,
|
||||
# before any other node may take it. See the Longhorn section: this is the STRUCTURAL half of
|
||||
# the inter-node gate, deliberately not dependent on an alert's `for:` timer having latched.
|
||||
lockReleaseDelay: 30m
|
||||
|
||||
# ── THE MAINTENANCE WINDOW: 03:00-06:00 EVERY DAY, America/New_York ──────────────────────
|
||||
# The timezone and the 03:00-06:00 band match renovate.json's existing
|
||||
# `"schedule": ["after 3am and before 6am"]` / `"timezone": "America/New_York"`, so every
|
||||
# unattended change in this repo lands in the same quiet 3-hour window.
|
||||
#
|
||||
# DAILY, NOT WEEKLY, AND THE REASON IS THE ALERT — not impatience. A weekly (`- sun`) window
|
||||
# was the first draft and it breaks the only alert that can tell you this mechanism has
|
||||
# stalled. With a Sunday-only window, a kernel upgraded on a Monday legitimately waits six
|
||||
# days, so "a reboot has been pending too long" cannot be distinguished from "it is Tuesday"
|
||||
# until the threshold exceeds a full 7-day cycle. That forces `for: 8d`+, which is longer than
|
||||
# this cluster's Prometheus retention (MONITORING_PROMETHEUS_RETENTION is 7d) and is reset by
|
||||
# any Prometheus restart beyond the 1h default for-outage-tolerance — i.e. the alert would be
|
||||
# unreliable in exactly the direction that matters. A daily window caps the legitimate wait at
|
||||
# under 24h, which makes KuredRebootPending's `for: 36h` both robust and meaningful: 36 hours
|
||||
# with a sentinel still set means at least one full window passed and SOMETHING BLOCKED IT.
|
||||
#
|
||||
# Daily does NOT weaken the quorum guarantee — that is `concurrency: 1` plus the lock, not the
|
||||
# calendar. It does mean a 3-node kernel rollout can finish in one to three nights instead of
|
||||
# three weeks, with each node still gated on the previous one's Longhorn rebuilds (below).
|
||||
rebootDays:
|
||||
- su
|
||||
- mo
|
||||
- tu
|
||||
- we
|
||||
- th
|
||||
- fr
|
||||
- sa
|
||||
startTime: "03:00"
|
||||
endTime: "06:00"
|
||||
timeZone: America/New_York
|
||||
# How often to re-evaluate (sentinel + window + alerts + lock). 1h is the chart default; at a
|
||||
# 3h window that is three evaluation opportunities per node per night.
|
||||
period: 1h
|
||||
|
||||
# ── THE SENTINEL ─────────────────────────────────────────────────────────────────────────
|
||||
# Written ONLY by a deliberate operator kernel upgrade (see the header). `update-notifier-common`
|
||||
# — the package that owns /etc/kernel/postinst.d and actually creates this file on Debian — was
|
||||
# measured ABSENT on all three nodes on 2026-10-17, which is why the unattended_upgrades role
|
||||
# installs it. Without that package nothing ever writes this path, kured correctly never
|
||||
# reboots, every probe looks healthy, and the entire mechanism is a silent no-op. That failure
|
||||
# mode is the reason KuredNotReporting exists.
|
||||
#
|
||||
# THE PATH kured LOGS IS NOT THIS PATH, and that trips people debugging it. The chart
|
||||
# defaults `useRebootSentinelHostPath: true` (kept), which bind-mounts the DIRNAME of this
|
||||
# value onto /sentinel read-only and renders the flag as
|
||||
# `--reboot-sentinel=/sentinel/reboot-required` (daemonset.yaml:134). So the container watches
|
||||
# /sentinel/reboot-required and the host file is /var/run/reboot-required. Same file, two
|
||||
# names. Set this value as the HOST path, as below.
|
||||
rebootSentinel: /var/run/reboot-required
|
||||
|
||||
# ── THE LONGHORN GATE: WHY POD READINESS IS NOT A SUFFICIENT INTER-NODE SIGNAL ────────────
|
||||
# Measured on prod 2026-10-17, and this is the finding that shaped this file:
|
||||
# * 25 Longhorn volumes, ALL `numberOfReplicas: 3`, all healthy, with exactly 25 replicas on
|
||||
# each of the 3 nodes. So there is no single-replica volume to lose today.
|
||||
# * BUT `replica-soft-anti-affinity: false` (replicas may not share a node) and
|
||||
# `disable-scheduling-on-cordoned-node: true`. With 3 nodes and 3 replicas, cordoning ONE
|
||||
# node leaves nowhere to place a replacement — so every drain takes ALL 25 volumes to
|
||||
# degraded 2/3 for the whole reboot.
|
||||
# * Recovery is slow and serialized: `replica-replenishment-wait-interval: 600` (10 minutes
|
||||
# before a rebuild even starts) and `concurrent-replica-rebuild-per-node-limit: 1`.
|
||||
# Therefore: if the next node is rebooted before rebuilds finish, those volumes go to 1/3, and
|
||||
# the drain after that genuinely CAN hit Longhorn's `node-drain-policy:
|
||||
# block-if-contains-last-replica` and block forever. The inter-node gate must be VOLUME
|
||||
# ROBUSTNESS, not "are the pods Ready again".
|
||||
#
|
||||
# It is gated twice, on purpose, because each half covers the other's blind spot:
|
||||
# 1. lockReleaseDelay: 30m (above) — unconditional. Covers the 600s replenishment wait with
|
||||
# margin and does not depend on any alert having fired.
|
||||
# 2. The Prometheus alert gate below — semantic. Covers a rebuild that takes LONGER than 30
|
||||
# minutes (25 serialized rebuilds on one node can), which a fixed delay cannot.
|
||||
prometheusUrl: http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090
|
||||
# Only consider alerts that are FIRING, not merely pending. A `pending` alert is a 10-minute
|
||||
# guess; blocking a weekly reboot on one would make the gate fire on noise.
|
||||
alertFiringOnly: true
|
||||
# ── alertFilterMatchOnly: true IS LOAD-BEARING. DO NOT DROP IT. ──────────────────────────
|
||||
# kured's two alert modes are near-opposites and the flag name does not advertise it:
|
||||
# * false (the chart default): alertFilterRegexp is an IGNORE list — kured blocks on ANY
|
||||
# firing alert except those matching.
|
||||
# * true: alertFilterRegexp is an ALLOW list — kured blocks ONLY on alerts that match.
|
||||
# We need `true`, because this cluster always has something firing. Measured via the Prometheus
|
||||
# API on 2026-10-17, these five were firing with no reboot pending and no end in sight:
|
||||
# CNPGPrimariesCoLocated, KubeDeploymentReplicasMismatch, KubeDeploymentRolloutStuck,
|
||||
# KubePodNotReady, OAuth2ProxyGateUnobserved (the last three all trace to the pm-agent pod,
|
||||
# ImagePullBackOff on a placeholder digest for 12 days). Under the default `false`, kured would
|
||||
# have been blocked by those on day one and would never have rebooted anything — a silent
|
||||
# no-op indistinguishable, from outside, from "no reboots were needed".
|
||||
alertFilterMatchOnly: true
|
||||
# The ALLOW list: the conditions under which rebooting ANOTHER node risks data or quorum.
|
||||
# Anchored and explicit — a prefix match here would silently re-admit the chronic five above.
|
||||
# LonghornVolumeDegraded — the inter-node gate (see above). `longhorn_volume_robustness
|
||||
# {state="degraded"} == 1`, for: 10m, in platform/monitoring/alerts.yaml.
|
||||
# LonghornVolumeFaulted — data already at risk; never reboot into this.
|
||||
# KubeNodeNotReady — a node is already down. Rebooting a second one loses quorum. This
|
||||
# is the single most important entry for the etcd constraint.
|
||||
# NodeExporterDown — a node we cannot see the health of. Treated as "assume the worst".
|
||||
# DELIBERATELY NOT INCLUDED: NodeEtcdRequestsSlow and the NodeIOPressure* family. They are
|
||||
# exactly the conditions you would want to pause for, but prod-1 has a documented history of
|
||||
# etcd WAL I/O starvation, so including them risks a permanent wedge that looks like safety.
|
||||
# Revisit once those are quiet for a sustained period; KuredRebootPending makes a wedge visible
|
||||
# either way.
|
||||
alertFilterRegexp: ^(LonghornVolumeDegraded|LonghornVolumeFaulted|KubeNodeNotReady|NodeExporterDown)$
|
||||
|
||||
# ── THE DRAIN ────────────────────────────────────────────────────────────────────────────
|
||||
# NOTE ON FLAGS THAT DO NOT EXIST: there is no `--ignore-daemonsets`, no
|
||||
# `--delete-emptydir-data` and no `--force` in kured 1.23.0. Read from
|
||||
# cmd/kured/main.go:438-449 on 2026-10-17, kured HARDCODES
|
||||
# `IgnoreAllDaemonSets: true`, `DeleteEmptyDirData: true` and `Force: true` in the drain
|
||||
# helper. Passing them as extraArgs would CrashLoop the DaemonSet on an unknown flag. This
|
||||
# matters because all three are genuinely REQUIRED here and it is easy to assume they must be
|
||||
# configured: 12 DaemonSets and 135 emptyDir-carrying pods were measured on 2026-10-17, and a
|
||||
# drain without those three settings refuses to start.
|
||||
#
|
||||
# drainTimeout: 15m — the chart/kured default is 0, meaning INFINITE, and infinite is the wrong
|
||||
# answer here. Measured 2026-10-17, six PodDisruptionBudgets currently allow ZERO voluntary
|
||||
# disruptions, and a plain drain loops on them indefinitely:
|
||||
# * 3x longhorn-system/instance-manager-* — one per node, `minAvailable: 1` against a
|
||||
# selector pinned to a single node's single pod, so disruptionsAllowed is STRUCTURALLY 0
|
||||
# forever. This is the mechanism by which Longhorn's node-drain-policy blocks: the
|
||||
# longhorn-manager deletes the node's instance-manager PDB only once it judges the drain
|
||||
# safe. The drain is SUPPOSED to wait on this one.
|
||||
# * 3x CNPG `<cluster>-primary` (keycloak-db, nextcloud-db, semaphore-db) — `minAvailable: 1`
|
||||
# against a selector matching exactly one pod (the primary), so again structurally 0 until
|
||||
# the CNPG operator switches the primary away. All three primaries were on k3s-prod-1.
|
||||
# With a timeout, a drain that is genuinely stuck ABORTS, the lock is released, and kured
|
||||
# retries next period instead of holding the cluster lock forever on a wedged node. 15m is
|
||||
# chosen to be longer than a CNPG switchover and longer than the `unifi` VM's eviction (below)
|
||||
# usually needs, while still being well inside the 3h window.
|
||||
drainTimeout: 15m
|
||||
# Taint a node PreferNoSchedule while its reboot is pending, so the scheduler stops placing new
|
||||
# pods on a node that is about to go away. With concurrency 1 this is a small win (there is no
|
||||
# second rebooting node to shed pods onto) but it is free and it shortens the drain.
|
||||
preferNoScheduleTaint: weave.works/kured-node-reboot
|
||||
# Record reboot state on the Node objects (`weave.works/kured-reboot-in-progress`,
|
||||
# `weave.works/kured-most-recent-reboot-needed`). Off by default. Turned on because the only
|
||||
# other observability kured offers is ONE gauge, and during an incident `kubectl get node -o
|
||||
# yaml` is the thing an operator can reach without Prometheus.
|
||||
annotateNodes: true
|
||||
logFormat: json
|
||||
|
||||
# ── METRICS: BOTH OF THESE MUST BE true OR THE ALERTS IN alerts.yaml CANNOT FIRE ───────────
|
||||
# The chart ships `service.create: false` AND `metrics.create: false`. Leaving either at its
|
||||
# default means no Service, or a ServiceMonitor with nothing to select, so `kured_reboot_required`
|
||||
# is never scraped — and a PrometheusRule over a series that no target produces is INERT: quiet
|
||||
# forever, indistinguishable from healthy. This repo has shipped that exact bug twice
|
||||
# (VeleroNoRecentBackup over an absent series; the inert gotk_* rules in
|
||||
# runbooks/image-automation.md), which is why platform/monitoring/tests/run.sh exists at all.
|
||||
# KuredNotReporting (alerts.yaml) is the standing detector for this pair regressing.
|
||||
# Prometheus selects ServiceMonitors cluster-wide: measured 2026-10-17, `serviceMonitorSelector`
|
||||
# and `serviceMonitorNamespaceSelector` are both `{}`, so no label plumbing is needed here.
|
||||
service:
|
||||
create: true
|
||||
port: 8080
|
||||
metrics:
|
||||
create: true
|
||||
interval: 60s
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
# HelmRepository for the official kured chart (github.com/kubereboot/charts). Renovate maintains the
|
||||
# pinned chart version in helmrelease.yaml via its `flux` manager, whose managerFilePatterns already
|
||||
# cover `platform/**` (renovate.json). Same Flux source pattern as platform/headlamp.
|
||||
apiVersion: source.toolkit.fluxcd.io/v1
|
||||
kind: HelmRepository
|
||||
metadata:
|
||||
name: kured
|
||||
namespace: kured
|
||||
spec:
|
||||
interval: 1h
|
||||
url: https://kubereboot.github.io/charts
|
||||
@@ -0,0 +1,36 @@
|
||||
---
|
||||
# platform/kured — the Kubernetes Reboot Daemon.
|
||||
#
|
||||
# Rendered by its OWN Flux Kustomization (clusters/base/kured.yaml, dependsOn platform), NOT by a
|
||||
# line in platform/kustomization.yaml. The reasoning is the same one the Cilium split-out wrote down
|
||||
# and intel-gpu-plugin reused: inside the shared `wait: true` platform tier, this component's health
|
||||
# would be indistinguishable from eleven others, the only emergency valve would be suspending the
|
||||
# whole tier, and that tier's 5m timeout is wrong for a 3-node DaemonSet rollout. It matters more
|
||||
# than usual here, because `platform` gates platform-config which gates every apps wave — so a wedged
|
||||
# reboot daemon placed there could stall every application in the cluster. A component whose entire
|
||||
# purpose is to make maintenance safer must not be able to cause an outage.
|
||||
#
|
||||
# `namespace: kured` applies to every resource below, so none of them repeats it. Keep it: the
|
||||
# CiliumNetworkPolicy in networkpolicy.yaml is a CR whose namespace kustomize would otherwise leave
|
||||
# to chance.
|
||||
#
|
||||
# NO SECRET OF ANY KIND. kured authenticates to the API server with its own ServiceAccount token and
|
||||
# talks to nothing off-cluster — no registry credential, no webhook URL, no notification token. So
|
||||
# neither clusters/dev nor clusters/prod adds a SOPS decryption patch for this tier (same as
|
||||
# homepage/runbooks/intel-gpu-plugin). If Slack or shoutrrr notifications are ever wanted, the chart
|
||||
# takes them via `configuration.notifyUrl`, and THAT would need a SOPS-encrypted Secret plus a
|
||||
# decryption patch — do not inline a webhook URL here.
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
namespace: kured
|
||||
resources:
|
||||
- namespace.yaml
|
||||
- helmrepository.yaml
|
||||
- helmrelease.yaml # chart installs the DaemonSet / RBAC / SA / Service / ServiceMonitor
|
||||
# default-deny baseline + the Prometheus scrape ingress + the apiserver/Prometheus egress.
|
||||
# The scrape ingress is NOT optional decoration: without it the ServiceMonitor's targets are
|
||||
# discovered and every scrape times out, leaving the TSDB empty and all three rules in alerts.yaml
|
||||
# permanently quiet. See the header in that file for the measured Longhorn precedent.
|
||||
- networkpolicy.yaml
|
||||
# Three rules over kured's single metric, with a companion promtool unit test beside them.
|
||||
- alerts.yaml
|
||||
@@ -0,0 +1,34 @@
|
||||
---
|
||||
# kured — namespace for the Kubernetes Reboot Daemon DaemonSet.
|
||||
#
|
||||
# PLATFORM INFRA, NOT AN APPLICATION NAMESPACE, and here that distinction is the difference between
|
||||
# a working component and an admission rejection. kured runs `hostPID: true` with a `privileged: true`
|
||||
# container (it `nsenter`s /proc/1/ns/mnt to run `/bin/systemctl reboot` on the host). Both are
|
||||
# forbidden by platform/kyverno/policies/restrict-host-namespaces.yaml — but that policy, like every
|
||||
# other policy in this repo, is scoped by
|
||||
# `namespaceSelector: matchLabels: homelab.keithjackson.dev/app-namespace: "true"`
|
||||
# and its own header states the carve-out: "platform components that do [need these] live in platform
|
||||
# namespaces that do NOT carry the app-namespace label, so they are out of scope."
|
||||
# Verified by reading the policies on this branch, not assumed:
|
||||
# * restrict-host-namespaces.yaml rule `disallow-host-namespaces` (hostPID)
|
||||
# * restrict-host-namespaces.yaml rule `disallow-privileged-containers` (privileged)
|
||||
#
|
||||
# We set the label EXPLICITLY to "false" rather than leaving it unset, following
|
||||
# platform/intel-gpu-plugin/namespace.yaml and platform/vpn-egress: an unset label and a deliberate
|
||||
# "false" are identical to the admission engine but not to a reviewer, and this component's
|
||||
# correctness depends entirely on the answer.
|
||||
#
|
||||
# NO PodSecurity labels. There is no cluster-wide PSA baseline in this repo — `pod-security.*`
|
||||
# appears only where a namespace opts IN (conformance/incluster, platform/kyverno/admission,
|
||||
# platform/pm-agent), all at `enforce: restricted`. Adding them here would reject this DaemonSet for
|
||||
# the same reasons Kyverno would. Leaving them off is the convention for a privileged platform
|
||||
# namespace (platform/vpn-ingress/namespace.yaml sets none either).
|
||||
#
|
||||
# NO `gateway-access` label. kured serves no HTTP to users and attaches no HTTPRoute; its only
|
||||
# listener is the :8080 metrics endpoint, reached in-cluster by Prometheus. See networkpolicy.yaml.
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: kured
|
||||
labels:
|
||||
homelab.keithjackson.dev/app-namespace: "false"
|
||||
@@ -0,0 +1,125 @@
|
||||
---
|
||||
# Default-deny baseline (R-NET-01): deny all ingress + egress, allow only DNS. `kured` is a PLATFORM
|
||||
# namespace, so Kyverno's generator — scoped to `app-namespace=true` — does not produce this; we ship
|
||||
# it explicitly, as crafty/headlamp/homepage/runbooks all do.
|
||||
#
|
||||
# THESE POLICIES DO APPLY TO kured, UNLIKE platform/vpn-ingress'. That component's identical-looking
|
||||
# baseline is honest-but-inert because its DaemonSet runs `hostNetwork: true` and therefore has no
|
||||
# Cilium endpoint for any policy to select. kured's chart defaults to `hostNetwork: false` and we
|
||||
# keep that default (it needs the host PID namespace to reboot, not the host network stack), so it
|
||||
# has a normal pod identity and every rule here is enforced.
|
||||
#
|
||||
# Cilium enforces the UNION of these policies (allow if ANY policy allows), so the CiliumNetworkPolicy
|
||||
# below adds the kube-apiserver path on top of this deny-all floor.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: default-deny
|
||||
namespace: kured
|
||||
labels:
|
||||
homelab.keithjackson.dev/baseline: default-deny
|
||||
spec:
|
||||
podSelector: {}
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress: []
|
||||
egress:
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
---
|
||||
# ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# PROMETHEUS MUST REACH :8080, OR EVERY RULE IN alerts.yaml IS INERT.
|
||||
# ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# This is not a theoretical hazard in this repo, it is a measured one. From the Longhorn entry in
|
||||
# conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh: with the ServiceMonitor
|
||||
# applied and no NetworkPolicy allowance, all three targets were DISCOVERED and every scrape failed
|
||||
# with "context deadline exceeded" after 10.0s — "a drop, not a refusal" — and the TSDB stayed
|
||||
# empty. Target discovery succeeding is therefore NOT evidence the scrape works, and the alerts
|
||||
# would have been quiet rather than broken. KuredNotReporting is the standing detector for exactly
|
||||
# this regression, and it can only work if this policy is correct.
|
||||
#
|
||||
# THE WHOLE `monitoring` NAMESPACE, NOT THE PROMETHEUS POD LABELS — following the same choice
|
||||
# longhorn/networkpolicy-metrics.yaml made and for the same reason: the prometheus-operator owns
|
||||
# those pod labels and rewrites them on upgrade (measured 2026-10-17, the pod carries
|
||||
# `app.kubernetes.io/name: prometheus` plus five operator-managed labels including a
|
||||
# controller-revision-hash), so selecting on them makes the scrape silently fail on a chart bump.
|
||||
# The namespace holds only the monitoring stack.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: kured-allow-prometheus-scrape
|
||||
namespace: kured
|
||||
labels:
|
||||
app.kubernetes.io/name: kured
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
# Set by the chart's `kured.matchLabels` helper (_helpers.tpl:70-71) with release name `kured`.
|
||||
app.kubernetes.io/name: kured
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: monitoring
|
||||
ports:
|
||||
- protocol: TCP
|
||||
# The container port the chart names `metrics` (daemonset.yaml:211-213), from
|
||||
# `configuration.metricsPort`, which we leave at the chart default of 8080 and mirror in
|
||||
# the HelmRelease's `service.port`.
|
||||
port: 8080
|
||||
---
|
||||
# Egress allows. kured talks to exactly two things and nothing else; both are required and the
|
||||
# component is useless without either.
|
||||
#
|
||||
# 1. THE KUBE-APISERVER — the lock annotation on its own DaemonSet (`weave.works/kured-node-lock`),
|
||||
# cordon/uncordon (`nodes` patch), the node annotations, listing pods, and creating
|
||||
# `pods/eviction` objects during the drain. Reached via the `kube-apiserver` Cilium entity
|
||||
# rather than an IP: on this cluster the API is fronted by a kube-vip VIP that moves between
|
||||
# nodes on every reboot, so any CIDR-based rule here would break during the exact operation
|
||||
# this component exists to perform. Same construct as platform/headlamp/networkpolicy.yaml.
|
||||
#
|
||||
# 2. PROMETHEUS, on 9090 — the alert gate. The HelmRelease sets
|
||||
# `prometheusUrl: http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090`
|
||||
# and `alertFilterMatchOnly: true`, so before taking the cluster lock kured asks Prometheus
|
||||
# whether any of LonghornVolumeDegraded / LonghornVolumeFaulted / KubeNodeNotReady /
|
||||
# NodeExporterDown is firing. IF THIS EGRESS IS BLOCKED, kured cannot evaluate the gate.
|
||||
# It treats the query error as "do not proceed" and skips the reboot — which is the correct
|
||||
# fail-safe direction, but it fails SILENTLY and indefinitely: reboots simply stop happening
|
||||
# and nothing goes red except, eventually, KuredRebootPending. Do not remove this rule
|
||||
# thinking it is only for metrics; metrics ingress is the policy above, this is the gate.
|
||||
#
|
||||
# NOTHING ELSE. No egress to the internet (kured pulls no content at runtime), no DNS beyond the
|
||||
# baseline above, no Longhorn or etcd access — kured reads Longhorn's health THROUGH Prometheus
|
||||
# alerts, never from Longhorn's own API, which is why that allow is absent here.
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: kured-allow-egress
|
||||
namespace: kured
|
||||
labels:
|
||||
app.kubernetes.io/name: kured
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: kured
|
||||
egress:
|
||||
- toEntities:
|
||||
- kube-apiserver
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: monitoring
|
||||
app.kubernetes.io/name: prometheus
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: "9090"
|
||||
protocol: TCP
|
||||
+31
-5
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
|
||||
"_comment": "Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). Renovate owns HELM CHART + DEPENDENCY bumps (opens PRs; allowed classes auto-merge on green CI). Flux image-automation owns per-service container IMAGE TAGS (the $imagepolicy setters). The two do NOT overlap: this config DISABLES image-tag updates so Renovate never fights Flux over a running version. Self-hosted against the bootstrap Gitea; platform: gitea set at runtime by the operator's renovate runner.",
|
||||
"_comment": "Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). AS OF 2026-10-17 RENOVATE OWNS BOTH HALVES: Helm chart/dependency bumps AND per-service container image tags. It opens PRs for everything; allowed low-risk classes auto-merge on green CI. The ADR-0009 split that gave image tags to Flux image-automation is formally closed — see .gitea/workflows/renovate.yaml for the three measured reasons (it pushes commits rather than opening PRs, so it bypasses the CLAUDE.md §1 gate; `main` is protected with user_can_push:false and the flux-system GitRepository has no secretRef, so it cannot write anyway; and enabling it would need an in-cluster git WRITE credential for the repo that defines the cluster). image-automation has been commented out at platform/kustomization.yaml:52 since it was authored and the cluster holds zero ImagePolicy/ImageRepository/ImageUpdateAutomation objects, so nothing is being torn down. CONSEQUENCE: the `docker` and `kubernetes` managers, previously disabled ONLY to avoid fighting that controller, are now ENABLED — without them every plain `image: repo/name:tag` line in this repo was maintained by nothing at all, which is how the oauth2-proxy auth sidecars reached 7 minors behind the oauth2-proxy running in the same cluster. RUNNER: .gitea/workflows/renovate.yaml (scheduled daily); platform/endpoint are set there, not here.",
|
||||
"extends": [
|
||||
"config:recommended",
|
||||
":dependencyDashboard",
|
||||
@@ -15,13 +15,24 @@
|
||||
"labels": ["dependencies", "automated-update"],
|
||||
"ignoreUnstable": true,
|
||||
"respectLatest": true,
|
||||
"enabledManagers": ["flux", "kustomize", "helmv3", "regex", "github-actions"],
|
||||
"enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "regex", "github-actions"],
|
||||
"flux": {
|
||||
"managerFilePatterns": ["/(^|/)platform/.+\\.ya?ml$/", "/(^|/)apps/.+\\.ya?ml$/"]
|
||||
},
|
||||
"kustomize": {
|
||||
"managerFilePatterns": ["/(^|/)kustomization\\.ya?ml$/"]
|
||||
},
|
||||
"_kubernetes_comment": "ADDED 2026-10-17. The `kubernetes` manager reads plain `image:` fields out of Kubernetes manifests, and it is NOT auto-detected — it only looks at files matched here, which is why this block is as broad as it is. Scope covers every place this repo writes a container image: platform components, the rendered app waves, the ServiceDefinition catalog that generates them, the runbooks web server and the pm-agent job template. `catalog/` matters as much as `apps/`: apps/wave-*/*.yaml are GENERATED by catalog/render.py, so a bump applied only to the rendered output is reverted by the next render and would also fail ci/render-golden.sh. Renovate will raise BOTH, and both must land together.",
|
||||
"kubernetes": {
|
||||
"managerFilePatterns": [
|
||||
"/(^|/)platform/.+\\.ya?ml$/",
|
||||
"/(^|/)platform-config/.+\\.ya?ml$/",
|
||||
"/(^|/)apps/.+\\.ya?ml$/",
|
||||
"/(^|/)catalog/.+\\.ya?ml$/",
|
||||
"/(^|/)runbooks/serving/.+\\.ya?ml$/",
|
||||
"/(^|/)infra/agent/.+\\.ya?ml$/"
|
||||
]
|
||||
},
|
||||
"customManagers": [
|
||||
{
|
||||
"customType": "regex",
|
||||
@@ -60,14 +71,29 @@
|
||||
"matchFileNames": ["ci/versions.env", "infra/devcontainer/versions.env"],
|
||||
"automerge": false,
|
||||
"addLabels": ["ci-tooling"]
|
||||
},
|
||||
{
|
||||
"description": "ADDED 2026-10-17. NEVER AUTO-MERGE THE BLAST-RADIUS COMPONENTS, not even a patch. The default rule above auto-merges minor/patch/digest on a green gate, which is right for an app image: a bad Sonarr bump inconveniences one pod. It is wrong for these five, because the gate is a STATIC check — it renders manifests and lints them, it does not boot a node, drain a volume or form an etcd quorum (vars.CLUSTER_AVAILABLE is unset, so L2/L3 conformance does not run in CI at all). So for anything whose failure mode is the CLUSTER rather than a service, a green gate is not evidence and a human must read the changelog. kured is here because it reboots control-plane nodes; longhorn because a storage-engine bump migrates live volume data; cilium because it is the CNI and a bad bump is a total partition (and its version is DUAL-PINNED in infra/ansible/roles/cilium/defaults/main.yml, which Renovate does not see, so the two would silently desync); cloudnative-pg because it drives Postgres switchovers; kube-prometheus-stack because ci/versions.env asserts an exact promtool/Prometheus lockstep and a chart bump that moves Prometheus fails the gate until that file moves with it.",
|
||||
"matchPackageNames": [
|
||||
"kured",
|
||||
"longhorn",
|
||||
"cilium",
|
||||
"cloudnative-pg",
|
||||
"kube-prometheus-stack"
|
||||
],
|
||||
"automerge": false,
|
||||
"addLabels": ["needs-review", "blast-radius"]
|
||||
}
|
||||
],
|
||||
"_unmanageable_comment": "WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A FLOATING tag has no version to compare, so Renovate skips it silently — it does not pin it and does not warn. As of 2026-10-17 that is: plexinc/pms-docker:latest, lscr.io/linuxserver/jackett:latest, pihole/pihole (NO TAG AT ALL), quay.io/keycloak/keycloak (no tag), nextcloud:31-apache and docker:27-dind. Those must be pinned BY HAND to a concrete version before any automation can maintain them, and until then they drift on every pull with nothing recording what is running. Separately, the HelmRelease minor-locking ranges (9 of 12, e.g. `>=88.0.0 <89.0.0` on kube-prometheus-stack) cap Renovate at patches inside a line it has been told not to leave; raising those ceilings is deliberate per-component work and is NOT part of enabling the bot. Both lists are in decisions/2026-10-17-automated-upgrades.md.",
|
||||
"ignorePaths": [
|
||||
"**/*.example",
|
||||
"**/*.sops.yaml",
|
||||
"conformance/**",
|
||||
"platform/image-automation/per-service/**"
|
||||
"platform/image-automation/per-service/**",
|
||||
"platform/kyverno/tests/**"
|
||||
],
|
||||
"docker": { "enabled": false },
|
||||
"kubernetes": { "enabled": false }
|
||||
"_ignorePaths_comment": "platform/kyverno/tests/** ADDED 2026-10-17: those Kyverno policy fixtures pin `alpine:latest` and a `conformance-runner:latest`, which are test inputs whose whole point is to be a fixed shape, not software to keep current. `conformance/**` was already ignored for the same reason; this closes the matching hole that enabling the `kubernetes` manager would otherwise have opened.",
|
||||
"_docker_comment": "The `docker` manager (Dockerfiles, docker-compose) and the `kubernetes` manager above were BOTH previously `enabled: false`, solely so Renovate would not fight Flux image-automation over a running tag. That controller was never enabled and never will be here (see _comment), so the only thing those two flags were achieving was leaving every plain `image:` tag in the repo unmanaged. `docker` is left out of enabledManagers because this repo contains no Dockerfile that ships to the cluster; `kubernetes` is now ON and is the one that matters.",
|
||||
"docker": { "enabled": true }
|
||||
}
|
||||
@@ -85,6 +85,9 @@ configMapGenerator:
|
||||
- node-io=node-io.md
|
||||
- node-memory=node-memory.md
|
||||
- node-memory-swap=node-memory-swap.md
|
||||
# Covers all three platform-health.node-reboots rules (KuredRebootPending,
|
||||
# KuredNodeCoverageGap, KuredNotReporting) from platform/kured/alerts.yaml.
|
||||
- node-reboot-automation=node-reboot-automation.md
|
||||
# Linked from OAuth2ProxyGateUnobserved, and the diagnostic procedure for any gated app
|
||||
# that "is up but does not work" — the gate answers an unauthenticated XHR with a 302,
|
||||
# not a 401, so the failure arrives as an HTTP 200 carrying the wrong body.
|
||||
|
||||
@@ -0,0 +1,254 @@
|
||||
# Node reboot automation (kured)
|
||||
|
||||
**Alerts:** `KuredRebootPending`, `KuredNodeCoverageGap`, `KuredNotReporting`
|
||||
**File:** `platform/kured/alerts.yaml`
|
||||
**Expressions:** `max by (node) (kured_reboot_required) == 1` / `count(kured_reboot_required) < count(kube_node_info)` / `absent(kured_reboot_required)`
|
||||
|
||||
## What this means here
|
||||
|
||||
These three alerts cover the OS reboot half of the automated upgrade process. The design is two
|
||||
pieces that only work together:
|
||||
|
||||
- **`roles/unattended_upgrades`** (Ansible, on the host) installs Debian **security** updates
|
||||
nightly. It never upgrades the kernel, never restarts k3s, and never reboots. When something it
|
||||
installed needs a reboot, it leaves `/var/run/reboot-required` behind.
|
||||
- **kured** (`platform/kured`, a DaemonSet) watches that file. When it exists, kured takes a
|
||||
**cluster-wide lock**, cordons the node, drains it, reboots it, and uncordons it — then holds the
|
||||
lock for a further 30 minutes before any other node may go.
|
||||
|
||||
**kured does not decide whether to reboot.** On these nodes the sentinel is written almost
|
||||
exclusively by a deliberate operator kernel upgrade
|
||||
(`playbooks/node-baseline.yml --tags kernel`), because the kernel is blacklisted from unattended
|
||||
upgrades. So a pending reboot normally means *a human already asked for this*.
|
||||
|
||||
Which alert you have tells you which half is wrong:
|
||||
|
||||
| alert | meaning |
|
||||
|---|---|
|
||||
| `KuredRebootPending` | The mechanism is alive and a reboot was offered, but it did not happen for >36h. **Something is blocking it.** |
|
||||
| `KuredNodeCoverageGap` | kured is not running on every node. The uncovered node will never be rebooted, silently. |
|
||||
| `KuredNotReporting` | kured is publishing nothing at all. Reboots are not happening, **and the other two alerts are inert.** Treat this as the most serious of the three. |
|
||||
|
||||
**All three nodes are control-plane + etcd, so quorum is 2 of 3.** One node down is degraded but
|
||||
up; two is a lost cluster. Everything below is written to protect that. `concurrency: 1` plus
|
||||
`lockTtl: 0` means a node that hangs on reboot keeps the lock **forever** and no second node is ever
|
||||
rebooted behind it. That is deliberate: the cycle stops and pages a human rather than stepping over
|
||||
a corpse.
|
||||
|
||||
**A reboot is not seamless.** Most workloads here are single-replica, so each reboot cycle takes a
|
||||
brief outage of whatever runs on that node — including `coredns` (1 replica), which means a
|
||||
cluster-wide DNS gap while it reschedules. See "Expected outage per cycle" below.
|
||||
|
||||
## Running the commands in this runbook
|
||||
|
||||
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
||||
container. Every command below runs as:
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
||||
|
||||
Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
|
||||
picks a current-context per invocation, and that default has already changed once. Do not run two
|
||||
`dev.sh` invocations at once; they share one container and interfere.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `uncordon`, `rollout`) and `flux reconcile` are
|
||||
operator actions, not agent actions. Where this runbook gives a mutating command it is for an
|
||||
operator, and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
Start here, in order. The first question is always *which half is broken*.
|
||||
|
||||
# 1. Is kured running at all, and on how many nodes?
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds,pod -o wide'
|
||||
|
||||
# 2. Which node is asking for a reboot, and who holds the lock?
|
||||
# The lock is an annotation on kured's OWN DaemonSet. An empty/absent value means no holder.
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds kured \
|
||||
-o jsonpath="{.metadata.annotations}"; echo'
|
||||
|
||||
# 3. Is a node cordoned, and is a reboot in progress?
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes \
|
||||
-o custom-columns=NAME:.metadata.name,SCHEDULABLE:.spec.unschedulable,TAINTS:.spec.taints'
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes \
|
||||
-o jsonpath="{range .items[*]}{.metadata.name}{\" \"}{.metadata.annotations.weave\.works/kured-reboot-in-progress}{\"\n\"}{end}"'
|
||||
|
||||
# 4. What is kured itself saying? It logs its decision every period (1h).
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured logs ds/kured --tail=80'
|
||||
|
||||
# 5. Is the gate blocking it? These are the ONLY alerts kured blocks on.
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n longhorn-system \
|
||||
get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness' \
|
||||
| grep -v healthy
|
||||
|
||||
# 6. Does the node actually want a reboot? (read-only; SSH as k3s, no sudo needed)
|
||||
ssh -o BatchMode=yes 192.168.2.10 'ls -la /var/run/reboot-required; cat /var/run/reboot-required.pkgs 2>/dev/null'
|
||||
|
||||
## Likely causes, most likely first
|
||||
|
||||
### 1. The Longhorn gate is holding it (the common `KuredRebootPending` cause)
|
||||
|
||||
kured queries Prometheus before taking the lock and refuses to proceed while
|
||||
`LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `KubeNodeNotReady` or `NodeExporterDown` is
|
||||
firing. This is working as designed and it is the **most likely** cause.
|
||||
|
||||
Why it lingers: there are 3 nodes, every volume has 3 replicas, and
|
||||
`replica-soft-anti-affinity: false` means replicas may not share a node. So cordoning one node
|
||||
leaves nowhere to put a replacement and **every volume goes degraded 2/3** for the duration.
|
||||
Recovery is slow on purpose: `replica-replenishment-wait-interval` is 600s before a rebuild even
|
||||
starts, and `concurrent-replica-rebuild-per-node-limit` is 1, so ~25 volumes rebuild one at a time.
|
||||
|
||||
Check progress, then wait:
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n longhorn-system \
|
||||
get volumes.longhorn.io -o json' | python3 -c 'import json,sys,collections; \
|
||||
v=json.load(sys.stdin)["items"]; print(collections.Counter(x["status"]["robustness"] for x in v))'
|
||||
|
||||
If volumes are rebuilding, **do nothing**. If a volume is stuck `degraded` or `faulted` with no
|
||||
rebuild progressing, that is a storage problem to fix first — see `/runbooks/longhorn-degraded`.
|
||||
Rebooting another node while a volume is at 1/3 is how you lose data.
|
||||
|
||||
### 2. The drain timed out against a PodDisruptionBudget
|
||||
|
||||
Six PDBs currently allow **zero** voluntary disruptions, and two kinds matter:
|
||||
|
||||
- **`longhorn-system/instance-manager-*`** (one per node). `minAvailable: 1` against a selector
|
||||
pinned to a single node's single pod, so `disruptionsAllowed` is structurally **always 0**.
|
||||
Longhorn's `longhorn-manager` *deletes* the PDB when its `node-drain-policy`
|
||||
(`block-if-contains-last-replica`) judges the drain safe. The drain is **supposed** to wait here.
|
||||
- **CNPG `<cluster>-primary`** (`keycloak-db`, `nextcloud-db`, `semaphore-db`). Also structurally 0
|
||||
until the CNPG operator switches the primary to another instance. If all three primaries are on
|
||||
the node being drained, three switchovers must happen.
|
||||
|
||||
kured's `drainTimeout` is 15m: a genuinely stuck drain **aborts**, releases the lock, and is retried
|
||||
next period rather than wedging forever. Repeated timeouts show in kured's logs.
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pdb -A -o wide' | awk '$NF==0 || NR==1'
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
|
||||
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PRIMARY:.status.currentPrimary'
|
||||
|
||||
If CNPG will not switch over, an **operator** can move the primary deliberately before the next
|
||||
window rather than fighting the drain.
|
||||
|
||||
### 3. A node hung on reboot and still holds the lock
|
||||
|
||||
`k3s-prod-2` has done this before (2026-09-03: powered, off the LAN ~38 minutes, physical power
|
||||
button required). `lockTtl: 0` means the lock does **not** expire, so the cycle stops here —
|
||||
correctly, since the cluster is at 2/3 and must not go to 1/3.
|
||||
|
||||
Symptoms: one node `NotReady` or absent, `KuredNodeCoverageGap` firing alongside
|
||||
`KuredRebootPending`, and the lock annotation naming the missing node.
|
||||
|
||||
**Operator recovery:** attend to the node physically. Once it is back and `Ready`, kured clears its
|
||||
own lock on the next period. If the node is gone for good and the lock must be cleared by hand,
|
||||
that is an operator action:
|
||||
|
||||
# OPERATOR ONLY. Only when you have confirmed the named node is NOT mid-reboot.
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured annotate ds kured \
|
||||
weave.works/kured-node-lock- --overwrite'
|
||||
|
||||
If the node was pruned mid-reboot and is left **cordoned** with no daemon to uncordon it:
|
||||
|
||||
# OPERATOR ONLY.
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod uncordon <node>'
|
||||
|
||||
### 4. Nothing can write the sentinel (so nothing ever reboots)
|
||||
|
||||
If `KuredRebootPending` never fires and reboots never happen even after a kernel upgrade, suspect
|
||||
the sentinel writer. On Debian, `/var/run/reboot-required` is created by a hook shipped in
|
||||
**`update-notifier-common`** — and it was absent on all three nodes before this automation landed.
|
||||
Without it kured is permanently, invisibly idle.
|
||||
|
||||
ssh -o BatchMode=yes 192.168.2.10 'dpkg -l update-notifier-common needrestart unattended-upgrades | tail -4'
|
||||
ssh -o BatchMode=yes 192.168.2.10 'ls /etc/kernel/postinst.d/ /etc/apt/apt.conf.d/ | sort'
|
||||
|
||||
Fix by re-running the role, which asserts this explicitly:
|
||||
|
||||
cd infra/ansible && ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify
|
||||
|
||||
### 5. kured is not being scraped (`KuredNotReporting`)
|
||||
|
||||
The chart defaults `service.create` and `metrics.create` to **false**, and the `kured` namespace
|
||||
ships a default-deny NetworkPolicy. If either enablement regressed, or the Prometheus ingress allow
|
||||
was dropped, targets are *discovered* and every scrape times out — the TSDB stays empty and all
|
||||
three rules go quiet. Longhorn hit exactly this failure on this cluster.
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get svc,servicemonitor,networkpolicy,ciliumnetworkpolicy'
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n monitoring exec \
|
||||
sts/prometheus-kube-prometheus-stack-prometheus -c prometheus -- \
|
||||
wget -qO- "http://localhost:9090/api/v1/targets?state=active" ' | grep -o '"job":"kured[^,]*' | head
|
||||
|
||||
### 6. A node is uncovered by the DaemonSet (`KuredNodeCoverageGap`)
|
||||
|
||||
kured declares **no tolerations**, because no prod node is tainted. A newly provisioned or
|
||||
re-provisioned node carrying a taint gets no kured pod and will never be rebooted.
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes \
|
||||
-o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints'
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds kured \
|
||||
-o custom-columns=DESIRED:.status.desiredNumberScheduled,READY:.status.numberReady'
|
||||
|
||||
The fix is a PR adding the needed toleration to `platform/kured/helmrelease.yaml` — not a `kubectl
|
||||
patch`, which Flux reverts.
|
||||
|
||||
## Expected outage per reboot cycle
|
||||
|
||||
Stated plainly, because this process is **not** seamless. Every workload at 1 replica on the
|
||||
draining node is down until it reschedules and becomes Ready elsewhere. Measured 2026-10-17: **45
|
||||
Deployments and 3 StatefulSets run a single replica.** The ones worth knowing about:
|
||||
|
||||
- **`kube-system/coredns` — 1 replica.** A cluster-wide in-cluster DNS gap while it moves. This is
|
||||
the single most impactful item and it is **not** fixed by this PR; raising coredns to 2–3 replicas
|
||||
is recommended follow-up work.
|
||||
- **`monitoring/prometheus-…-0`, `monitoring/loki-0`, `monitoring/alertmanager-…-0`** — all
|
||||
single-replica StatefulSets on Longhorn RWO volumes, so they additionally wait for volume
|
||||
detach/reattach. **You lose monitoring and alerting for part of the cycle**, including these very
|
||||
alerts.
|
||||
- **All three CNPG Postgres primaries** blip during switchover, which means Keycloak, Nextcloud and
|
||||
Semaphore briefly lose their database.
|
||||
- **`gitea`, `nextcloud`, `plex`, `jackett`, `qbittorrent`, `radarr`, `sonarr`, `seerr`, `crafty`,
|
||||
`homepage`, `headlamp`, `semaphore`** — each down for its reschedule.
|
||||
- **`unifi` (a KubeVirt VM)** has `evictionStrategy: LiveMigrate` but
|
||||
`migrationMethod: BlockMigration`, so evicting it copies the whole root disk over the network.
|
||||
Expect the drain of its node to be slow.
|
||||
- **`kube-vip`** fails the control-plane VIP over on every node reboot.
|
||||
|
||||
Multi-replica and therefore fine: `cilium-operator`, `oauth2-proxy`, `kyverno-admission-controller`,
|
||||
the Longhorn CSI sidecars, `keycloak`, `pihole`, `runbooks`.
|
||||
|
||||
## When to escalate or stop
|
||||
|
||||
**STOP and do not intervene** if:
|
||||
|
||||
- Longhorn volumes are `degraded` **and rebuilding**. This is the gate doing its job. Wait.
|
||||
- A single node is mid-reboot and under 15 minutes have passed.
|
||||
|
||||
**STOP and get a human to the hardware** if:
|
||||
|
||||
- Any node is `NotReady` while another reboot is pending. **Never clear the kured lock in this
|
||||
state** — that is the one action that can take etcd from 2/3 to 1/3 and lose the cluster.
|
||||
- Two nodes are down. The cluster has no quorum; this runbook cannot help. Recover hardware first.
|
||||
|
||||
**Escalate to a deliberate maintenance decision** if:
|
||||
|
||||
- `KuredRebootPending` has been firing for over a week. The reboot is not going to happen on its
|
||||
own; find the blocker above, or take the reboot manually during a watched window.
|
||||
- A volume is `faulted`. Fix storage before any further reboot.
|
||||
|
||||
**Turn the automation off** if it is misbehaving and you need the cluster stable. The honest way is
|
||||
a PR, but for an emergency an operator may suspend the tier — this stops future reboots and leaves
|
||||
running nodes untouched:
|
||||
|
||||
# OPERATOR ONLY. Stops kured being reconciled; does NOT uncordon a node mid-cycle.
|
||||
infra/devcontainer/dev.sh run bash -c 'flux --context k3s-prod suspend kustomization kured'
|
||||
|
||||
Check afterwards whether a node was left cordoned (First commands, step 3) and uncordon it if so.
|
||||
Resume with `flux resume kustomization kured`. Nothing in the graph depends on `kured`, so
|
||||
suspending it stalls nothing else.
|
||||
|
||||
---
|
||||
|
||||
Source of truth: `runbooks/node-reboot-automation.md`. Index: `/runbooks/`
|
||||
|
||||
Records: `decisions/2026-10-17-automated-upgrades.md` (design + what was not verified),
|
||||
`platform/kured/helmrelease.yaml` (every tuned value and why),
|
||||
`infra/ansible/roles/unattended_upgrades/README.md` (the package half, and the kernel exclusion).
|
||||
Reference in new issue
Block a user