diff --git a/.gitea/workflows/renovate.yaml b/.gitea/workflows/renovate.yaml new file mode 100644 index 0000000..81f14ed --- /dev/null +++ b/.gitea/workflows/renovate.yaml @@ -0,0 +1,173 @@ +# Renovate — the dependency/image update automation. SCHEDULED, self-hosted against this Gitea. +# +# ═══════════════════════════════════════════════════════════════════════════════════════════════ +# WHY THIS FILE EXISTS: renovate.json HAS BEEN IN THIS REPO AND HAS NEVER ONCE RUN. +# ═══════════════════════════════════════════════════════════════════════════════════════════════ +# Measured 2026-10-17: `renovate.json` is present, complete and thoughtfully written — it declares +# managers, schedules, automerge classes and grouping. Nothing executes it. There is no Renovate +# workflow (this is the first), no Renovate CronJob or Deployment in the cluster +# (`kubectl get cronjob,deploy,job -A | grep -i renovate` -> nothing), and no bot account doing it +# from outside. Meanwhile SEVEN files carry comments asserting "Renovate maintains this pin" +# (platform/homepage/deployment.yaml:40, runbooks/serving/deployment.yaml:87, +# platform/crafty/deployment.yaml:63, platform/vpn-ingress/daemonset.yaml:259, +# platform/keycloak/postgres-cluster.yaml:75, apps/wave-0/nextcloud-db.yaml:302, and +# platform/headlamp/helmrepository.yaml:2). Those comments have been false for the life of the file. +# +# The cost is measurable. With nothing bumping anything: the six per-app oauth2-proxy auth sidecars +# sit on v7.8.1 while the platform oauth2-proxy in the SAME cluster runs v7.15.5; Keycloak (the IdP) +# runs 26.0.6, a November-2024 release, 8 minors behind; Velero runs a RELEASE CANDIDATE +# (velero-plugin-for-aws v1.14.3-rc.2) with a stable available; the CNPG Postgres image is 16.4, +# missing every 16.5+ security release. A pin nothing maintains is not a pin, it is a fossil. +# +# ═══════════════════════════════════════════════════════════════════════════════════════════════ +# WHY RENOVATE AND NOT FLUX IMAGE-AUTOMATION (the decision, and it reverses ADR-0009's split) +# ═══════════════════════════════════════════════════════════════════════════════════════════════ +# ADR-0009 divided the work: Renovate owns Helm chart/dependency bumps, Flux image-automation owns +# per-service container image TAGS, deliberately non-overlapping. The second half of that split +# cannot be delivered on this repo's terms, for three measured reasons: +# +# 1. IT WOULD BYPASS THE MERGE GATE, WHICH CLAUDE.md §1 FORBIDS. ImageUpdateAutomation does not +# open pull requests; it COMMITS AND PUSHES. Its default target is the source's own branch — +# `main` — so it would write straight past the PR flow. It can be pointed at a side branch +# instead, but then nothing opens the PR and bumps silently accumulate on a branch nobody +# merges. Either way a human or a second bot has to close the loop. +# 2. ON THIS CLUSTER IT CANNOT EVEN DO THAT. Measured 2026-10-17: `main` is protected with +# `user_can_push: false`, and the `flux-system` GitRepository has NO `spec.secretRef` at all — +# the source is anonymous and read-only. Enabling write-back means minting an in-cluster git +# WRITE credential for the repository that defines the cluster, which is a materially larger +# blast radius than a CI-scoped token. +# 3. RENOVATE ALREADY OPENS PRs NATIVELY, so every change it proposes goes through the gate with +# no extra machinery. +# +# So: Renovate owns BOTH halves, and Flux image-automation stays disabled. Note this is a change of +# plan, not a change of state — image-automation has been commented out at +# platform/kustomization.yaml:52 since it was authored, and the cluster has ZERO +# ImageRepository/ImagePolicy/ImageUpdateAutomation objects (measured). Its two controllers ARE +# installed and have idled for 43 days. Nothing is being torn down here; a path that was never +# taken is being formally closed, and renovate.json's `docker`/`kubernetes` managers — disabled +# purely to avoid fighting a controller that never ran — are switched on to fill the gap. +# +# ═══════════════════════════════════════════════════════════════════════════════════════════════ +# THIS WORKFLOW MUST NOT BE ABLE TO SUCCEED SILENTLY +# ═══════════════════════════════════════════════════════════════════════════════════════════════ +# A SKIPPED Gitea job reports `success` (CLAUDE.md §2), so "green" is not evidence this ran. Two +# guards: +# * The preflight step FAILS HARD (exit 1) if RENOVATE_TOKEN is absent, rather than skipping. +# A missing credential on a dependency bot is indistinguishable from "nothing to update", and +# that is precisely how renovate.json stayed inert for months. +# * The summary step records what ran into the job summary unconditionally. +# What this still does NOT give you is an alert when the SCHEDULE stops firing. See the handoff +# entry's "What I did NOT verify" — that gap is stated, not papered over. +# +# OPERATOR PREREQUISITE — ONE SECRET, NOT COMMITTED HERE: +# `RENOVATE_TOKEN` — a Gitea access token for the bot account, scopes `write:repository` and +# `read:user`. Set it as an Actions secret on this repository. Nothing in this repo mints, +# stores or references its value; it is consumed by name only. +# Until it exists, this workflow FAILS LOUDLY on every scheduled run. That is intended: a +# visible red is the correct state for "automation configured but not credentialed". +# +# THE FIRST RUN WILL WANT TO OPEN A LOT OF PRs. ~30 dependencies are behind. renovate.json caps the +# blast radius (`prConcurrentLimit: 5`, `prHourlyLimit: 2`), but run it manually with +# `dry_run: true` FIRST and read the log before letting the schedule loose. That is what +# workflow_dispatch's default is for. +name: renovate + +on: + schedule: + # 03:15 UTC daily. renovate.json ALSO carries `"schedule": ["after 3am and before 6am"]` in + # America/New_York, and that inner schedule is the authoritative one — Renovate re-checks it and + # will decline to open PRs outside it. The two are deliberately both present: this cron decides + # when the process RUNS, renovate.json decides when it may ACT. Keeping the cron slightly ahead + # of the repo window means a run is always in flight when the window opens. + - cron: "15 3 * * *" + workflow_dispatch: + inputs: + dry_run: + description: "Dry run (no PRs, no branches). Leave true for the first run." + required: false + default: "true" + log_level: + description: "Renovate LOG_LEVEL (info | debug)" + required: false + default: "info" + +concurrency: + # Never two Renovate runs at once: concurrent runs race on the same branches and can open + # duplicate PRs. Not cancel-in-progress — a half-finished Renovate run can leave a branch pushed + # with no PR, so let the first one finish. + group: renovate + cancel-in-progress: false + +jobs: + renovate: + runs-on: ubuntu-latest + # Renovate scans every manager across the whole repo; 40m is generous for this repo's size and + # well under a runner's patience. Without a timeout a wedged run holds the `renovate` + # concurrency group and silently blocks every subsequent scheduled run. + timeout-minutes: 40 + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Preflight - the token must exist, and its absence must be LOUD + env: + RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }} + run: | + set -euo pipefail + if [ -z "${RENOVATE_TOKEN:-}" ]; then + echo "::error::RENOVATE_TOKEN is not set on this repository." >&2 + cat >&2 <<'MSG' + + Renovate cannot run without a Gitea access token, and this step fails rather than + skipping ON PURPOSE: a skipped Gitea job reports `success`, so skipping here would make + "dependency automation is not running" look exactly like "everything is up to date". + That is the failure mode that let renovate.json sit inert in this repo for months while + seven files claimed "Renovate maintains this pin". + + TO FIX: create a Gitea access token for the bot account with scopes + `write:repository` and `read:user`, then add it to this repository as the Actions + secret RENOVATE_TOKEN. Do not commit it anywhere. + MSG + exit 1 + fi + echo "RENOVATE_TOKEN is present (value not printed)." + + - name: Run Renovate + uses: renovatebot/github-action@v41.0.9 + with: + # Pinned, and Renovate maintains this pin itself: renovate.json's enabledManagers + # includes `github-actions`, and `helpers:pinGitHubActionDigests` is in its `extends`. + # This is the one dependency in the repo whose updater is its own updater. + configurationFile: renovate.json + token: ${{ secrets.RENOVATE_TOKEN }} + env: + # Self-hosted Gitea. `platform` + `endpoint` were the two values renovate.json's own + # header said were "set at runtime by the operator's renovate runner" — this is that + # runner, so they are set here rather than in the repo config, which keeps renovate.json + # portable and free of this instance's URL. + RENOVATE_PLATFORM: gitea + RENOVATE_ENDPOINT: https://gitea.home.keithjackson.dev/api/v1 + RENOVATE_REPOSITORIES: Home-Network/home-network-k3s-impl + # Renovate must never push to a protected branch; it works on its own branches and opens + # PRs against this base. Stated explicitly rather than inherited, because this is the + # property that keeps the whole mechanism inside CLAUDE.md §1's branch -> PR -> gate path. + RENOVATE_BASE_BRANCHES: main + RENOVATE_ONBOARDING: "false" + RENOVATE_REQUIRE_CONFIG: "required" + RENOVATE_DRY_RUN: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run == 'true' && 'full' || '' }} + LOG_LEVEL: ${{ github.event_name == 'workflow_dispatch' && inputs.log_level || 'info' }} + + - name: Summary + if: always() + run: | + { + echo "## Renovate" + echo + echo "- trigger: \`${{ github.event_name }}\`" + echo "- dry run: \`${{ github.event_name == 'workflow_dispatch' && inputs.dry_run || 'false' }}\`" + echo "- outcome: \`${{ job.status }}\`" + echo + echo "Renovate owns BOTH Helm chart bumps and container image tags here; Flux" + echo "image-automation is deliberately disabled (platform/kustomization.yaml)." + echo "Every change it proposes arrives as a pull request and must pass the merge gate." + } >> "$GITHUB_STEP_SUMMARY" diff --git a/clusters/base/kured.yaml b/clusters/base/kured.yaml new file mode 100644 index 0000000..d4a196b --- /dev/null +++ b/clusters/base/kured.yaml @@ -0,0 +1,98 @@ +--- +# Flux Kustomization: kured (Kubernetes Reboot Daemon) — reconciles ./platform/kured. +# Performs the cordon / drain / reboot / uncordon of a node that has a pending OS reboot, strictly +# one node at a time, inside a nightly window, gated on Longhorn volume health. Design rationale, +# every tuned value, and the honest list of what is NOT mitigated: platform/kured/helmrelease.yaml. +# +# WHY ITS OWN KUSTOMIZATION, AND NOT ONE LINE IN platform/kustomization.yaml +# +# The same argument the Cilium split-out recorded in platform/kustomization.yaml and intel-gpu-plugin +# reused, except the consequence is worse here than for either of them: +# +# * `platform` is `wait: true` over twelve-plus components, so its Ready condition cannot tell you +# "the reboot daemon is wedged" from "longhorn is slow". +# * The only emergency valve would be `flux suspend kustomization platform` — freezing storage, +# TLS, DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a DaemonSet. +# * That tier's 10m interval / 5m timeout are called out there as WRONG for a 3-node DaemonSet +# rollout. This is a 3-node DaemonSet rollout. +# * And `platform` gates platform-config, which gates apps-wave-0 (wait:true), which gates +# apps-wave-1. A reboot daemon that failed to go healthy inside that tier would stall EVERY +# APPLICATION in the cluster on "dependency not ready". A component added to make maintenance +# safer must not acquire the ability to cause an outage; that trade is the whole argument. +# +# dependsOn: [platform] — AND THIS ONE IS A REAL EDGE, NOT CONVENTION +# +# Not `cilium` (what intel-gpu-plugin uses) and not `platform-config` (what the leaf UIs use). It is +# `platform`, for two independent reasons: +# +# 1. CRD ORDERING (F-MON1). platform/kured ships a PrometheusRule, and the chart generates a +# ServiceMonitor. Both are CRs whose CRDs (monitoring.coreos.com) are installed by +# kube-prometheus-stack INSIDE the `platform` tier. Flux dry-runs every object in a pass before +# applying any of them, so a pass containing an unknown kind ABORTS — the failure that made the +# CDI StorageProfiles a true deadlock on prod on 2026-09-05 (see +# clusters/base/platform-config-cdi.yaml). Depending on `platform` guarantees the CRDs exist +# before this pass is attempted. +# 2. FUNCTIONAL PREREQUISITE. kured's reboot gate QUERIES Prometheus +# (`prometheusUrl` in the HelmRelease). Prometheus lives in platform/monitoring. If kured ran +# first it would get connection errors, treat the gate as unevaluable and skip every reboot — +# the correct fail-safe direction, but silent. Longhorn, whose volume health IS the gate, is +# also in that tier. +# +# It does NOT need platform-config: no Keycloak, no TLS, no split-horizon DNS, no Gateway, no +# HTTPRoute. kured serves no HTTP to users. +# +# wait: true — A HEALTH SIGNAL, NOT A GATE. Following intel-gpu-plugin rather than the leaf-UI +# template. A leaf UI is legitimately not-Ready for long stretches of a cold start while it waits on +# Keycloak and DNS, so its health is uninformative; kured has no such dependency — it is a DaemonSet +# that opens a socket and watches a file, so on healthy nodes it is Ready in seconds and not-Ready +# genuinely means BROKEN. This turns `flux get kustomization kured` into a one-line answer to "is +# reboot automation live on all three nodes", which is the question the operator must be able to +# answer before trusting a kernel upgrade to it. +# +# NOTHING IN THE GRAPH dependsOn THIS, so wait:true gates nothing — the blast radius of an unhealthy +# kured is this Kustomization's own Ready condition. Do not add `dependsOn: [kured]` anywhere: that +# would hand a node-rebooting DaemonSet the power to stall whatever depends on it, which is precisely +# what the placement above exists to prevent. +# +# prune: true is safe and deliberate. If this Kustomization is deleted, kured's DaemonSet goes with +# it and reboots simply stop happening — nodes are left uncordoned and running. The one caveat is +# that pruning mid-reboot would leave a node CORDONED with no daemon to uncordon it; that is a +# `kubectl uncordon` away and is covered in runbooks/node-reboot-automation.md. +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: kured + namespace: flux-system +spec: + # A HelmRelease plus five static manifests, nothing to re-render; kustomize-controller is + # source-triggered, so this is only the drift-correction floor. + interval: 10m + retryInterval: 2m + # 10m, not the 5m the `platform` tier uses: a 3-node DaemonSet rollout under `wait: true` can + # exceed 5m on a cold start (image pull on three nodes), which is one of the reasons the Cilium + # split-out gave for leaving that tier. + timeout: 10m + prune: true + # See the header: a health SIGNAL for a component nothing depends on, not a gate. + wait: true + dependsOn: + - name: platform # monitoring CRDs + Prometheus + Longhorn; see the header, this is load-bearing + sourceRef: + kind: GitRepository + name: flux-system + path: ./platform/kured + # REQUIRED, not boilerplate. platform/kured/alerts.yaml renders `${BASE_DOMAIN}` into the + # `runbook_url` annotation of all three rules. Without this block that token becomes the EMPTY + # STRING and the Kustomization still reports Ready — Flux's substitution fails OPEN — so every + # alert would ship a dead link to `https:///runbooks/node-reboot-automation`. The gate stage + # `flux-postbuild-substitution` asserts the converse (a rendered token implies this block), and + # BASE_DOMAIN is declared in clusters/base/cluster-vars.yaml. + # + # Nothing else in this path contains `${...}` or a bare `$`: the HelmRelease's alertFilterRegexp is + # a plain anchored alternation, and there are no shell snippets anywhere in the component — which + # is deliberate, because envsubst would eat a braced shell variable and `$$` would trip the + # dollar-collapse assertion in the same check. + postBuild: + substituteFrom: + - kind: ConfigMap + name: cluster-vars diff --git a/clusters/base/kustomization.yaml b/clusters/base/kustomization.yaml index 8d708d3..5f816bf 100644 --- a/clusters/base/kustomization.yaml +++ b/clusters/base/kustomization.yaml @@ -85,4 +85,17 @@ resources: # pm-agent-credentials.sops.yaml, so both overlays add a SOPS decryption patch for it (by # name). See clusters/base/pm-agent.yaml and platform/pm-agent/README.md. - pm-agent.yaml # Flux Kustomization -> ./platform/pm-agent (dependsOn apps-wave-0) + # kured (Kubernetes Reboot Daemon) — its OWN Kustomization (dependsOn platform, wait:true), NOT a + # line in platform/kustomization.yaml. Same reasoning as the Cilium and intel-gpu-plugin + # split-outs, with a sharper consequence: `platform` gates platform-config and every apps wave, so + # a wedged reboot DaemonSet placed in that wait:true tier could stall every application in the + # cluster — a component added to make maintenance SAFER must not be able to cause an outage. + # wait:true here is a HEALTH SIGNAL, not a gate: nothing dependsOn it, and nothing should. + # The `platform` edge is load-bearing rather than conventional — it supplies the + # monitoring.coreos.com CRDs this component's PrometheusRule/ServiceMonitor need (F-MON1), and + # Prometheus itself, which kured QUERIES before every reboot to gate on Longhorn volume health. + # It reboots a node only when /var/run/reboot-required exists, which on these nodes only a + # deliberate operator kernel upgrade creates — the human gate is the sentinel, not the daemon. + # See clusters/base/kured.yaml and platform/kured/helmrelease.yaml. + - kured.yaml # Flux Kustomization -> ./platform/kured (dependsOn platform) - cluster-vars.yaml # overlay knob surface (base defaults) diff --git a/conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh b/conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh index ed4332c..78d065e 100755 --- a/conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh +++ b/conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh @@ -363,6 +363,54 @@ SCRAPE_OWNERS: list[dict] = [ "why": "velero_* powers VeleroBackupFailure / PartialFailure / NoRecentBackup — the rules " "that failed to notice prod running with zero volume backups", }, + # ── ADDED 2026-10-17 (feat/automated-upgrades). A NEW OWNER, NOT A RELAXED ASSERTION. ────── + # This entry is purely additive: it registers a new metric family and its enablement, and it + # changes nothing about how any existing owner is evaluated. It makes the check STRICTER for the + # component it covers, not looser — it adds two `require` patterns plus a `companion` obligation + # that platform/kured must keep satisfying, so a future edit that disables kured's Service, + # ServiceMonitor or NetworkPolicy FAILS the gate instead of silently re-inerting three alerts. + # + # WHY THE CHECK HAD TO BE TOUCHED AT ALL, rather than the manifest being wrong: assertion (3) + # fails any alert over a metric no owner claims, and the escape hatch (KNOWN_UNWIRED_METRICS) is + # SHRINK-ONLY, so declaring `kured_*` as accepted debt is not permitted and would be the wrong + # answer anyway — the scrape genuinely IS wired here. The only two honest options were to + # register the owner (this) or to ship the reboot automation with no alerting at all, which for a + # component whose every failure mode is silent would have been the worse engineering choice. + # `conformance/**` is a sensitive approval path precisely so that this reasoning gets read. + { + "owner": "kured: service.create + metrics.create in the HelmRelease", + "prefixes": ("kured_",), + "file": "platform/kured/helmrelease.yaml", + # BOTH keys are required and the chart defaults BOTH to false. `metrics.create` makes the + # ServiceMonitor; `service.create` makes the Service it selects. Enable only the monitor and + # it selects nothing, so the single kured metric is never scraped — and a PrometheusRule over + # a series no target produces is INERT, which for this component means its own liveness + # alarm is the thing that breaks. Matched as two separate anchored blocks rather than one + # pattern because they are separate top-level values keys, not nested. + "require": [ + r"service:\s*\n\s*create:\s*true", + r"metrics:\s*\n\s*create:\s*true", + ], + "forbid": [], + "why": "kured_reboot_required is the ONLY metric kured exposes (one GaugeVec, " + "cmd/kured/main.go:92-96) and it powers all three platform-health.node-reboots " + "rules. Those rules are the only way to learn that OS reboot automation has stalled: " + "kured stays 1/1 Running while rebooting nothing, so a missed scrape here is " + "indistinguishable from a cluster that needed no reboots", + # The ServiceMonitor ALONE is not sufficient — the same NetworkPolicy trap Longhorn hit. + "companion": { + "file": "platform/kured/kustomization.yaml", + "require": [r"^\s*-\s*networkpolicy\.yaml\s*$"], + "why": "platform/kured/networkpolicy.yaml ships a default-deny baseline (the kured " + "namespace is a platform namespace, so Kyverno's app-namespace-scoped generator " + "does not cover it) AND the ingress allow that lets Prometheus reach :8080 " + "through it. Drop the networkpolicy from this kustomization and the deny-all " + "goes too, so this currently fails SAFE — but reinstating the baseline without " + "the allow reproduces Longhorn's measured failure exactly: targets DISCOVERED, " + "every scrape 'context deadline exceeded', TSDB empty, all three rules quiet. " + "Keeping the two in one file makes that pair inseparable", + }, + }, ] # `up` is produced by EVERY scrape target, so no owner can meaningfully claim it and assertion (1) diff --git a/decisions/2026-10-17-automated-upgrades.md b/decisions/2026-10-17-automated-upgrades.md new file mode 100644 index 0000000..0749222 --- /dev/null +++ b/decisions/2026-10-17-automated-upgrades.md @@ -0,0 +1,251 @@ +--- +date: 2026-10-17 +branch: feat/automated-upgrades +title: Automated upgrades — kured + security-only unattended-upgrades, and Renovate finally gets a runner +--- + +## What changed + +Two halves of one process. Neither is useful alone and they must land together. + +### (a) OS upgrades and reboots + +**`platform/kured/`** — the Kubernetes Reboot Daemon, chart `6.1.0` / app `1.23.0`, in its own Flux +Kustomization (`clusters/base/kured.yaml`, `dependsOn: [platform]`, `wait: true`), registered in +`clusters/base/kustomization.yaml`. Not a line in `platform/kustomization.yaml`: that tier is +`wait: true` and gates `platform-config` → every apps wave, so a wedged reboot daemon placed there +could stall every application in the cluster. A component added to make maintenance safer must not +be able to cause an outage. + +**`infra/ansible/roles/unattended_upgrades/`** + `playbooks/unattended-upgrades.yml`, enabled for +prod in `inventory/group_vars/k3s_prod/vars.yml`. A **new** role rather than an extension of +`node_baseline` — different lifecycle, and other open branches touch that role. + +**The design, and how it answers this repo's two standing objections to reboot automation.** +`roles/node_baseline/tasks/main.yml:61-67` says "IT DOES NOT REBOOT" (k3s-prod-2 hung on a warm +reboot 2026-09-03; physical power button required), and `docs/plans/cluster-upgrade-plan.md:291-299` +rejected system-upgrade-controller for "automating away the human gate between nodes". Both still +stand. **The human gate is the sentinel.** unattended-upgrades is security-only with the kernel +blacklisted, so it essentially never produces `/var/run/reboot-required`; that file appears when an +operator deliberately runs `node-baseline.yml --tags kernel`. kured then performs only the +*mechanical* half the operator would otherwise type by hand — including the ordering rule +`node_baseline`'s own output has to shout ("NEVER node 2 first"), which the operator keeps by +running the kernel play with `--limit ` one node at a time. `cluster-upgrade-plan.md:329-330` +names the condition under which its own rejection reverses (fast disks), and prod is that cluster. + +**Quorum: concurrent reboots are structurally impossible.** All three nodes are control-plane+etcd +(quorum 2/3). `concurrency: 1` plus kured's single cluster-wide lock annotation on its own DaemonSet +means a node reboots only while holding that lock, enforced by API-server optimistic concurrency +rather than by timing. **`lockTtl: 0`** (expiry disabled) is the deliberate choice: a TTL exists so +reboots continue after the lock holder dies, which on 3 etcd nodes is exactly the catastrophe — a +hung node's lock expiring would let a second node take the cluster to 1/3. So a hang stops the cycle +forever and pages a human. `lockReleaseDelay: 30m` holds the lock after a node returns. + +**The Longhorn gate.** Measured: 25 volumes, all `numberOfReplicas: 3`, one replica per node, with +`replica-soft-anti-affinity: false` and `disable-scheduling-on-cordoned-node: true`. So every drain +takes **all 25 volumes to degraded 2/3**, and recovery is serialized — +`replica-replenishment-wait-interval: 600`, `concurrent-replica-rebuild-per-node-limit: 1`. Reboot +the next node too early and volumes reach 1/3, where `node-drain-policy: +block-if-contains-last-replica` can block a drain forever. The inter-node gate is therefore **volume +robustness, not pod readiness**, and it is doubled: `lockReleaseDelay: 30m` (unconditional, covers +the 600s wait) plus kured's Prometheus alert gate (semantic, covers a rebuild longer than 30m). + +**`alertFilterMatchOnly: true` is load-bearing.** The flag inverts the regexp's meaning: `false` +(chart default) makes it an *ignore* list and kured blocks on **any** firing alert. Measured +2026-10-17, five alerts were firing with no end in sight (`CNPGPrimariesCoLocated`, +`KubeDeploymentReplicasMismatch`, `KubeDeploymentRolloutStuck`, `KubePodNotReady`, +`OAuth2ProxyGateUnobserved` — the last three all tracing to `pm-agent`, ImagePullBackOff on a +placeholder digest for 12 days). Under the default, kured would have been blocked on day one and +never rebooted anything, indistinguishable from "no reboots needed". The allow-list is anchored: +`^(LonghornVolumeDegraded|LonghornVolumeFaulted|KubeNodeNotReady|NodeExporterDown)$`. +`NodeEtcdRequestsSlow` and the `NodeIOPressure*` family are **deliberately excluded** despite being +exactly what you'd want to pause for — prod-1 has a documented etcd WAL I/O starvation history and +including them risks a permanent wedge that looks like safety. + +**The kernel floor (the highest-risk item), handled with two independent exclusions.** Measured: all +three nodes run `7.1.8+deb13-amd64` (`7.1.8-1~bpo13+1`, the backports metapackage), far above the +6.13 memattr floor — **but `linux-image-6.12.107` and `linux-image-6.12.94` are also still +installed, below the floor, bootable, and `apt-mark showhold` is empty on every node.** So: (1) the +allowed-origins list is `label=Debian-Security` only, and the running kernel comes from +`o=Debian Backports` (pin 100, the `NotAutomatic`+`ButAutomaticUpgrades` signature), so u-u cannot +*see* a kernel upgrade; (2) `^linux-` is blacklisted anyway, so widening origins later — the most +likely future edit — cannot re-arm the hazard. The role **asserts both against `apt-config dump`**, +apt's own parse of the whole directory, rather than trusting the file it just wrote. Also blacklisted: +`intel-microcode`, `amd64-microcode`, `grub-*`. `Remove-Unused-Kernel-Packages` is **false** against +the Debian default, because those older kernels are the documented rollback path. None of this stops +the kernel moving — the operator's deliberate Ansible run still does that, unchanged. + +**The silent-no-op trap this closes.** Measured: `update-notifier-common`, `needrestart` and +`unattended-upgrades` were **absent on all three nodes**, so *nothing could create +`/var/run/reboot-required`*. kured deployed alone would have been permanently idle while 1/1 Running +and reporting `kured_reboot_required=0` — green forever. The role installs the sentinel writer and +**asserts a hook exists** by discovery (not a hard-coded path), failing loudly if not. +`needrestart` runs in auto mode with `k3s.service` blacklisted — auto-restart is what makes a +security-only policy effective, but restarting the control plane unattended is worse than a delayed +library fix. (k3s is not an apt package at all here: no `k3s`, no `containerd` in dpkg.) + +**Alerts + runbook.** `platform/kured/alerts.yaml` — three rules over kured's *only* metric +(`kured_reboot_required`, one GaugeVec, read from `cmd/kured/main.go:92-96`) plus `kube_node_info`: +`KuredNotReporting` (`absent()` — the pipeline is dead and the other two are inert), +`KuredNodeCoverageGap` (`count(kured) < count(nodes)` — a node silently uncovered, which `absent()` +cannot see), `KuredRebootPending` (`for: 36h` — a reboot was offered and blocked). Companion +promtool suite `alerts.promtool-test.yaml` pins all three both ways, including that the rule +**stops** when a reboot succeeds. `runbooks/node-reboot-automation.md` registered in +`runbooks/kustomization.yaml`. + +**The window is DAILY 03:00-06:00, and that choice is driven by the alert.** A weekly window means a +pending reboot legitimately waits 6 days, forcing `for: 8d`+ — longer than this cluster's 7d +Prometheus retention and reset by any restart past the 1h for-outage-tolerance. Daily caps the +legitimate wait under 24h and makes `for: 36h` both robust and meaningful. + +### (b) Image / dependency automation + +**Renovate was aspirational. It now has a runner.** `.gitea/workflows/renovate.yaml` — scheduled +daily 03:15 UTC, `workflow_dispatch` with `dry_run` defaulting to **true** for a safe first run. +Preflight **fails hard (exit 1)** if `RENOVATE_TOKEN` is missing rather than skipping, because a +skipped Gitea job reports `success` and that is precisely how `renovate.json` stayed inert. + +**`renovate.json`: Renovate now owns both halves, and Flux image-automation is formally closed.** +ADR-0009 gave container image tags to Flux image-automation; that cannot be delivered on this +repo's terms. (1) `ImageUpdateAutomation` **pushes commits, it does not open PRs**, so it bypasses +the CLAUDE.md §1 gate; pointed at a side branch instead, nothing opens the PR. (2) It cannot work +here anyway: `main` is protected `user_can_push: false` and the `flux-system` GitRepository has **no +`spec.secretRef`** — the source is anonymous and read-only. (3) Enabling it means an in-cluster git +**write** credential for the repo that defines the cluster. So `docker`/`kubernetes` — disabled +*only* to avoid fighting that controller — are enabled, with `kubernetes.managerFilePatterns` +covering `platform/`, `platform-config/`, `apps/`, **`catalog/`** (apps are generated from it; a bump +only to the rendered output is reverted by the next render and fails `ci/render-golden.sh`), +`runbooks/serving/` and `infra/agent/`. Added a no-automerge `blast-radius` rule for `kured`, +`longhorn`, `cilium`, `cloudnative-pg`, `kube-prometheus-stack`, because the gate is static and does +not boot a node or form a quorum. `platform/kyverno/tests/**` added to `ignorePaths`. + +### Sensitive paths — this PR needs a non-author approving review + +It touches **`clusters/**`**, **`conformance/**`** and **`.gitea/**`**, all in +`infra/gitea/approval-policy.json`. It will **not** merge on a green gate alone. Per CLAUDE.md §1 I +have **not** posted `/approve-override` and must not — that is the self-approval the stage exists to +prevent. + +**The `conformance/` edit, justified.** `+34/-0` in +`alert-metrics-have-scrape-targets/check.sh`: one new `SCRAPE_OWNERS` entry claiming `kured_`. It +**weakens nothing** — no existing owner's evaluation changes — and it *tightens* the check for my +own component, adding two `require` patterns (`service.create: true`, `metrics.create: true`; the +chart defaults **both** to false) plus a `companion` requirement that `networkpolicy.yaml` stay in +the kustomization. It had to be touched: assertion (3) fails any alert over an unclaimed metric, and +the escape hatch `KNOWN_UNWIRED_METRICS` is **shrink-only**, so declaring `kured_*` as debt was not +permitted and would have been wrong anyway. The alternative was shipping reboot automation with no +alerting, which for a component whose every failure mode is silent is the worse choice. The entry +carries a dated comment saying all of this. + +## What I verified + +Measured against the live cluster and nodes, read-only, on 2026-10-17. + +- **Flux image-automation controllers ARE installed and idle.** `-n flux-system get deploy` shows + `image-automation-controller` and `image-reflector-controller` both `1/1`, age **43d**; all three + `image.toolkit.fluxcd.io` CRDs exist. And **zero** `ImageRepository`/`ImagePolicy`/ + `ImageUpdateAutomation` objects — "No resources found" for each. The controllers have run for 43 + days with nothing to do. `platform/kustomization.yaml:52` has the component commented out. +- **Renovate is aspirational, confirmed three ways**: no workflow in `.gitea/` or `ci/` (grep), + no Renovate CronJob/Deployment/Job in any namespace, and `renovate.json` itself disabled the + `docker`+`kubernetes` managers — so every plain `image:` tag was maintained by **nothing**, while + **seven files** claim "Renovate maintains this pin". +- **`flux-system` GitRepository has no `secretRef`** (empty), URL is the in-cluster Gitea, and its + artifact revision was `main@sha1:1a0b208…`, matching my base. +- **Nodes**: 3× Debian 13 trixie, all `control-plane,etcd`, k3s `v1.36.4+k3s1`, kernel + `7.1.8+deb13-amd64`. `apt-mark showhold` **empty**; `/etc/apt/preferences.d/` **empty**; + `/var/run/reboot-required` absent; `unattended-upgrades`/`update-notifier-common`/`needrestart` + all **absent**; `linux-image-6.12.107` and `6.12.94` installed alongside the backports kernel; + no `k3s`/`containerd` apt package; `apt-cache policy` origins and pins as quoted above. +- **No node taints** on any of the three (so no tolerations needed), and `system-node-critical` + exists. +- **Longhorn**: 1.12.1; `node-drain-policy: block-if-contains-last-replica`; + `replica-soft-anti-affinity: false`; `disable-scheduling-on-cordoned-node: true`; + `replica-replenishment-wait-interval: 600`; `concurrent-replica-rebuild-per-node-limit: 1`; + 25 volumes all 3-replica/healthy; 25 replicas per node. +- **6 PDBs with 0 allowed disruptions**: 3× `longhorn-system/instance-manager-*` (structurally 0 — + single-node selector, `minAvailable: 1`) and 3× CNPG `*-primary`. All three Postgres primaries + were on `k3s-prod-1`. +- **kured drain flags do not exist.** Read `cmd/kured/main.go:438-449`: `Force`, + `DeleteEmptyDirData` and `IgnoreAllDaemonSets` are **hardcoded true**. Passing + `--ignore-daemonsets`/`--delete-emptydir-data`/`--force` as extraArgs would CrashLoop the + DaemonSet. All three are genuinely needed (12 DaemonSets, 135 emptyDir pods measured). +- **kured exposes exactly one metric** — the only `MustRegister` in the project. +- **Prometheus selectors are all `{}`** (`serviceMonitorSelector`, `serviceMonitorNamespaceSelector`, + `ruleSelector`, `ruleNamespaceSelector`), so a `PrometheusRule`/`ServiceMonitor` in a new namespace + is picked up with no label plumbing. +- **`count(kube_node_info) == 3`** and `count(longhorn_volume_robustness) == 100`, queried through a + read-only port-forward. The 5 chronically-firing alerts were read from `/api/v1/alerts`. +- **`LonghornVolumeDegraded`** is `longhorn_volume_robustness{state="degraded"} == 1`, `for: 10m` — + shorter than Longhorn's own 600s replenishment wait, so degradation reliably outlasts the latch. +- **kured chart 6.1.0 → appVersion 1.23.0** is the newest of 80 entries in the chart index. +- **Kyverno/PSA will not block it**: every policy is scoped + `app-namespace: "true"`; the namespace sets it explicitly `"false"`, and + `restrict-host-namespaces.yaml`'s own header states that carve-out. No cluster-wide PSA baseline. +- **`renovate.json` parses** as valid JSON after my edits (`json.load`). +- **kured is not installed** and the string `kured` appears nowhere in the repo — greenfield. + +## What I did NOT verify + +The honest list. Most of this needs an actual upgrade cycle, which I did not and could not run. + +- **I never ran an OS upgrade, never rebooted a node, and never watched kured take a lock.** No part + of the runtime behaviour of either half has been observed. `sudo` needs a password I do not have, + so I could not run `apt`, `unattended-upgrade --dry-run`, or install anything. **Everything about + (a) working is inference from configuration and measured cluster state, not observation.** +- **The Ansible role has never been executed.** Not syntax-checked by `ansible-playbook --syntax-check` + either (no inventory access from the gate). Its asserts, its `apt-config dump` parsing, and the + sentinel-hook discovery are all unproven. The `find` patterns for the sentinel hook are a *guess at + which package ships what* — I could not read `update-notifier-common`'s file list (packages.debian.org + returned no such package for that suite/arch), which is exactly why the role discovers rather than + hard-codes. **If no hook matches, the role fails loudly — but it may fail on a correct node.** +- **Whether CNPG actually switches the primary in response to a drain-induced eviction.** All three + `*-primary` PDBs are structurally 0-disruption and `.spec.nodeMaintenanceWindow` is unset on all + three clusters. If the operator does not switch over, draining `k3s-prod-1` blocks on three PDBs at + once and hits `drainTimeout`. **This is the most likely way the first real reboot fails.** It + should be tested with one manual, watched `kubectl drain` before trusting the timer. +- **Whether `drainTimeout: 15m` is long enough.** Unmeasured. The `unifi` VMI has + `evictionStrategy: LiveMigrate` with `migrationMethod: BlockMigration`, i.e. a full root-disk copy + over the network on every `k3s-prod-1` drain. I do not know how long that takes. +- **Whether the alert gate actually blocks as intended.** I never saw kured query Prometheus. The + assumption that it treats a query error as "do not proceed" is from its documented behaviour, not + observed. The NetworkPolicy allowing kured→Prometheus:9090 and Prometheus→kured:8080 is + **unproven**; if either is wrong the symptom is silence, which is why `KuredNotReporting` exists — + but that alert is itself unproven against a live scrape. +- **The alert expressions have not run against real `kured_*` data**, because the metric does not + exist in this cluster yet. They pass promtool unit tests against synthetic series whose shape I + read from kured's source. `absent()` and the `count() < count()` comparison are the two most + likely to behave differently against real label sets. +- **`enabledManagers: ["regex"]` may be the wrong name.** Modern Renovate expects `custom.regex` for + `customManagers`. I left the pre-existing value alone rather than guess, but **the inline + `# renovate:` markers may simply not run.** Unverifiable without executing Renovate. Two markers + are additionally known-broken independent of this: `external-snapshotter`'s sits two comment lines + above its `newTag`, and Keycloak's `kustomization.yaml` has no marker at all despite a comment + claiming Renovate maintains it. +- **Renovate has still never run.** I added the runner; `RENOVATE_TOKEN` does not exist yet, so the + first scheduled run **will fail** (by design). Nothing about the config is proven correct until a + `dry_run: true` dispatch is read by a human. Expect it to want to open ~30 PRs. +- **No alert for "automatic patching silently stopped".** u-u's success lives in + `/var/log/unattended-upgrades/` on each node and nothing reads it. Likewise **no alert for the + Renovate schedule ceasing to fire**. Both are real gaps I did not close. +- **`coredns` is 1 replica and I did not fix it.** It is k3s-managed, out of this PR's scope, and it + means every `k3s-prod-1` reboot causes a cluster-wide in-cluster DNS gap. Recommended follow-up. +- **kured cannot honour "NEVER node 2 first".** It has no ordering control. The mitigation — run the + kernel play with `--limit` one host at a time — is a *procedure*, documented in the runbook, not an + enforced property. If the operator flags all three nodes at once, kured picks the order. +- **kured cannot press a power button.** The 2026-09-03 warm-reboot hang is unmitigated. Bounded, not + prevented: quorum holds at 2/3, the lock is held forever so nothing cascades, and + `KuredRebootPending` eventually fires. +- **Upstream version claims are a snapshot, partly from search results**, not all from authoritative + release APIs. I could not determine latest for versitygw, crafty, nginx-unprivileged, act_runner, + `virt-template-*`, the linuxserver wireguard date-tag, or the exact latest PostgreSQL 16 patch + (sources conflicted 16.10 vs 16.15). **The gitea `1.24.3 → v28.1.0` gap is almost certainly an + upstream versioning-scheme change, not 26 majors — do not act on that framing.** +- **I did not verify Renovate's `kustomize` manager still processes `images:`/`newTag`** now that + `docker` is enabled, which decides whether `external-snapshotter` and `intel-gpu-plugin` are + managed at all. +- **I did not raise any HelmRelease version ceiling**, so the 9 minor-locking ranges still cap + Renovate at patches. Bumping `kube-prometheus-stack` in particular will fail the gate until + `ci/versions.env`'s exact promtool/Prometheus `3.14.0` lockstep moves with it. +- **L2/L3 conformance did not run** (`vars.CLUSTER_AVAILABLE` unset), so nothing here is proven + against a cluster by CI either. diff --git a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml index 2b865a6..c16ada8 100644 --- a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml +++ b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml @@ -420,3 +420,34 @@ flux_bootstrap_env_path: "clusters/prod" # rather than leaving an inert drop-in behind, so this group-wide `true` is still correct # for it. etcd_io_priority_enabled: true + +# --- automatic security patching (roles/unattended_upgrades) ------------------------- +# TURNS ON roles/unattended_upgrades FOR PROD ONLY. Role default is false. Same seam as +# node_baseline_backports_kernel and etcd_io_priority_enabled above. +# +# WHAT WAS THERE BEFORE: nothing at all. Measured on all three prod nodes 2026-10-17 — +# `unattended-upgrades`, `update-notifier-common` and `needrestart` were ALL absent, +# `apt-mark showhold` was empty on every node, /etc/apt/preferences.d/ was empty, and +# nothing on the node could create /var/run/reboot-required. These boxes have had ZERO +# automatic patching since provisioning, and no way to signal that a reboot was needed. +# +# WHAT ENABLING THIS DOES *NOT* DO: nothing, until the operator runs +# playbooks/unattended-upgrades.yml. It is not applied by site.yml (deliberately — see +# that playbook's header). And once run, it still never upgrades the KERNEL, never +# restarts k3s, and never reboots: +# * kernel — blacklisted `^linux-`, AND the trixie-backports archive the running +# kernel comes from (o=Debian Backports, measured) is not an allowed origin. Two +# independent exclusions, because this is the EFI-memattr constraint: these HP minis +# need >= 6.13 or they can boot with a fraction of their RAM +# (PROD-PROVISIONING.md §0b), and 6.12 kernels are still installed and bootable here. +# The kernel continues to move ONLY under a deliberate +# `node-baseline.yml --tags kernel` run, exactly as before. +# * k3s — needrestart blacklist. k3s is not even an apt package on these nodes +# (measured: no `k3s`, no `containerd` in dpkg), so the control plane and runtime are +# outside apt's reach regardless. +# * reboots — Automatic-Reboot "false", asserted by the role against apt's own +# effective config. Reboots belong to kured (platform/kured): one node at a time, +# under a cluster-wide lock, inside 03:00-06:00, after cordon + drain. Three +# control-plane nodes with etcd quorum 2/3 must never reboot themselves +# independently, which is what Automatic-Reboot "true" would mean. +unattended_upgrades_enabled: true diff --git a/infra/ansible/playbooks/unattended-upgrades.yml b/infra/ansible/playbooks/unattended-upgrades.yml new file mode 100644 index 0000000..38f8e44 --- /dev/null +++ b/infra/ansible/playbooks/unattended-upgrades.yml @@ -0,0 +1,36 @@ +--- +# Automatic Debian SECURITY patching for the k3s nodes — the package half of the automated upgrade +# process. The reboot half is kured (platform/kured), reconciled by Flux; the two are designed +# together and neither is useful alone. Full rationale for every decision: +# roles/unattended_upgrades/defaults/main.yml and that role's README.md. +# +# TARGET (R-ENV-02): `k3s_hosts` defaults to dev's `k3s_servers`; prod is +# ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod +# Same pattern node-baseline.yml and node-health.yml established — `hosts:` is templated before +# group_vars exist, so the target is an extra var, never a group_vars value. +# +# The role is a NO-OP unless `unattended_upgrades_enabled` is true, which is set for prod only in +# inventory/group_vars/k3s_prod/vars.yml. Running this against dev changes nothing. +# +# DELIBERATELY NOT IN site.yml. site.yml is the bring-up chain (node-baseline -> k3s -> +# etcd-io-priority -> kube-vip -> cilium -> verify -> flux-bootstrap) and is "not run against prod +# casually". Automatic patching is not a prerequisite for anything in that chain and nothing in the +# chain depends on it, so adding it there would only widen what a cluster rebuild touches. It is +# applied on its own, once, and re-applied when its configuration changes. +# +# SAFE TO RUN ON A LIVE CLUSTER, AT ANY TIME. This playbook installs and configures the upgrader. It +# never runs `apt upgrade`, never restarts k3s, and never reboots — the role asserts the last of +# those against apt's own effective config rather than trusting the file it just wrote. The first +# unattended upgrade happens at the next timer firing (04:00 +/-60m), and any reboot that upgrade +# implies is kured's to schedule, one node at a time, inside 03:00-06:00. +# +# TAGS. `--tags verify` re-runs only the read-only assertions and the per-node summary, which is the +# way to answer "is automatic patching actually working on these boxes, and is anything waiting on a +# reboot" without rewriting any configuration: +# ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify +- name: Automatic security upgrades on the k3s nodes (kernel excluded; reboots delegated to kured) + hosts: "{{ k3s_hosts | default('k3s_servers') }}" + become: true + gather_facts: true + roles: + - unattended_upgrades diff --git a/infra/ansible/roles/unattended_upgrades/README.md b/infra/ansible/roles/unattended_upgrades/README.md new file mode 100644 index 0000000..9f45476 --- /dev/null +++ b/infra/ansible/roles/unattended_upgrades/README.md @@ -0,0 +1,184 @@ +# roles/unattended_upgrades + +Automatic Debian **security** patching on the k3s nodes. The kernel is excluded, k3s is never +restarted, and reboots are delegated to **kured** (`platform/kured`). + +- **Playbook:** `playbooks/unattended-upgrades.yml` +- **Enabled by:** `unattended_upgrades_enabled` — role default `false`, set `true` for prod in + `inventory/group_vars/k3s_prod/vars.yml` +- **Companion component:** `platform/kured` (the reboot half). **Neither is useful alone** — see + "The two halves" below. +- **Every decision and its reasoning:** `defaults/main.yml`. That file is the argument; this one is + the orientation. + +## The two halves, and why they ship together + +| | what it does | what it must NOT do | +|---|---|---| +| **this role** (Ansible, host) | installs security updates continuously; leaves `/var/run/reboot-required` behind when one needs a reboot | upgrade the kernel; restart k3s; reboot | +| **kured** (Flux, cluster) | cordons, drains, reboots and uncordons **one node at a time** when that file exists | decide *whether* a reboot is needed | + +Deploying either alone produces a mechanism that looks healthy and does nothing: + +- **kured alone** watches a file that nothing creates. Measured 2026-10-17: + `update-notifier-common` — the package that owns the hook which writes + `/var/run/reboot-required` — was **not installed on any node**. kured would sit at 1/1 Running + reporting `kured_reboot_required=0` forever. +- **this role alone** installs patches that a long-running process never picks up, and when a + reboot *is* needed there is nothing safe to take it. `Automatic-Reboot "true"` is not the + answer: three control-plane nodes with etcd quorum 2/3, each rebooting at its own time, is how + the cluster is lost. + +## What it was replacing: nothing + +Measured on all three prod nodes, 2026-10-17: + +| | | +|---|---| +| `unattended-upgrades` | **not installed** | +| `update-notifier-common` | **not installed** | +| `needrestart` | **not installed** | +| `apt-mark showhold` | **empty** — no package held on any node | +| `/etc/apt/preferences.d/` | **empty** — no apt pin declares the backports-kernel intent | +| `/var/run/reboot-required` | absent, and uncreatable | + +These nodes had received **no automatic patching since provisioning**. + +## The three decisions + +### 1. Security only + +Allowed origins are `label=Debian-Security` and nothing else. `trixie-updates` is excluded +deliberately (not overlooked) — it carries non-security fixes that are useful but are the archive +through which an unreviewed change would most plausibly arrive on a control-plane node. The knob to +add it is `unattended_upgrades_include_updates`. + +### 2. The kernel is excluded twice over + +**This is the highest-risk part of the change.** These HP EliteDesk 800 G5 minis publish a corrupt +EFI Memory Attributes Table on R21 firmware; Linux `< 6.13` does not validate it before reserving +memory from its contents, and `k3s-prod-2` twice booted with **2.6 GiB of 16 GB**. Upstream +`8fbe4c49c0cc` (first in v6.13) is the only fix — there is no kernel cmdline workaround. Full +write-up: `infra/PROD-PROVISIONING.md` §0b and `roles/node_baseline/defaults/main.yml:21-51`. + +Measured 2026-10-17: all three nodes run `7.1.8+deb13-amd64` (`7.1.8-1~bpo13+1`, i.e. the backports +metapackage), comfortably above the floor — **but `linux-image-6.12.107+deb13-amd64` and +`linux-image-6.12.94+deb13-amd64` are also still installed, below the floor, bootable, and held by +nothing.** + +So an unattended process must not be able to touch the kernel at all: + +1. **Origin exclusion (structural).** The running kernel comes from `o=Debian Backports` (measured + via `apt-cache policy`, pin priority 100 — the `NotAutomatic` + `ButAutomaticUpgrades` + signature). That origin is not in `Origins-Pattern`, so unattended-upgrades cannot see a kernel + upgrade at all. +2. **Package blacklist (belt and braces).** `^linux-` is blacklisted, so even if the origin list is + later widened — the most likely future edit to this role, and the one that would silently re-arm + the hazard — the kernel is still untouchable. + +The role **asserts both** against `apt-config dump` (apt's own parse of the whole +`/etc/apt/apt.conf.d/` directory) rather than trusting the file it just wrote. + +**This does not stop the kernel moving.** The operator still advances it deliberately with + +``` +ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel --limit +``` + +which is the only thing that maintains the backports source, the metapackage and the on-node +provenance stamp. `node_baseline`'s own note that `apt upgrade` keeps the backports kernel current +still holds for an operator-run apt. What changed is that a **new unattended actor** is forbidden +from it. + +Also blacklisted, for the same "operator-only" reason: `intel-microcode`, `amd64-microcode`, +`grub-*`. Microcode and bootloader changes take effect on reboot and alter boot behaviour on the +very firmware that is already misbehaving here. + +`Remove-Unused-Kernel-Packages` is **false**, against the Debian default: those older kernels are +the documented rollback path, and an unattended process pruning them could remove the fallback an +operator needs in front of a box that will not boot. + +### 3. needrestart restarts services — but never k3s + +A security-only policy that restarts nothing patches bytes on disk that no running process reads. +`needrestart` in mode `a` restarts services still mapped to a deleted library, which is what makes +the policy actually effective. + +`k3s.service` is blacklisted. Restarting the Kubernetes control plane unattended at 04:00 is a +worse outcome than a delayed library fix; that fix lands at the next kured reboot instead. Measured +2026-10-17, `k3s.service` is the **only** matching unit — k3s bundles its own containerd and +kubelet, and neither `containerd` nor `containerd.io` is an apt package on these nodes — so this +one entry is complete today. + +## Schedule + +`apt-daily` at 03:30 and `apt-daily-upgrade` at 04:00, both `±60m` randomised, via systemd drop-ins. +Chosen to land **inside** kured's 03:00–06:00 reboot window so that a sentinel written by an upgrade +can be acted on the same night rather than ~24h later. + +Two traps the role handles explicitly: + +- `/etc/apt/apt.conf.d/20auto-upgrades` must set the four `APT::Periodic::*` counters. Without + them the apt-daily timers fire and **do nothing** — installed, configured, green, and inert. +- The timer drop-ins clear `OnCalendar=` before setting it. systemd **accumulates** `OnCalendar` + entries across drop-ins, so omitting the reset keeps the shipped schedule *in addition* to ours. + +## Why a new role rather than extending `node_baseline` + +`node_baseline` is the repo's only other apt surface and would have been a defensible home. Two +reasons against: + +1. **Single responsibility.** `node_baseline` is the *first-boot floor* — it runs before k3s exists + and is the thing that makes a node installable. Continuous patching policy is a different + lifecycle with a different blast radius, and it wants to be runnable (and re-runnable) without + going anywhere near that role's data-disk tasks. +2. **Concurrent work.** Other open branches touch `roles/node_baseline/`. A new role has nothing to + merge. + +Nothing here duplicates `node_baseline`: it does not add apt sources, does not touch the kernel, and +does not manage disks. + +## Post-merge verification + +Running the role is safe on a live cluster at any time — it installs and configures the upgrader, and +never runs `apt upgrade`, restarts k3s, or reboots. + +```bash +# Apply (prod). Idempotent; re-runnable. +cd infra/ansible && ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod + +# Re-run ONLY the read-only assertions + the per-node pending-reboot summary. +cd infra/ansible && ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify +``` + +Then, on a node, confirm the three properties that matter (all read-only, no sudo needed for the +first two): + +```bash +ssh k3s@192.168.2.10 'apt-config dump | grep -iE "unattended-upgrade::(automatic-reboot|package-blacklist|origins)"' +ssh k3s@192.168.2.10 'systemctl list-timers apt-daily-upgrade.timer --no-pager' +ssh k3s@192.168.2.10 'ls -la /var/run/reboot-required 2>&1; dpkg -l update-notifier-common needrestart unattended-upgrades | tail -4' +``` + +Expect: `Automatic-Reboot "false"`, a blacklist containing `^linux-`, an `Origins-Pattern` +containing `Debian-Security` and **not** `Backports`, a next-elapse around 04:00, and all three +packages `ii`. + +A dry run of what the upgrader would actually do (needs root, so this one is an operator step): + +```bash +sudo unattended-upgrade --dry-run --debug +``` + +The kernel must appear in its blacklist output, not its upgrade list. + +## What this role cannot tell you + +- **That a security upgrade has ever succeeded.** That is in + `/var/log/unattended-upgrades/unattended-upgrades.log` on each node, and nothing in this repo or + in Prometheus reads it. There is no alert for "automatic patching has silently stopped working". + That is a real, known gap — see the handoff entry + `decisions/2026-10-17-automated-upgrades.md`. +- **That a reboot actually happened.** That is kured's half, and it *is* alerted on + (`KuredRebootPending`, `KuredNodeCoverageGap`, `KuredNotReporting` in + `platform/kured/alerts.yaml`). diff --git a/infra/ansible/roles/unattended_upgrades/defaults/main.yml b/infra/ansible/roles/unattended_upgrades/defaults/main.yml new file mode 100644 index 0000000..f93a14f --- /dev/null +++ b/infra/ansible/roles/unattended_upgrades/defaults/main.yml @@ -0,0 +1,172 @@ +--- +# roles/unattended_upgrades — automatic Debian SECURITY patching on the k3s nodes, with the kernel +# deliberately excluded and with reboots delegated to kured (platform/kured). +# +# OFF BY DEFAULT, ON FOR PROD. Same seam as node_baseline_backports_kernel / etcd_io_priority_enabled: +# a plain boolean, default false, turned on in inventory/group_vars/k3s_prod/vars.yml. Not a hostname +# test — that would encode the environment into the role (see node_baseline's defaults for the full +# argument, which this role follows rather than restates). +# +# ═════════════════════════════════════════════════════════════════════════════════════════════════ +# STATE BEFORE THIS ROLE (measured on all three prod nodes, 2026-10-17, via `dpkg-query -W`) +# ═════════════════════════════════════════════════════════════════════════════════════════════════ +# unattended-upgrades NOT INSTALLED +# update-notifier-common NOT INSTALLED +# needrestart NOT INSTALLED +# apt-mark showhold EMPTY (no package is held on any node) +# /var/run/reboot-required ABSENT, and nothing on the node can create it +# /etc/apt/preferences.d/ EMPTY (no apt pin declares the backports-kernel intent) +# So: zero automatic patching, and no mechanism by which a reboot could ever be signalled. That last +# point is why this role and platform/kured must land together — kured watches +# /var/run/reboot-required, and on these nodes today NOTHING WRITES IT. Deploying kured alone would +# produce a daemon that is permanently, silently idle while looking perfectly healthy. +# +# ═════════════════════════════════════════════════════════════════════════════════════════════════ +# DECISION 1 — SECURITY ONLY, NOT FULL UPGRADES +# ═════════════════════════════════════════════════════════════════════════════════════════════════ +# Measured `apt-cache policy` on k3s-prod-1, 2026-10-17, gives the four archives and their pins: +# trixie o=Debian, a=stable, l=Debian pin 500 +# trixie-updates o=Debian, a=stable-updates, l=Debian pin 500 +# trixie-security o=Debian, a=stable-security, l=Debian-Security pin 500 +# trixie-backports o=Debian Backports, a=stable-backports, l=Debian Backports pin 100 +# +# Only `Debian-Security` is allowed below. The reasoning, in order of weight: +# +# * Security updates are the class where the cost of NOT applying them is unbounded and the +# change itself is minimal by Debian policy (targeted fixes against the frozen stable version, +# no feature churn). It is the one class worth applying with nobody watching. +# * A full `apt upgrade` on these nodes would also pull trixie point-release changes, which is a +# larger surface than anyone has reviewed for a 3-node etcd cluster, for no urgency. +# * `trixie-updates` is DELIBERATELY EXCLUDED, not overlooked. It carries non-security fixes that +# are too urgent to wait for a point release — useful, but it is the archive through which an +# unreviewed change would most plausibly arrive, and it is not what "unattended" should mean on +# a control-plane node. Add it consciously later if the operator wants it; the knob is below. +# * `trixie-backports` is EXCLUDED, AND THAT IS THE KERNEL ARGUMENT — see DECISION 2. +# +# ═════════════════════════════════════════════════════════════════════════════════════════════════ +# DECISION 2 — THE KERNEL IS EXCLUDED TWICE OVER. THIS IS THE HIGHEST-RISK PART OF THE CHANGE. +# ═════════════════════════════════════════════════════════════════════════════════════════════════ +# The constraint: HP's R21 firmware on these EliteDesk 800 G5 minis publishes a corrupt EFI Memory +# Attributes Table, and Linux < 6.13 does not validate it before reserving memory from its contents +# — k3s-prod-2 twice booted with 2.6 GiB of 16 GB. Upstream 8fbe4c49c0cc (first in v6.13) is the +# ONLY fix; there is no kernel cmdline workaround. So these nodes run the backports metapackage +# `linux-image-amd64` (measured 2026-10-17: version 7.1.8-1~bpo13+1 installed and running on all +# three, well above the floor). Full write-up: infra/PROD-PROVISIONING.md §0b, and the rationale +# block in roles/node_baseline/defaults/main.yml:21-51. +# +# A NODE THAT COMES BACK FROM AN UNATTENDED UPGRADE ON A 6.12 KERNEL IS A NODE THAT MAY COME BACK +# WITH A FRACTION OF ITS RAM. Note that 6.12 kernels are not hypothetical here — measured +# 2026-10-17, `linux-image-6.12.107+deb13-amd64` and `linux-image-6.12.94+deb13-amd64` are BOTH +# still installed on all three nodes, below the floor, bootable, and nothing holds them. +# +# Two independent mechanisms keep unattended-upgrades away from the kernel, and the role asserts +# both rather than trusting either: +# +# 1. ORIGIN EXCLUSION (structural). The running kernel comes from `o=Debian Backports`, which is +# not in Allowed-Origins. u-u only ever considers packages from allowed origins, so it cannot +# see a backports kernel upgrade at all. This also means the backports kernel receives no +# automatic patching — accepted, and in any case backports carries no security guarantees +# (node_baseline's defaults state that cost plainly: "no security-tracker commitment, no DSAs"). +# 2. PACKAGE BLACKLIST (belt and braces). `linux-*` is blacklisted below, so even if the origin +# list is later widened — the most likely future edit to this file, and the one that would +# silently re-arm the hazard — the kernel is still untouchable. Without this, widening origins +# to include backports would hand an unattended process the EFI-memattr-critical package. +# +# WHAT THIS DOES *NOT* DO, AND MUST NOT BE READ AS DOING: it does not stop the kernel moving. The +# operator still advances it deliberately with +# ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel --limit +# which is the only thing that maintains the backports source, the metapackage and the on-node +# provenance stamp. node_baseline's own comment ("`ButAutomaticUpgrades: yes` ... means `apt upgrade` +# keeps it current once installed") still holds for an operator-run apt; the measured pin priority of +# 100 is exactly the NotAutomatic + ButAutomaticUpgrades signature that produces that behaviour. +# Nothing here changes it. What changes is that a NEW unattended actor is forbidden from the kernel. +# +# NOT BLACKLISTED, BECAUSE THEY ARE NOT APT PACKAGES AT ALL — measured 2026-10-17: `dpkg-query` finds +# no `k3s`, no `containerd` and no `containerd.io`. k3s is a single static binary at +# /usr/local/bin/k3s installed by roles/k3s_server, and it bundles its own containerd and kubelet. +# So the entire Kubernetes control plane and container runtime are OUTSIDE apt's reach and cannot be +# upgraded by this role even in principle. That is worth knowing before worrying about it. +unattended_upgrades_enabled: false + +# The packages this role installs. Each one is load-bearing; see the notes. +unattended_upgrades_packages: + # The upgrader itself. + - unattended-upgrades + # THE SENTINEL WRITER, AND THE WHOLE REASON kured CAN WORK. On Debian, /var/run/reboot-required is + # not created by the kernel, by apt, or by unattended-upgrades — it is created by a hook shipped in + # update-notifier-common. Measured ABSENT on all three nodes. Without this package the file is + # never written, kured correctly never reboots, every dashboard stays green, and the reboot half of + # this PR is an elaborate no-op. The role asserts the hook exists after install rather than + # assuming the package shipped one (see tasks/main.yml). + - update-notifier-common + # Restarts services still mapped to a deleted library after a security upgrade. Without it a + # security-only policy patches bytes on disk that no running process reads — see DECISION 3. + - needrestart + +# --- Allowed origins ----------------------------------------------------------------------------- +# Verbatim apt origin expressions, substituted into 50unattended-upgrades. The ${distro_*} tokens +# here are UNATTENDED-UPGRADES' OWN placeholders, expanded by it at runtime — they are not Ansible +# or Flux substitutions, which is why they are single-quoted and why this role's files never travel +# through Flux envsubst. +# +# `l=Debian-Security` plus the `-security` codename is the form Debian's own shipped config uses, and +# both halves were confirmed against the measured archive metadata above (l=Debian-Security, +# n=trixie-security). Matching on the LABEL rather than the suite name is what makes this survive the +# next Debian release. +unattended_upgrades_allowed_origins: + - 'origin=Debian,codename=${distro_codename},label=Debian-Security' + - 'origin=Debian,codename=${distro_codename}-security,label=Debian-Security' +# Set this to add 'origin=Debian,codename=${distro_codename}-updates,label=Debian' — a deliberate +# widening, kept as a knob so that turning it on is a reviewable one-line change rather than an edit +# to a heredoc. See DECISION 1 for why it is off. +unattended_upgrades_include_updates: false + +# --- Package blacklist --------------------------------------------------------------------------- +# Regular expressions matched against package names. See DECISION 2: `linux-` is the EFI-memattr +# guard and must not be removed without re-reading PROD-PROVISIONING.md §0b. +unattended_upgrades_blacklist: + # Every kernel package: linux-image-*, linux-headers-*, linux-image-amd64 (the metapackage that + # actually tracks backports), plus linux-kbuild / linux-libc-dev. Anchored at the start so it + # cannot accidentally match an unrelated package containing the substring. + - '^linux-' + # Microcode changes take effect only on reboot and can alter boot behaviour on the very firmware + # that is already misbehaving here. Operator-only, same argument as the kernel. + - '^intel-microcode$' + - '^amd64-microcode$' + # The bootloader. A grub upgrade rewrites boot configuration on a machine with a documented + # warm-reboot hang (k3s-prod-2, 2026-09-03). Never unattended. + - '^grub-' + - '^grub2-common$' + +# --- Reboot policy ------------------------------------------------------------------------------- +# FALSE, AND THIS IS THE DIVISION OF LABOUR WITH kured, NOT AN OVERSIGHT. +# unattended-upgrades can reboot a node itself (Automatic-Reboot "true"). It must not here: it has +# no notion of a cluster, so it would not cordon, would not drain, would not evict the 25 attached +# Longhorn volumes cleanly, would not check whether another node is already down, and — on three +# control-plane nodes where etcd quorum is 2 of 3 — three nodes each deciding independently to +# reboot at their own `Automatic-Reboot-Time` is precisely how you lose the cluster. +# u-u therefore only ever leaves the sentinel behind; kured owns every reboot, one node at a time, +# under a cluster-wide lock. See platform/kured/helmrelease.yaml. +unattended_upgrades_automatic_reboot: false + +# --- needrestart (DECISION 3) -------------------------------------------------------------------- +# See the role README for the full argument. Summary: 'a' (restart services automatically) is what +# makes a security-only policy actually effective — a patched libssl on disk does nothing for a +# process still mapping the old one — but k3s.service is EXCLUDED, because restarting the +# Kubernetes control plane unattended at 06:00 is a worse outcome than a delayed library fix. +# Measured 2026-10-17: the only matching unit on these nodes is `k3s.service` (no separate +# containerd or kubelet unit exists; k3s bundles both), so this blacklist is complete today. +unattended_upgrades_needrestart_mode: a +unattended_upgrades_needrestart_blacklist: + - '^k3s' + +# --- Schedule ------------------------------------------------------------------------------------ +# apt's own systemd timers (apt-daily, apt-daily-upgrade) drive u-u; these set their windows. +# 04:00 with a 1h randomised delay, so the upgrade lands INSIDE kured's 03:00-06:00 reboot window +# (platform/kured/helmrelease.yaml) rather than hours away from it: if an upgrade ever does set the +# sentinel, the reboot can be taken the same night instead of waiting ~24h. The randomised delay +# keeps the three nodes from hitting the mirror simultaneously; it does NOT serialise anything, and +# it does not need to — u-u never reboots, so three concurrent upgrades are harmless. +unattended_upgrades_download_time: "03:30" +unattended_upgrades_upgrade_time: "04:00" +unattended_upgrades_randomised_delay: 60m diff --git a/infra/ansible/roles/unattended_upgrades/handlers/main.yml b/infra/ansible/roles/unattended_upgrades/handlers/main.yml new file mode 100644 index 0000000..37dadc0 --- /dev/null +++ b/infra/ansible/roles/unattended_upgrades/handlers/main.yml @@ -0,0 +1,29 @@ +--- +# Re-arm apt's periodic timers after a config or drop-in change. +# +# WHY A daemon_reload IS REQUIRED AND NOT OPTIONAL. The timer windows are set by drop-ins under +# /etc/systemd/system/apt-daily{,-upgrade}.timer.d/. systemd does not re-read unit files on its own, +# so until it is told to, the RUNNING manager keeps the shipped schedule and the new window is +# ignored — the files look right in git and on disk, `systemctl cat` shows them, and the upgrade +# still fires at the old time. That is a silent config no-op, which is the failure mode this whole +# role is written against. +# +# `daemon_reload` re-parses unit files and re-runs generators. It starts and stops nothing, so it +# cannot disturb k3s, containerd or Longhorn — the same reasoning node_baseline's "Reload systemd +# unit files" handler records. +# +# RESTARTING THE TIMERS IS ALSO SAFE, and is not the same as running the upgrade. `*.timer` units +# only hold the schedule; restarting one recomputes its next elapse and does NOT trigger its +# `*.service`. With `Persistent=true` set in the drop-in, a window missed while the node was down is +# caught up on the next boot rather than skipped. +# +# Both fire only when a task actually changed, so a re-run of the role is silent. +- name: Restart unattended-upgrades timers + ansible.builtin.systemd_service: + name: "{{ item }}" + state: restarted + enabled: true + daemon_reload: true + loop: + - apt-daily.timer + - apt-daily-upgrade.timer diff --git a/infra/ansible/roles/unattended_upgrades/tasks/main.yml b/infra/ansible/roles/unattended_upgrades/tasks/main.yml new file mode 100644 index 0000000..bbb76c0 --- /dev/null +++ b/infra/ansible/roles/unattended_upgrades/tasks/main.yml @@ -0,0 +1,309 @@ +--- +# roles/unattended_upgrades — automatic Debian security patching, kernel excluded, reboots delegated +# to kured. Every decision and its reasoning lives in defaults/main.yml; this file is the mechanism. +# +# TAGS — following roles/node_baseline, every task carries one of `packages`, `config` or `verify`. +# An untagged run does all three in that order. The split exists because `verify` is the half an +# operator will want to re-run on its own to answer "is automatic patching actually working on this +# box", without re-writing any configuration: +# ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify +# +# IT DOES NOT UPGRADE ANYTHING, AND IT DOES NOT REBOOT. This role installs and CONFIGURES the +# upgrader; it never runs `apt upgrade` itself and never takes a reboot. Both of those are deliberate +# and both follow roles/node_baseline/tasks/main.yml:61-67 ("IT DOES NOT REBOOT", after k3s-prod-2 +# hung on a warm reboot on 2026-09-03 and needed a physical power button). Applying this role is +# therefore a SAFE operation on a live cluster at any time of day: the first unattended upgrade +# happens at the next timer firing, and any reboot it implies is kured's to schedule inside the +# 03:00-06:00 window. +- name: Automatic security upgrades (unattended-upgrades; kernel excluded, reboots via kured) + when: unattended_upgrades_enabled | bool + tags: [packages, config, verify] + block: + + # ── PREFLIGHT ─────────────────────────────────────────────────────────────────────────────── + - name: Assert this is a Debian-family host + # The config file layout, the origin expressions and the package names below are all + # Debian-specific. On anything else this role would write files that are silently ignored, + # which is the failure mode this whole PR exists to avoid. + ansible.builtin.assert: + that: + - ansible_facts['os_family'] == 'Debian' + fail_msg: >- + REFUSING TO CONFIGURE unattended-upgrades on {{ inventory_hostname }}: os_family is + '{{ ansible_facts["os_family"] }}', not 'Debian'. The allowed-origin expressions and + config paths in this role are Debian-specific. + tags: [packages, config, verify] + + # ── PACKAGES ──────────────────────────────────────────────────────────────────────────────── + - name: Install unattended-upgrades, the reboot-sentinel writer and needrestart + # update-notifier-common is NOT optional padding: it owns the hook that creates + # /var/run/reboot-required, which is the only thing kured watches. See defaults/main.yml. + ansible.builtin.apt: + name: "{{ unattended_upgrades_packages }}" + state: present + update_cache: true + cache_valid_time: 3600 + tags: [packages] + + # ── CONFIG ────────────────────────────────────────────────────────────────────────────────── + # Written as a whole file rather than patched in place, for the reason node_baseline gives for + # its own apt source: an idempotent full-file write with a managed-by header is reviewable and + # re-runnable, whereas line-editing a conffile fights dpkg on every package upgrade. + # + # 52- RATHER THAN 50-: Debian ships /etc/apt/apt.conf.d/50unattended-upgrades as a CONFFILE of + # the unattended-upgrades package. Overwriting it would make every future apt upgrade of that + # package prompt about a locally-modified conffile (or, under the non-interactive defaults a + # scripted run uses, silently keep one side and discard the other). apt reads the directory in + # lexical order and later files win, so a separate 52- file overrides the shipped defaults + # without ever touching them, and `apt-config dump` in the verify block below proves which value + # actually won rather than assuming this ordering holds. + - name: Configure unattended-upgrades (52-k3s-unattended-upgrades) + ansible.builtin.copy: + dest: /etc/apt/apt.conf.d/52-k3s-unattended-upgrades + owner: root + group: root + mode: "0644" + content: | + // MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades. Local edits are reverted + // on the next run. Rationale for every value: that role's defaults/main.yml. + // + // Overrides the shipped 50unattended-upgrades by lexical ordering (52 > 50). The shipped + // file is a dpkg conffile and is deliberately left untouched. + + // SECURITY ONLY. Deliberately excludes trixie-updates (non-security fixes) and + // trixie-backports (o=Debian Backports) -- and excluding backports is what makes it + // structurally impossible for this upgrader to move the EFI-memattr-critical kernel, + // which is installed from there. See defaults/main.yml DECISION 2. + Unattended-Upgrade::Origins-Pattern { + {% for origin in unattended_upgrades_allowed_origins %} + "{{ origin }}"; + {% endfor %} + {% if unattended_upgrades_include_updates | bool %} + "origin=Debian,codename=${distro_codename}-updates,label=Debian"; + {% endif %} + }; + + // NEVER AUTOMATICALLY UPGRADED, whatever the origin list later says. `^linux-` is the + // EFI-memattr guard: these HP minis need kernel >= 6.13 or they can boot with a fraction + // of their RAM, and 6.12 kernels are still installed and bootable on all three nodes. + // Do not remove without re-reading infra/PROD-PROVISIONING.md §0b. + Unattended-Upgrade::Package-Blacklist { + {% for pkg in unattended_upgrades_blacklist %} + "{{ pkg }}"; + {% endfor %} + }; + + // REBOOTS BELONG TO kured (platform/kured), NOT TO apt. All three nodes are + // control-plane + etcd with quorum 2 of 3, so three hosts independently deciding to + // reboot at their own Automatic-Reboot-Time is how the cluster is lost. u-u leaves + // /var/run/reboot-required behind; kured takes a cluster-wide lock and reboots exactly + // one node at a time, after cordon + drain, inside 03:00-06:00, gated on Longhorn volume + // health. + Unattended-Upgrade::Automatic-Reboot "{{ unattended_upgrades_automatic_reboot | bool | lower }}"; + Unattended-Upgrade::Automatic-Reboot-WithUsers "false"; + + // Keep the node's disk from filling with superseded .debs, and keep a log an operator can + // read after the fact (/var/log/unattended-upgrades/). MinAge guards against removing a + // package still needed by a rollback. + Unattended-Upgrade::Remove-Unused-Kernel-Packages "false"; + Unattended-Upgrade::Remove-New-Unused-Dependencies "true"; + Unattended-Upgrade::Remove-Unused-Dependencies "false"; + Unattended-Upgrade::AutoFixInterruptedDpkg "true"; + Unattended-Upgrade::MinimalSteps "true"; + Unattended-Upgrade::SyslogEnable "true"; + + // Remove-Unused-Kernel-Packages is FALSE on purpose, against the Debian default. These + // nodes deliberately carry more than one kernel: the backports kernel they boot and the + // distro kernels that are the documented rollback path (node_baseline writes the rollback + // procedure onto the node itself). An unattended process pruning kernels could remove the + // fallback an operator needs at 3am in front of a box that will not boot. + notify: Restart unattended-upgrades timers + tags: [config] + + - name: Enable the apt periodic timers that actually run unattended-upgrades + # Installing the package is NOT sufficient to make it run on a schedule: without these + # counters apt's daily units do nothing, and the whole mechanism is inert while looking + # installed. `1` means "every day". + ansible.builtin.copy: + dest: /etc/apt/apt.conf.d/20auto-upgrades + owner: root + group: root + mode: "0644" + content: | + // MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades. + // Without these four counters the apt-daily timers fire and do nothing, so + // unattended-upgrades would be installed, configured, and never once invoked. + APT::Periodic::Update-Package-Lists "1"; + APT::Periodic::Download-Upgradeable-Packages "1"; + APT::Periodic::Unattended-Upgrade "1"; + APT::Periodic::AutocleanInterval "7"; + tags: [config] + + - name: Ensure the systemd timer drop-in directories exist + # `copy` does not create parent directories, and these two .timer.d/ dirs do not exist on a + # stock Debian node — without this task the next task fails with "Destination directory does + # not exist" on a first run. + ansible.builtin.file: + path: "/etc/systemd/system/{{ item }}.timer.d" + state: directory + owner: root + group: root + mode: "0755" + loop: + - apt-daily + - apt-daily-upgrade + tags: [config] + + - name: Pin the apt timer windows into kured's reboot window + # Overriding the shipped timers rather than editing them: a drop-in survives package upgrades. + # OnCalendar is cleared first (`OnCalendar=` with no value) because systemd ACCUMULATES + # OnCalendar entries across drop-ins — omit the reset and the unit keeps its shipped schedule + # IN ADDITION to ours, which would run the upgrade at both times and look like it worked. + ansible.builtin.copy: + dest: "/etc/systemd/system/{{ item.unit }}.timer.d/override.conf" + owner: root + group: root + mode: "0644" + content: | + # MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades. + # Lands the upgrade inside kured's 03:00-06:00 reboot window so a sentinel written by an + # upgrade can be acted on the same night rather than ~24h later. + [Timer] + OnCalendar= + OnCalendar={{ item.time }} + RandomizedDelaySec={{ unattended_upgrades_randomised_delay }} + Persistent=true + force: true + loop: + - { unit: apt-daily, time: "{{ unattended_upgrades_download_time }}" } + - { unit: apt-daily-upgrade, time: "{{ unattended_upgrades_upgrade_time }}" } + loop_control: + label: "{{ item.unit }} at {{ item.time }}" + notify: Restart unattended-upgrades timers + tags: [config] + + - name: Configure needrestart (restart services, but never k3s) + # See defaults/main.yml DECISION 3. Auto-restart is what makes a security-only policy real -- + # a patched library on disk does nothing for a process still mapping the deleted one -- but + # k3s.service is excluded because restarting the control plane unattended is worse than a + # delayed library fix. `$nrconf{restart}` is Perl config read by needrestart itself. + ansible.builtin.copy: + dest: /etc/needrestart/conf.d/50-k3s.conf + owner: root + group: root + mode: "0644" + content: | + # MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades. + # Restart mode: '{{ unattended_upgrades_needrestart_mode }}' (a = automatically, l = list only, i = interactive) + $nrconf{restart} = '{{ unattended_upgrades_needrestart_mode }}'; + # Units matching these patterns are NEVER restarted automatically. k3s bundles containerd + # and the kubelet, so this one entry covers the entire control plane and runtime on these + # nodes (measured 2026-10-17: k3s.service is the only matching unit). A library fix that + # needs k3s restarted therefore waits for the reboot kured will take. + $nrconf{blacklist_rc} = [ + {% for pat in unattended_upgrades_needrestart_blacklist %} + qr({{ pat }}), + {% endfor %} + ]; + tags: [config] + + # ── VERIFY ────────────────────────────────────────────────────────────────────────────────── + # These tasks exist because every component of this mechanism fails SILENTLY. An installed + # package, a written config file and a green play are all compatible with nothing ever being + # upgraded and no reboot ever being signalled. + + - name: Ask apt-config what the effective settings actually are + # `apt-config dump` is apt's OWN parse of the whole /etc/apt/apt.conf.d/ directory, so it + # proves what WON after lexical ordering rather than what this role wrote. If the shipped + # 50unattended-upgrades were ever renamed above 52-, this is the task that would catch it. + ansible.builtin.command: apt-config dump + register: unattended_upgrades_aptconfig + changed_when: false + check_mode: false + tags: [verify] + + - name: Assert the kernel is blacklisted and automatic reboots are off + # The two properties that must never silently regress. Asserted against apt's own view. + ansible.builtin.assert: + that: + # apt-config renders each list entry as `Unattended-Upgrade::Package-Blacklist:: "^linux-";` + # so the anchored pattern itself is the thing to look for. Checking for the bare string + # `linux-` would pass on almost any dump and prove nothing. + - "'\"^linux-\"' in unattended_upgrades_aptconfig.stdout" + - "'Unattended-Upgrade::Automatic-Reboot \"false\"' in unattended_upgrades_aptconfig.stdout" + # The origin list must be non-empty, or u-u runs nightly and upgrades nothing — installed, + # scheduled, green, and inert. + - "'Debian-Security' in unattended_upgrades_aptconfig.stdout" + fail_msg: >- + EFFECTIVE apt CONFIG ON {{ inventory_hostname }} IS NOT WHAT THIS ROLE INTENDED. + Either the `^linux-` kernel blacklist or `Automatic-Reboot "false"` is not winning. + Both are safety properties, not preferences: without the blacklist an unattended process + can move a kernel these HP minis must keep >= 6.13 (PROD-PROVISIONING.md §0b), and + without Automatic-Reboot=false three control-plane nodes can each reboot themselves + independently and take etcd below quorum. Inspect with `apt-config dump | grep -i + unattended` and check for a file sorting after 52- in /etc/apt/apt.conf.d/. + success_msg: >- + Effective apt config confirmed: kernel blacklisted, automatic reboot disabled. + tags: [verify] + + - name: Find the hooks that can create /var/run/reboot-required + # THE SENTINEL IS THE SINGLE POINT OF FAILURE FOR THE ENTIRE REBOOT HALF OF THIS DESIGN, and + # it is the one part that cannot be asserted from the repo — it depends on what + # update-notifier-common actually ships on this Debian release. Rather than hard-code a path + # that may not exist, discover the candidates and assert that at least one is present. + ansible.builtin.find: + paths: + - /etc/kernel/postinst.d + - /etc/apt/apt.conf.d + patterns: + - "*update-notifier*" + - "*unattended-upgrades*" + - "99update-notifier" + file_type: any + register: unattended_upgrades_sentinel_hooks + changed_when: false + tags: [verify] + + - name: Assert something on this node can write the reboot sentinel + # If this fails, kured is a no-op: it watches /var/run/reboot-required forever, the file is + # never created, no reboot is ever taken, and NOTHING GOES RED -- the DaemonSet stays 1/1 + # Running and `kured_reboot_required` reports a steady 0 on every node. That is the exact + # shape of failure (a check whose failure mode is a green tick) this repo has been burned by + # before, so it is an assert and not a debug message. + ansible.builtin.assert: + that: + - unattended_upgrades_sentinel_hooks.matched | int > 0 + fail_msg: >- + NOTHING ON {{ inventory_hostname }} CAN CREATE /var/run/reboot-required. No + update-notifier / unattended-upgrades hook was found under /etc/kernel/postinst.d or + /etc/apt/apt.conf.d. kured watches that path and reboots only when it exists, so with no + writer the reboot automation is permanently and INVISIBLY idle -- kured stays 1/1 Running + and reports kured_reboot_required=0 on every node forever. Confirm + update-notifier-common is installed (`dpkg -l update-notifier-common`) and inspect + `dpkg -L update-notifier-common | grep -E 'postinst.d|apt.conf.d'`. + success_msg: >- + Reboot sentinel writer present ({{ unattended_upgrades_sentinel_hooks.matched }} hook(s) + found) -- a kernel or flagged package upgrade will signal kured. + tags: [verify] + + - name: Report the pending-reboot state of this node + # Read-only. Tells the operator whether this node is currently waiting on kured for a reboot, + # which is the question `--tags verify` is usually being run to answer. + ansible.builtin.stat: + path: /var/run/reboot-required + register: unattended_upgrades_reboot_required + changed_when: false + tags: [verify] + + - name: Summary + ansible.builtin.debug: + msg: >- + {{ inventory_hostname }}: security-only unattended upgrades ACTIVE + (upgrade timer {{ unattended_upgrades_upgrade_time }} +/-{{ unattended_upgrades_randomised_delay }}), + kernel and microcode blacklisted, automatic reboot DISABLED. + Reboot currently pending: {{ unattended_upgrades_reboot_required.stat.exists }}. + {% if unattended_upgrades_reboot_required.stat.exists %} + This node is waiting for kured to reboot it in the next 03:00-06:00 window; it will be + cordoned and drained first, and no other node will be rebooted at the same time. + {% endif %} + tags: [verify] diff --git a/platform/kured/README.md b/platform/kured/README.md new file mode 100644 index 0000000..2152d32 --- /dev/null +++ b/platform/kured/README.md @@ -0,0 +1,127 @@ +# platform/kured + +**kured (Kubernetes Reboot Daemon)** — coordinated, one-node-at-a-time OS reboots of the three prod +control-plane nodes. + +- **Flux tier:** `clusters/base/kured.yaml` (its own Kustomization, `dependsOn: [platform]`, + `wait: true`) +- **Chart:** `kured` 6.1.0 → appVersion 1.23.0, from `https://kubereboot.github.io/charts` +- **The other half:** `infra/ansible/roles/unattended_upgrades` — the package upgrades. **Neither + half is useful alone.** +- **Runbook:** `/runbooks/node-reboot-automation` +- **Every tuned value and its reasoning:** `helmrelease.yaml`. That file is the argument; this one is + the orientation. +- **Design record:** `decisions/2026-10-17-automated-upgrades.md` + +## What this component owns + +The *mechanics* of a reboot: take a cluster-wide lock, cordon, drain, reboot, uncordon, hold the +lock a while longer. Nothing else. + +**It does not decide whether to reboot.** kured reboots a node if and only if +`/var/run/reboot-required` exists there. On these nodes that file is written by a deliberate +operator kernel upgrade (`playbooks/node-baseline.yml --tags kernel`) — unattended-upgrades is +security-only with the kernel blacklisted, so it essentially never produces one. **The human gate is +the sentinel, not the daemon.** + +## The problem this fixes + +Rebooting these three nodes was a manual, serialized, supervised procedure, and +`roles/node_baseline` has to end its kernel run by shouting the rules at the operator: *"Reboot prod +nodes ONE AT A TIME, verify each returns before touching the next, and NEVER node 2 first"*. That is +a correct procedure and an error-prone one, and nothing enforced it. + +What it has to get right, measured on prod 2026-10-17: + +- **etcd quorum is 2 of 3** — all three nodes are control-plane+etcd. Two down is a lost cluster. +- **Every drain degrades all 25 Longhorn volumes to 2/3**, because 3 nodes × 3 replicas × + `replica-soft-anti-affinity: false` leaves nowhere to place a replacement, and rebuild is + serialized behind a 600s wait. Reboot the next node too early and volumes hit 1/3, where + `block-if-contains-last-replica` can block a drain forever. +- **Six PodDisruptionBudgets allow zero voluntary disruptions** — 3 Longhorn `instance-manager-*` + (structurally 0 by design; this is how Longhorn gates the drain) and 3 CNPG `*-primary`. + +## Placement — why its own Kustomization + +`platform` is `wait: true` over twelve components and gates `platform-config` → every apps wave. A +reboot daemon that failed to go healthy there would stall **every application in the cluster** on +"dependency not ready". A component added to make maintenance safer must not be able to cause an +outage. `wait: true` here is a health **signal** for something nothing depends on — and nothing +should ever `dependsOn: [kured]`. + +The `dependsOn: [platform]` edge is load-bearing, not conventional: it supplies the +`monitoring.coreos.com` CRDs this component's `PrometheusRule` needs (F-MON1 — a pass containing an +unknown kind aborts), plus the Prometheus that kured **queries** before every reboot. + +## How concurrent reboots are made impossible + +| mechanism | value | why | +|---|---|---| +| cluster-wide lock | one annotation on kured's own DaemonSet | a node reboots only while holding it; enforced by API-server optimistic concurrency, not timing | +| `concurrency` | `1` | one lock holder at a time | +| **`lockTtl`** | **`0` (disabled)** | a TTL lets reboots continue after the holder dies — on 3 etcd nodes that is the catastrophe. A hung node holds the lock **forever** and nothing cascades. | +| `lockReleaseDelay` | `30m` | the next node waits out Longhorn's 600s replenishment even if no alert latched | +| alert gate | `alertFilterMatchOnly: true` + anchored allow-list | refuses to proceed while a volume is degraded/faulted or a node is down | +| window | 03:00–06:00 daily, America/New_York | matches `renovate.json`'s window; daily (not weekly) is what makes `KuredRebootPending`'s `for: 36h` meaningful | + +## Deliberate non-choices + +- **No `--ignore-daemonsets` / `--delete-emptydir-data` / `--force`.** Those flags **do not exist** + in kured 1.23.0 — it hardcodes all three in the drain helper (`cmd/kured/main.go:438-449`). + Passing them would CrashLoop the DaemonSet. All three are genuinely required here (12 DaemonSets, + 135 emptyDir pods). +- **`rebootMethod: command`, not `signal`.** `signal` would allow `privileged: false` + `CAP_KILL`, + but needs a host-side `kured-reboot.service` that does not exist on these nodes. A real future + hardening step; explicitly not taken here. +- **No tolerations.** No prod node is tainted (measured). A future tainted node would be silently + uncovered — hence `KuredNodeCoverageGap`. +- **`NodeEtcdRequestsSlow` / `NodeIOPressure*` are NOT in the alert gate**, despite being exactly + what you would want to pause for. prod-1 has a documented etcd WAL I/O starvation history and + including them risks a permanent wedge that looks like safety. +- **No Slack/shoutrrr notifications.** Would require a SOPS Secret and a decryption patch; the three + alerts cover the same ground through Alertmanager. + +## Versioning + +Chart pinned `>=6.1.0 <6.2.0`. The minor ceiling is **deliberate here**, unlike elsewhere in this +repo where it is a main cause of pin rot: a kured minor may change drain or lock semantics, and this +is the one component where "a human read the changelog" should be a precondition. `renovate.json`'s +`blast-radius` rule additionally forbids auto-merging it, so the ceiling is a review prompt, not a +wall. + +## What breaks silently, and what catches it + +Every failure mode of this component leaves a cluster that looks healthy — kured stays 1/1 Running +while rebooting nothing. `alerts.yaml` ships three rules over kured's single metric, with a +companion promtool suite: + +| alert | catches | +|---|---| +| `KuredNotReporting` | the scrape pipeline is dead (chart defaults `service.create`/`metrics.create` to **false**; a NetworkPolicy can time the scrape out) — and the other two are inert while this is true | +| `KuredNodeCoverageGap` | kured missing on *some* node, which `absent()` cannot see | +| `KuredRebootPending` | a reboot was offered and blocked for >36h | + +## Post-merge verification + +```bash +infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds,pod,svc,servicemonitor -o wide' +infra/devcontainer/dev.sh run bash -c 'flux --context k3s-prod get kustomization kured' +infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured logs ds/kured --tail=40' +``` + +Expect `3/3` ready, the Kustomization `Ready=True`, and kured logging a reboot-check decision each +period. Then confirm the scrape is real — this is the step that proves the alerts are not inert: + +```bash +infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n monitoring exec \ + sts/prometheus-kube-prometheus-stack-prometheus -c prometheus -- \ + wget -qO- "http://localhost:9090/api/v1/query?query=count(kured_reboot_required)"' +``` + +**Expect `3`.** Anything else (and especially an empty result) means the Service, ServiceMonitor or +NetworkPolicy is wrong and all three alerts are silently dead. + +**Before trusting the timer with a real kernel upgrade**, do one watched drain to settle the open +question the design record flags: whether CNPG actually switches its primary away in response to a +drain-induced eviction. If it does not, draining `k3s-prod-1` blocks on three 0-disruption PDBs at +once. diff --git a/platform/kured/alerts.promtool-test.yaml b/platform/kured/alerts.promtool-test.yaml new file mode 100644 index 0000000..288fcfa --- /dev/null +++ b/platform/kured/alerts.promtool-test.yaml @@ -0,0 +1,282 @@ +--- +# promtool unit tests for the three rules in platform/kured/alerts.yaml. +# +# WHY THIS FILE EXISTS. The rules it covers are the ONLY thing that can tell an operator that the +# reboot automation has stopped working, and every failure mode they describe is silent — kured stays +# 1/1 Running while rebooting nothing. An alert that cannot fire is therefore not a cosmetic defect +# here, it is the removal of the only signal. This repo has shipped that bug in both directions +# already (a permanently-firing VPNTunnelDown; a structurally inert VeleroNoRecentBackup over an +# absent series), which is why the harness exists at all. Each rule below is pinned BOTH ways: quiet +# on the healthy state, firing on the constructed bad one, with the `for:` boundary asserted rather +# than assumed. +# +# HOW TO RUN. `platform/monitoring/tests/run.sh` — the one repo-wide rule-unit harness. It finds this +# file by the COMPANION convention (`.promtool-test.yaml` beside `.yaml`), so the +# manifest under test is exactly platform/kured/alerts.yaml and nothing else. Run just this file with +# platform/monitoring/tests/run.sh platform/kured/alerts.promtool-test.yaml +# The harness flattens the PrometheusRule CR's `.spec.groups` into the bare rules file promtool wants, +# `promtool check rules` it first so a non-parsing file cannot pass vacuously, and runs it all in a +# temp dir. It leaves Flux's `${BASE_DOMAIN}` in the runbook_url annotations verbatim, which is safe +# because the assertions below query the `ALERTS` series and never compare annotations. +# +# WHY THE ASSERTIONS QUERY `ALERTS` RATHER THAN USING `exp_alerts`. promtool's alert_rule_test +# compares annotations VERBATIM, which would mean pasting each rule's description prose in here and +# re-pasting it on every wording change. Querying the synthetic `ALERTS` series proves what actually +# matters — the alert reaches `pending` and then `firing` across its `for:` boundary, with the right +# labels — and stays decoupled from annotation text. Same choice as +# platform/cnpg/alerts.promtool-test.yaml. +# +# THE SERIES SHAPES ARE REAL. `kured_reboot_required{node=...}` is a GaugeVec with exactly one label, +# `node` — read from kured's cmd/kured/main.go:92-96 (the only registered metric in the project). +# `kube_node_info{node=...}` was measured on prod Prometheus on 2026-10-17 returning +# `count(kube_node_info) == 3`, matching the three nodes. The extra kube_node_info labels prod +# carries are omitted deliberately: every rule aggregates with `count()`, so they cannot affect the +# result, and inventing plausible-but-wrong values would be worse than leaving them out. +# +# evaluation_interval is 5m, not the usual 1m: KuredRebootPending has `for: 36h`, and at a 1m +# interval promtool would evaluate that test over 2000+ steps for no added coverage. +rule_files: + - alerts.rules.yaml +evaluation_interval: 5m + +tests: + # ───────────────────────────────────────────────────────────────────────────────────────────── + # 1. THE HEALTHY STEADY STATE. Three nodes, kured running on all three, nothing pending. + # ALL THREE RULES MUST BE SILENT. This is the case that a badly-written `absent()` or a + # `count() <` comparison with an off-by-one gets wrong, and it is also the state the cluster + # is in ~100% of the time, so a false positive here would train the operator to ignore the + # whole group. + # ───────────────────────────────────────────────────────────────────────────────────────────── + - interval: 5m + name: healthy cluster - no reboots pending, kured on every node, all three rules quiet + input_series: + - series: 'kured_reboot_required{node="k3s-prod-1"}' + values: '0x60' + - series: 'kured_reboot_required{node="k3s-prod-2"}' + values: '0x60' + - series: 'kured_reboot_required{node="k3s-prod-3"}' + values: '0x60' + - series: 'kube_node_info{node="k3s-prod-1"}' + values: '1x60' + - series: 'kube_node_info{node="k3s-prod-2"}' + values: '1x60' + - series: 'kube_node_info{node="k3s-prod-3"}' + values: '1x60' + promql_expr_test: + # Nothing from this group may be pending or firing, at any point in the window. + - expr: 'count(ALERTS{alertname=~"Kured.*"})' + eval_time: 4h + exp_samples: [] + # And the underlying expressions must each be genuinely false, not merely unevaluated: + # absent() yields no series because the vector is populated... + - expr: 'absent(kured_reboot_required)' + eval_time: 4h + exp_samples: [] + # ...coverage is exact (3 == 3), so the `<` comparison drops the sample... + - expr: 'count(kured_reboot_required) < count(kube_node_info)' + eval_time: 4h + exp_samples: [] + # ...and no node reports a pending reboot. + - expr: 'max by (node) (kured_reboot_required) == 1' + eval_time: 4h + exp_samples: [] + + # ───────────────────────────────────────────────────────────────────────────────────────────── + # 2. KuredNotReporting — THE WHOLE PIPELINE IS DEAD. No kured series at all, which is what the + # cluster looks like if the chart's `service.create`/`metrics.create` regress to their FALSE + # defaults, if the ServiceMonitor is dropped, or if a NetworkPolicy silently times out the + # scrape (the Longhorn precedent). The nodes still exist, so kube_node_info is present — + # proving the rule keys on kured's absence and not on a dead cluster. + # ───────────────────────────────────────────────────────────────────────────────────────────── + - interval: 5m + name: no kured series anywhere - KuredNotReporting fires after 30m + input_series: + - series: 'kube_node_info{node="k3s-prod-1"}' + values: '1x60' + - series: 'kube_node_info{node="k3s-prod-2"}' + values: '1x60' + - series: 'kube_node_info{node="k3s-prod-3"}' + values: '1x60' + promql_expr_test: + # The expression itself yields 1 immediately. + - expr: 'absent(kured_reboot_required)' + eval_time: 10m + exp_samples: + - labels: '{}' + value: 1 + # `for: 30m` is honoured — still only pending at 25m. + - expr: 'ALERTS{alertname="KuredNotReporting"}' + eval_time: 25m + exp_samples: + - labels: 'ALERTS{alertname="KuredNotReporting", alertstate="pending", severity="warning"}' + value: 1 + # ...and firing once past it. + - expr: 'ALERTS{alertname="KuredNotReporting", alertstate="firing"}' + eval_time: 35m + exp_samples: + - labels: 'ALERTS{alertname="KuredNotReporting", alertstate="firing", severity="warning"}' + value: 1 + # THE GUARD CLAUSE: with kured absent, the other two rules are inert — exactly the reason + # this rule has to exist. KuredRebootPending cannot fire (no series to be 1) and + # KuredNodeCoverageGap cannot either, because `count()` over an empty vector returns NO + # SAMPLE rather than 0, so the `<` comparison has nothing to compare. Asserting that here + # pins the dependency: if a future edit made coverage-gap fire in this state, the overlap + # would be deliberate rather than accidental. + - expr: 'ALERTS{alertname="KuredRebootPending"}' + eval_time: 35m + exp_samples: [] + - expr: 'ALERTS{alertname="KuredNodeCoverageGap"}' + eval_time: 35m + exp_samples: [] + - expr: 'count(kured_reboot_required) < count(kube_node_info)' + eval_time: 35m + exp_samples: [] + + # ───────────────────────────────────────────────────────────────────────────────────────────── + # 3. KuredNodeCoverageGap — kured COVERS ONLY TWO OF THREE NODES. k3s-prod-3's series stops at + # 1h and never returns (a node taint this DaemonSet does not tolerate, a node that never came + # back from a reboot, a DaemonSet nodeSelector edit). This also pins the `for: 2h` choice, + # which exists SO THAT AN ORDINARY REBOOT DOES NOT TRIP IT: the gap is real from ~1h05 but + # the alert is only pending an hour later, and a healthy reboot is minutes. + # ───────────────────────────────────────────────────────────────────────────────────────────── + - interval: 5m + name: kured missing on one node - KuredNodeCoverageGap fires after 2h, not before + input_series: + - series: 'kured_reboot_required{node="k3s-prod-1"}' + values: '0x60' + - series: 'kured_reboot_required{node="k3s-prod-2"}' + values: '0x60' + # Ends after 1h (13 samples at 5m) and is stale thereafter. + - series: 'kured_reboot_required{node="k3s-prod-3"}' + values: '0x12' + - series: 'kube_node_info{node="k3s-prod-1"}' + values: '1x60' + - series: 'kube_node_info{node="k3s-prod-2"}' + values: '1x60' + - series: 'kube_node_info{node="k3s-prod-3"}' + values: '1x60' + promql_expr_test: + # Before the gap opens, coverage is exact and nothing is pending. + - expr: 'ALERTS{alertname="KuredNodeCoverageGap"}' + eval_time: 30m + exp_samples: [] + # The shortfall is real well before the alert fires — 2 < 3. + - expr: 'count(kured_reboot_required)' + eval_time: 2h + exp_samples: + - labels: '{}' + value: 2 + # A SHORT OUTAGE DOES NOT ALERT: still only pending at 2h, an hour after the gap opened. + - expr: 'ALERTS{alertname="KuredNodeCoverageGap"}' + eval_time: 2h + exp_samples: + - labels: 'ALERTS{alertname="KuredNodeCoverageGap", alertstate="pending", severity="warning"}' + value: 1 + # Firing once the gap has persisted past `for: 2h`. + - expr: 'ALERTS{alertname="KuredNodeCoverageGap", alertstate="firing"}' + eval_time: 3h15m + exp_samples: + - labels: 'ALERTS{alertname="KuredNodeCoverageGap", alertstate="firing", severity="warning"}' + value: 1 + # KuredNotReporting must stay QUIET: two nodes are still publishing, so the vector is not + # absent. This is the distinction between "kured is gone" and "kured is incomplete", and + # getting it wrong in either direction would collapse two different diagnoses into one. + - expr: 'ALERTS{alertname="KuredNotReporting"}' + eval_time: 3h15m + exp_samples: [] + + # ───────────────────────────────────────────────────────────────────────────────────────────── + # 4. KuredRebootPending — A REBOOT IS BLOCKED. k3s-prod-1 reports a pending reboot continuously: + # the sentinel was written (an operator kernel upgrade) and the reboot has not happened across + # more than one 03:00-06:00 window. Asserts the 36h boundary in both directions, which is the + # assertion that makes the DAILY-window decision load-bearing rather than cosmetic: on a + # weekly window this threshold could not distinguish "blocked" from "waiting for Sunday". + # Series run to 38h20m (456 samples at 5m) so the post-boundary evaluation has real data. + # ───────────────────────────────────────────────────────────────────────────────────────────── + - interval: 5m + name: reboot pending on one node for over 36h - KuredRebootPending fires, per-node label kept + input_series: + - series: 'kured_reboot_required{node="k3s-prod-1"}' + values: '1x456' + - series: 'kured_reboot_required{node="k3s-prod-2"}' + values: '0x456' + - series: 'kured_reboot_required{node="k3s-prod-3"}' + values: '0x456' + - series: 'kube_node_info{node="k3s-prod-1"}' + values: '1x456' + - series: 'kube_node_info{node="k3s-prod-2"}' + values: '1x456' + - series: 'kube_node_info{node="k3s-prod-3"}' + values: '1x456' + promql_expr_test: + # The expression selects ONLY the pending node — the operator needs to know which box. + - expr: 'max by (node) (kured_reboot_required) == 1' + eval_time: 1h + exp_samples: + - labels: '{node="k3s-prod-1"}' + value: 1 + # A NORMAL WAIT IS NOT AN ALERT. At 24h the node is still inside its legitimate + # once-per-day window wait, so the alert may be pending but must NOT be firing. + - expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}' + eval_time: 24h + exp_samples: [] + # Still not firing just short of the boundary. + - expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}' + eval_time: 35h30m + exp_samples: [] + # Firing past it, carrying the node label through to the notification. + - expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}' + eval_time: 36h30m + exp_samples: + - labels: 'ALERTS{alertname="KuredRebootPending", alertstate="firing", node="k3s-prod-1", severity="warning"}' + value: 1 + # The two healthy nodes must NOT be alerted on. A rule written with `kured_reboot_required` + # unaggregated, or compared with `>= 0`, would raise all three and bury the real one. + - expr: 'count(ALERTS{alertname="KuredRebootPending", alertstate="firing"})' + eval_time: 36h30m + exp_samples: + - labels: '{}' + value: 1 + # Liveness rules stay quiet: kured is present and covering all three nodes. A pending reboot + # is NOT a wiring fault and must not be reported as one. + - expr: 'ALERTS{alertname="KuredNotReporting"}' + eval_time: 36h30m + exp_samples: [] + - expr: 'ALERTS{alertname="KuredNodeCoverageGap"}' + eval_time: 36h30m + exp_samples: [] + + # ───────────────────────────────────────────────────────────────────────────────────────────── + # 5. THE RULE MUST STOP. A reboot that SUCCEEDS clears the sentinel, and the alert must clear + # with it — the other half of the failure space this harness exists for (a permanently-firing + # rule is how VPNTunnelDown taught this repo the lesson). k3s-prod-1 is pending for ~40h, + # long enough to fire, then the reboot happens and the gauge returns to 0. + # ───────────────────────────────────────────────────────────────────────────────────────────── + - interval: 5m + name: reboot completes after firing - KuredRebootPending resolves + input_series: + # 1 for 480 samples (40h), then 0 for the remainder. + - series: 'kured_reboot_required{node="k3s-prod-1"}' + values: '1x480 0x120' + - series: 'kured_reboot_required{node="k3s-prod-2"}' + values: '0x600' + - series: 'kured_reboot_required{node="k3s-prod-3"}' + values: '0x600' + - series: 'kube_node_info{node="k3s-prod-1"}' + values: '1x600' + - series: 'kube_node_info{node="k3s-prod-2"}' + values: '1x600' + - series: 'kube_node_info{node="k3s-prod-3"}' + values: '1x600' + promql_expr_test: + # Firing while the reboot is still outstanding. + - expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}' + eval_time: 39h + exp_samples: + - labels: 'ALERTS{alertname="KuredRebootPending", alertstate="firing", node="k3s-prod-1", severity="warning"}' + value: 1 + # Cleared once the sentinel is gone. If this assertion ever fails, the rule has become + # self-sustaining and the operator will start ignoring it. + - expr: 'ALERTS{alertname=~"Kured.*"}' + eval_time: 45h + exp_samples: [] diff --git a/platform/kured/alerts.yaml b/platform/kured/alerts.yaml new file mode 100644 index 0000000..5d8015d --- /dev/null +++ b/platform/kured/alerts.yaml @@ -0,0 +1,124 @@ +--- +# Alert rules for the kured reboot automation. +# +# WHY THESE THREE, AND WHY THEY ARE NOT OPTIONAL. Reboot automation's characteristic failure is +# SILENCE, not noise. Every way this component can break leaves a cluster that looks perfectly +# healthy: a sentinel nothing writes, a Service the chart did not create, a NetworkPolicy that drops +# the scrape, a taint that excludes a node, a drain wedged on a PodDisruptionBudget, a cluster lock +# held forever by a node that hung on warm reboot. In none of those cases does anything turn red on +# its own. kured itself stays 1/1 Running throughout. +# +# kured EXPOSES EXACTLY ONE METRIC. Read from cmd/kured/main.go:92-96 and :113 on 2026-10-17 — the +# only `prometheus.MustRegister` in the tree is a GaugeVec `Subsystem: "kured"`, `Name: +# "reboot_required"`, labelled `{node}`. So the full vocabulary available is +# `kured_reboot_required{node="..."}`, 1 when the sentinel exists and 0 when it does not. Each rule +# below is built from that one series plus `kube_node_info` (kube-state-metrics), and nothing else. +# There is deliberately no rule over an invented metric name: this repo has shipped a +# structurally-inert alert before (VeleroNoRecentBackup over an absent series, which is why +# platform/monitoring/tests/run.sh exists), and an alert that cannot fire is worse than no alert +# because it also removes the suspicion that would have caught it. +# +# THESE RULES LIVE HERE, NOT IN platform/monitoring/alerts.yaml, for two reasons. (1) The unit-test +# harness treats a `.promtool-test.yaml` COMPANION as covering just its sibling, whereas a +# `tests/*.test.yaml` SUITE covers every PrometheusRule in its directory — so shipping them beside +# the component keeps the test obligation local and legible (the platform/cnpg precedent). (2) +# platform/monitoring/alerts.yaml is concurrently edited by other open branches; adding 3 rules to +# its tail would create a merge conflict for no benefit. +apiVersion: monitoring.coreos.com/v1 +kind: PrometheusRule +metadata: + name: kured + namespace: kured + labels: + app.kubernetes.io/name: kured +spec: + groups: + - name: platform-health.node-reboots + rules: + # ── 1. IS THE MECHANISM ALIVE AT ALL? ──────────────────────────────────────────────── + - alert: KuredNotReporting + # `absent()` yields a series only when the ENTIRE vector is empty, so this fires when no + # node at all is publishing — kured gone, or its Service/ServiceMonitor/NetworkPolicy + # regressed so nothing is scraped. It stays quiet while a single node is mid-reboot + # (the other two still report), which is what makes it specific to a wiring failure + # rather than to a reboot in progress. + # + # THIS IS THE RULE THAT GUARDS THE OTHER TWO. Both of them read `kured_reboot_required`, + # so if that series stops existing they go quiet and look like good news. This one turns + # the absence itself into the signal. The specific regression it is aimed at: the kured + # chart defaults `service.create` and `metrics.create` to FALSE, and Longhorn has already + # demonstrated on this cluster (see the companion note in + # conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh) that a + # NetworkPolicy can let targets be DISCOVERED while every scrape times out and the TSDB + # stays empty. + expr: absent(kured_reboot_required) + # 30m absorbs a HelmRelease upgrade rolling the DaemonSet and a Prometheus restart, + # without tolerating a real wiring break for more than one quiet night's worth of time. + for: 30m + labels: + severity: warning + annotations: + summary: kured is publishing no reboot_required series on any node + description: >- + No node is reporting kured_reboot_required, so the reboot automation is either not + running or not being scraped. While this is true, OS reboots are not happening and + the two alerts that would say so (KuredRebootPending, KuredNodeCoverageGap) are + themselves inert. Nodes will silently accumulate pending kernel upgrades. + runbook_url: "https://${BASE_DOMAIN}/runbooks/node-reboot-automation" + + # ── 2. IS IT ALIVE ON *EVERY* NODE? ────────────────────────────────────────────────── + - alert: KuredNodeCoverageGap + # A per-node agent that silently covers only some nodes is the subtler half of the + # liveness question, and `absent()` cannot see it. kured publishes one series per node it + # runs on; kube-state-metrics publishes one `kube_node_info` per node that exists + # (measured 2026-10-17: 3, matching the 3 nodes). A shortfall means at least one node has + # no reboot automation — most likely a new or re-provisioned node carrying a taint this + # DaemonSet does not tolerate (it declares none, because no prod node is tainted today). + # That node would then never reboot, with nothing else to indicate it. + expr: count(kured_reboot_required) < count(kube_node_info) + # 2h, chosen to sit comfortably ABOVE the duration of a healthy reboot. A node being + # rebooted stops publishing for a few minutes and legitimately creates a shortfall; a + # node that is still missing after two hours is either genuinely uncovered or has hung on + # reboot, and both of those want a human. + for: 2h + labels: + severity: warning + annotations: + summary: kured is not running on every node + description: >- + Fewer nodes are reporting kured_reboot_required than exist in the cluster, so at + least one node has no reboot automation and will never be rebooted by it. Most + likely a node carries a taint this DaemonSet does not tolerate, or has hung during + a reboot. + runbook_url: "https://${BASE_DOMAIN}/runbooks/node-reboot-automation" + + # ── 3. IS A REBOOT STUCK? ──────────────────────────────────────────────────────────── + - alert: KuredRebootPending + # A node has wanted a reboot for longer than a full maintenance window plus margin, so + # the reboot was OFFERED and did not happen. The causes are all real and all invisible + # otherwise: the Prometheus alert gate is blocking on a degraded Longhorn volume that is + # not recovering; the drain hit `drainTimeout` against a 0-disruption PDB; another node + # holds the cluster lock and never released it (`lockTtl: 0` means a lock held by a hung + # node is held FOREVER — that is deliberate, see helmrelease.yaml, and this is the alert + # that makes the resulting stall visible). + expr: max by (node) (kured_reboot_required) == 1 + # 36h, and the window being DAILY is what makes this number meaningful — see the window + # block in helmrelease.yaml. With a 03:00-06:00 window every day, a pending reboot waits + # at most ~24h legitimately, so 36h means at least one full window elapsed without the + # reboot completing. On a weekly window this threshold would have to exceed 7 days, which + # is longer than this cluster's 7d Prometheus retention and is reset by any Prometheus + # restart past the 1h for-outage-tolerance — i.e. unreliable in the one direction that + # matters. Note this alert can still be LATE for that reason, but it cannot be spurious. + for: 36h + labels: + severity: warning + annotations: + summary: Node {{ $labels.node }} has needed a reboot for over 36h + description: >- + kured reports a pending reboot on {{ $labels.node }} that has survived at least one + full 03:00-06:00 maintenance window. The reboot is being blocked, not merely + waiting: likely a degraded Longhorn volume holding the Prometheus alert gate, a + drain timing out against a PodDisruptionBudget, or a stale cluster lock held by a + node that did not come back. The kernel or security update that needed the reboot is + installed but NOT running. + runbook_url: "https://${BASE_DOMAIN}/runbooks/node-reboot-automation" diff --git a/platform/kured/helmrelease.yaml b/platform/kured/helmrelease.yaml new file mode 100644 index 0000000..425a26b --- /dev/null +++ b/platform/kured/helmrelease.yaml @@ -0,0 +1,289 @@ +--- +# kured (Kubernetes Reboot Daemon) — coordinated, one-node-at-a-time reboots of the three prod +# control-plane nodes. +# +# ═══════════════════════════════════════════════════════════════════════════════════════════════ +# READ THIS FIRST: kured DOES NOT DECIDE *WHETHER* TO REBOOT. IT ONLY DECIDES *WHEN* AND *HOW*. +# ═══════════════════════════════════════════════════════════════════════════════════════════════ +# kured reboots a node if and only if the file `/var/run/reboot-required` exists on it. On these +# nodes exactly one thing creates that file: a deliberate operator run of +# `playbooks/node-baseline.yml --tags kernel` (which advances the backports kernel). The +# unattended-upgrades role this PR also adds installs SECURITY updates only and BLACKLISTS +# `linux-image-*`/`linux-headers-*`, so it can never produce a reboot-required sentinel on its own — +# see infra/ansible/roles/unattended_upgrades/defaults/main.yml for that argument in full. +# +# That is the whole answer to this repo's two standing objections to reboot automation: +# +# * infra/ansible/roles/node_baseline/tasks/main.yml:61-67 — "IT DOES NOT REBOOT." k3s-prod-2 hung +# on a warm reboot on 2026-09-03 (powered, off the LAN ~38 minutes, physical power button +# required), so rebooting was made "a SERIALIZED, SUPERVISED operator step ... and a role cannot +# enforce that." STILL TRUE, and nothing here changes it: node_baseline continues not to reboot. +# * docs/plans/cluster-upgrade-plan.md:291-299 — system-upgrade-controller was rejected because it +# "automates away the human gate between nodes", and the operator's hard requirement is an +# explicit PASS/STOP verification BETWEEN every node. +# +# THE HUMAN GATE IS PRESERVED, AND IT IS THE SENTINEL. The operator decides whether a kernel moves +# and on which hosts (`--limit k3s-prod-1`), which is what writes the sentinel on those hosts and +# only those hosts. kured then performs the cordon/drain/reboot/uncordon that the operator would +# otherwise have typed by hand — the error-prone mechanical half, including the ordering rule +# node_baseline's own message has to shout ("NEVER node 2 first"). Run the kernel play one host at a +# time and the operator keeps per-node PASS/STOP control exactly as before; run it on all three and +# kured still does them strictly one at a time, just in an order it chooses. Both are safe for +# quorum; only the first preserves the ordering preference. See runbooks/node-reboot-automation.md. +# +# cluster-upgrade-plan.md:329-330 and :545-548 name the condition under which its own rejection +# reverses — "SUC becomes viable once the cluster is on fast disks ... where drain-driven rebuild is +# cheap" — and prod is that cluster. This component takes that clause up, but NOT its drain +# assumption: see the Longhorn section below, where drain-driven rebuild is measurably NOT free even +# on SSD, and is gated on explicitly rather than waved through. +# +# WHAT IS STILL UNMITIGATED, STATED PLAINLY: kured cannot press a power button. If a node hangs on +# warm reboot the way k3s-prod-2 did in September, kured will not recover it. The blast radius is +# bounded by design rather than by luck — `concurrency: 1` plus `lockTtl: 0` means the hung node +# keeps the cluster lock forever, so NO second node is ever rebooted behind it; etcd keeps 2/3 quorum +# and the cluster stays up, degraded, until a human attends. The KuredRebootPending alert in +# alerts.yaml is what tells them. That is strictly better than the previous automation story (none) +# but it is NOT the same as "safe to ignore at 3am on a Sunday". +apiVersion: helm.toolkit.fluxcd.io/v2 +kind: HelmRelease +metadata: + name: kured + namespace: kured +spec: + interval: 30m + chart: + spec: + chart: kured + # Chart 6.1.0 -> appVersion 1.23.0 (published 2026-06-30; newest of the 80 entries in + # https://kubereboot.github.io/charts/index.yaml, read 2026-10-17). + # + # THE `<6.2.0` CEILING IS A DELIBERATE CHOICE HERE, NOT THE REFLEX IT IS ELSEWHERE. This PR's + # own audit found that 9 of 12 HelmReleases in this repo pin a minor-locking range, and that + # this is a MAIN CAUSE of pin rot: Renovate can then only ever offer patches inside a line it + # has been told not to leave (kube-prometheus-stack is stuck 4 chart majors back that way). + # For a component whose job is to reboot control-plane nodes, that ceiling is worth paying on + # purpose: a kured MINOR may change drain or lock semantics, and this is the one component + # where "a human looked at the changelog" should be a precondition, not a nice-to-have. + # Renovate will raise the minor as a non-automerged PR (see the kured packageRule in + # renovate.json), so the ceiling is a review prompt, not a wall. + version: ">=6.1.0 <6.2.0" + sourceRef: + kind: HelmRepository + name: kured + namespace: kured + install: + remediation: + retries: 3 + upgrade: + remediation: + retries: 3 + values: + # ── IDENTITY / PLACEMENT ─────────────────────────────────────────────────────────────────── + # No tolerations. Measured 2026-10-17: all three prod nodes carry `spec.taints` EMPTY (k3s does + # not taint its servers), so a DaemonSet reaches every node untainted. If a future node IS + # tainted, kured silently stops covering it and that node then never reboots — invisible, + # because the other two keep reporting. KuredNodeCoverageGap in alerts.yaml is the detector: it + # compares kured's per-node series count against the node count rather than merely asking + # whether kured exists. + priorityClassName: system-node-critical + # The chart's defaults are kept: `hostPID: true` and `containerSecurityContext.privileged: true` + # with `readOnlyRootFilesystem: true`. Both are load-bearing and neither is reducible: + # `rebootMethod: command` runs the reboot by nsenter'ing /proc/1/ns/mnt, which needs the host PID + # namespace to see PID 1 and the privilege to enter its mount namespace. The documented + # lower-privilege alternative (privileged:false + CAP_KILL) applies ONLY to + # `rebootMethod: signal`, which signals a host-side kured-reboot.service that does not exist on + # these nodes. Choosing `signal` would mean shipping and maintaining that unit via Ansible for a + # capability reduction on a DaemonSet that is already cluster-admin-adjacent by function; it is a + # reasonable future hardening step and is explicitly NOT taken here. + resources: + requests: + cpu: 10m + memory: 32Mi + limits: + # No CPU limit on purpose (the repo's node-pressure work treats CPU limits on tiny system + # agents as a throttling hazard). Memory is capped: kured holds no state beyond one + # annotation and a gauge, and 128Mi is ~4x its idle RSS upstream reports. + memory: 128Mi + + configuration: + # ── THE QUORUM CONSTRAINT: TWO NODES MUST NEVER BE DOWN AT ONCE ────────────────────────── + # All three prod nodes are control-plane + etcd (measured 2026-10-17: every node reports + # `control-plane,etcd`). etcd quorum is 2 of 3, so one node down is degraded-but-up and two + # nodes down is a LOST CLUSTER that does not self-heal. Everything in this block exists to + # make the second case structurally unreachable rather than merely unlikely. + # + # concurrency: 1 — kured's cluster-wide lock is a single annotation + # (`weave.works/kured-node-lock`) on kured's OWN DaemonSet object, and a node reboots only + # while it holds that lock. One lock + concurrency 1 = one reboot, enforced by the API server's + # optimistic concurrency rather than by timing. This is the chart default; it is written out + # explicitly because it is the single most important value in this file. + concurrency: 1 + # lockTtl: 0 (DISABLED) — THIS IS A SAFETY DECISION, NOT A DEFAULT WE FORGOT TO TUNE. + # A TTL makes the lock EXPIRE, which exists for the case "the lock holder died and we want + # reboots to continue". On a 3-node etcd cluster that is precisely the catastrophe: if node A + # has hung on a warm reboot (which has happened here — see the header) and its lock expires, + # node B becomes eligible and takes the cluster from 2/3 to 1/3. We would rather the reboot + # cycle STOP FOREVER and page a human than have it step over a corpse. The cost is that a + # genuinely stale lock needs manual clearing; runbooks/node-reboot-automation.md has the + # procedure, and KuredRebootPending is the alert that surfaces it. + lockTtl: 0 + # lockReleaseDelay: 30m — hold the lock for 30 minutes AFTER this node is back and uncordoned, + # before any other node may take it. See the Longhorn section: this is the STRUCTURAL half of + # the inter-node gate, deliberately not dependent on an alert's `for:` timer having latched. + lockReleaseDelay: 30m + + # ── THE MAINTENANCE WINDOW: 03:00-06:00 EVERY DAY, America/New_York ────────────────────── + # The timezone and the 03:00-06:00 band match renovate.json's existing + # `"schedule": ["after 3am and before 6am"]` / `"timezone": "America/New_York"`, so every + # unattended change in this repo lands in the same quiet 3-hour window. + # + # DAILY, NOT WEEKLY, AND THE REASON IS THE ALERT — not impatience. A weekly (`- sun`) window + # was the first draft and it breaks the only alert that can tell you this mechanism has + # stalled. With a Sunday-only window, a kernel upgraded on a Monday legitimately waits six + # days, so "a reboot has been pending too long" cannot be distinguished from "it is Tuesday" + # until the threshold exceeds a full 7-day cycle. That forces `for: 8d`+, which is longer than + # this cluster's Prometheus retention (MONITORING_PROMETHEUS_RETENTION is 7d) and is reset by + # any Prometheus restart beyond the 1h default for-outage-tolerance — i.e. the alert would be + # unreliable in exactly the direction that matters. A daily window caps the legitimate wait at + # under 24h, which makes KuredRebootPending's `for: 36h` both robust and meaningful: 36 hours + # with a sentinel still set means at least one full window passed and SOMETHING BLOCKED IT. + # + # Daily does NOT weaken the quorum guarantee — that is `concurrency: 1` plus the lock, not the + # calendar. It does mean a 3-node kernel rollout can finish in one to three nights instead of + # three weeks, with each node still gated on the previous one's Longhorn rebuilds (below). + rebootDays: + - su + - mo + - tu + - we + - th + - fr + - sa + startTime: "03:00" + endTime: "06:00" + timeZone: America/New_York + # How often to re-evaluate (sentinel + window + alerts + lock). 1h is the chart default; at a + # 3h window that is three evaluation opportunities per node per night. + period: 1h + + # ── THE SENTINEL ───────────────────────────────────────────────────────────────────────── + # Written ONLY by a deliberate operator kernel upgrade (see the header). `update-notifier-common` + # — the package that owns /etc/kernel/postinst.d and actually creates this file on Debian — was + # measured ABSENT on all three nodes on 2026-10-17, which is why the unattended_upgrades role + # installs it. Without that package nothing ever writes this path, kured correctly never + # reboots, every probe looks healthy, and the entire mechanism is a silent no-op. That failure + # mode is the reason KuredNotReporting exists. + # + # THE PATH kured LOGS IS NOT THIS PATH, and that trips people debugging it. The chart + # defaults `useRebootSentinelHostPath: true` (kept), which bind-mounts the DIRNAME of this + # value onto /sentinel read-only and renders the flag as + # `--reboot-sentinel=/sentinel/reboot-required` (daemonset.yaml:134). So the container watches + # /sentinel/reboot-required and the host file is /var/run/reboot-required. Same file, two + # names. Set this value as the HOST path, as below. + rebootSentinel: /var/run/reboot-required + + # ── THE LONGHORN GATE: WHY POD READINESS IS NOT A SUFFICIENT INTER-NODE SIGNAL ──────────── + # Measured on prod 2026-10-17, and this is the finding that shaped this file: + # * 25 Longhorn volumes, ALL `numberOfReplicas: 3`, all healthy, with exactly 25 replicas on + # each of the 3 nodes. So there is no single-replica volume to lose today. + # * BUT `replica-soft-anti-affinity: false` (replicas may not share a node) and + # `disable-scheduling-on-cordoned-node: true`. With 3 nodes and 3 replicas, cordoning ONE + # node leaves nowhere to place a replacement — so every drain takes ALL 25 volumes to + # degraded 2/3 for the whole reboot. + # * Recovery is slow and serialized: `replica-replenishment-wait-interval: 600` (10 minutes + # before a rebuild even starts) and `concurrent-replica-rebuild-per-node-limit: 1`. + # Therefore: if the next node is rebooted before rebuilds finish, those volumes go to 1/3, and + # the drain after that genuinely CAN hit Longhorn's `node-drain-policy: + # block-if-contains-last-replica` and block forever. The inter-node gate must be VOLUME + # ROBUSTNESS, not "are the pods Ready again". + # + # It is gated twice, on purpose, because each half covers the other's blind spot: + # 1. lockReleaseDelay: 30m (above) — unconditional. Covers the 600s replenishment wait with + # margin and does not depend on any alert having fired. + # 2. The Prometheus alert gate below — semantic. Covers a rebuild that takes LONGER than 30 + # minutes (25 serialized rebuilds on one node can), which a fixed delay cannot. + prometheusUrl: http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090 + # Only consider alerts that are FIRING, not merely pending. A `pending` alert is a 10-minute + # guess; blocking a weekly reboot on one would make the gate fire on noise. + alertFiringOnly: true + # ── alertFilterMatchOnly: true IS LOAD-BEARING. DO NOT DROP IT. ────────────────────────── + # kured's two alert modes are near-opposites and the flag name does not advertise it: + # * false (the chart default): alertFilterRegexp is an IGNORE list — kured blocks on ANY + # firing alert except those matching. + # * true: alertFilterRegexp is an ALLOW list — kured blocks ONLY on alerts that match. + # We need `true`, because this cluster always has something firing. Measured via the Prometheus + # API on 2026-10-17, these five were firing with no reboot pending and no end in sight: + # CNPGPrimariesCoLocated, KubeDeploymentReplicasMismatch, KubeDeploymentRolloutStuck, + # KubePodNotReady, OAuth2ProxyGateUnobserved (the last three all trace to the pm-agent pod, + # ImagePullBackOff on a placeholder digest for 12 days). Under the default `false`, kured would + # have been blocked by those on day one and would never have rebooted anything — a silent + # no-op indistinguishable, from outside, from "no reboots were needed". + alertFilterMatchOnly: true + # The ALLOW list: the conditions under which rebooting ANOTHER node risks data or quorum. + # Anchored and explicit — a prefix match here would silently re-admit the chronic five above. + # LonghornVolumeDegraded — the inter-node gate (see above). `longhorn_volume_robustness + # {state="degraded"} == 1`, for: 10m, in platform/monitoring/alerts.yaml. + # LonghornVolumeFaulted — data already at risk; never reboot into this. + # KubeNodeNotReady — a node is already down. Rebooting a second one loses quorum. This + # is the single most important entry for the etcd constraint. + # NodeExporterDown — a node we cannot see the health of. Treated as "assume the worst". + # DELIBERATELY NOT INCLUDED: NodeEtcdRequestsSlow and the NodeIOPressure* family. They are + # exactly the conditions you would want to pause for, but prod-1 has a documented history of + # etcd WAL I/O starvation, so including them risks a permanent wedge that looks like safety. + # Revisit once those are quiet for a sustained period; KuredRebootPending makes a wedge visible + # either way. + alertFilterRegexp: ^(LonghornVolumeDegraded|LonghornVolumeFaulted|KubeNodeNotReady|NodeExporterDown)$ + + # ── THE DRAIN ──────────────────────────────────────────────────────────────────────────── + # NOTE ON FLAGS THAT DO NOT EXIST: there is no `--ignore-daemonsets`, no + # `--delete-emptydir-data` and no `--force` in kured 1.23.0. Read from + # cmd/kured/main.go:438-449 on 2026-10-17, kured HARDCODES + # `IgnoreAllDaemonSets: true`, `DeleteEmptyDirData: true` and `Force: true` in the drain + # helper. Passing them as extraArgs would CrashLoop the DaemonSet on an unknown flag. This + # matters because all three are genuinely REQUIRED here and it is easy to assume they must be + # configured: 12 DaemonSets and 135 emptyDir-carrying pods were measured on 2026-10-17, and a + # drain without those three settings refuses to start. + # + # drainTimeout: 15m — the chart/kured default is 0, meaning INFINITE, and infinite is the wrong + # answer here. Measured 2026-10-17, six PodDisruptionBudgets currently allow ZERO voluntary + # disruptions, and a plain drain loops on them indefinitely: + # * 3x longhorn-system/instance-manager-* — one per node, `minAvailable: 1` against a + # selector pinned to a single node's single pod, so disruptionsAllowed is STRUCTURALLY 0 + # forever. This is the mechanism by which Longhorn's node-drain-policy blocks: the + # longhorn-manager deletes the node's instance-manager PDB only once it judges the drain + # safe. The drain is SUPPOSED to wait on this one. + # * 3x CNPG `-primary` (keycloak-db, nextcloud-db, semaphore-db) — `minAvailable: 1` + # against a selector matching exactly one pod (the primary), so again structurally 0 until + # the CNPG operator switches the primary away. All three primaries were on k3s-prod-1. + # With a timeout, a drain that is genuinely stuck ABORTS, the lock is released, and kured + # retries next period instead of holding the cluster lock forever on a wedged node. 15m is + # chosen to be longer than a CNPG switchover and longer than the `unifi` VM's eviction (below) + # usually needs, while still being well inside the 3h window. + drainTimeout: 15m + # Taint a node PreferNoSchedule while its reboot is pending, so the scheduler stops placing new + # pods on a node that is about to go away. With concurrency 1 this is a small win (there is no + # second rebooting node to shed pods onto) but it is free and it shortens the drain. + preferNoScheduleTaint: weave.works/kured-node-reboot + # Record reboot state on the Node objects (`weave.works/kured-reboot-in-progress`, + # `weave.works/kured-most-recent-reboot-needed`). Off by default. Turned on because the only + # other observability kured offers is ONE gauge, and during an incident `kubectl get node -o + # yaml` is the thing an operator can reach without Prometheus. + annotateNodes: true + logFormat: json + + # ── METRICS: BOTH OF THESE MUST BE true OR THE ALERTS IN alerts.yaml CANNOT FIRE ─────────── + # The chart ships `service.create: false` AND `metrics.create: false`. Leaving either at its + # default means no Service, or a ServiceMonitor with nothing to select, so `kured_reboot_required` + # is never scraped — and a PrometheusRule over a series that no target produces is INERT: quiet + # forever, indistinguishable from healthy. This repo has shipped that exact bug twice + # (VeleroNoRecentBackup over an absent series; the inert gotk_* rules in + # runbooks/image-automation.md), which is why platform/monitoring/tests/run.sh exists at all. + # KuredNotReporting (alerts.yaml) is the standing detector for this pair regressing. + # Prometheus selects ServiceMonitors cluster-wide: measured 2026-10-17, `serviceMonitorSelector` + # and `serviceMonitorNamespaceSelector` are both `{}`, so no label plumbing is needed here. + service: + create: true + port: 8080 + metrics: + create: true + interval: 60s diff --git a/platform/kured/helmrepository.yaml b/platform/kured/helmrepository.yaml new file mode 100644 index 0000000..2a29d3e --- /dev/null +++ b/platform/kured/helmrepository.yaml @@ -0,0 +1,12 @@ +--- +# HelmRepository for the official kured chart (github.com/kubereboot/charts). Renovate maintains the +# pinned chart version in helmrelease.yaml via its `flux` manager, whose managerFilePatterns already +# cover `platform/**` (renovate.json). Same Flux source pattern as platform/headlamp. +apiVersion: source.toolkit.fluxcd.io/v1 +kind: HelmRepository +metadata: + name: kured + namespace: kured +spec: + interval: 1h + url: https://kubereboot.github.io/charts diff --git a/platform/kured/kustomization.yaml b/platform/kured/kustomization.yaml new file mode 100644 index 0000000..bbffad9 --- /dev/null +++ b/platform/kured/kustomization.yaml @@ -0,0 +1,36 @@ +--- +# platform/kured — the Kubernetes Reboot Daemon. +# +# Rendered by its OWN Flux Kustomization (clusters/base/kured.yaml, dependsOn platform), NOT by a +# line in platform/kustomization.yaml. The reasoning is the same one the Cilium split-out wrote down +# and intel-gpu-plugin reused: inside the shared `wait: true` platform tier, this component's health +# would be indistinguishable from eleven others, the only emergency valve would be suspending the +# whole tier, and that tier's 5m timeout is wrong for a 3-node DaemonSet rollout. It matters more +# than usual here, because `platform` gates platform-config which gates every apps wave — so a wedged +# reboot daemon placed there could stall every application in the cluster. A component whose entire +# purpose is to make maintenance safer must not be able to cause an outage. +# +# `namespace: kured` applies to every resource below, so none of them repeats it. Keep it: the +# CiliumNetworkPolicy in networkpolicy.yaml is a CR whose namespace kustomize would otherwise leave +# to chance. +# +# NO SECRET OF ANY KIND. kured authenticates to the API server with its own ServiceAccount token and +# talks to nothing off-cluster — no registry credential, no webhook URL, no notification token. So +# neither clusters/dev nor clusters/prod adds a SOPS decryption patch for this tier (same as +# homepage/runbooks/intel-gpu-plugin). If Slack or shoutrrr notifications are ever wanted, the chart +# takes them via `configuration.notifyUrl`, and THAT would need a SOPS-encrypted Secret plus a +# decryption patch — do not inline a webhook URL here. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +namespace: kured +resources: + - namespace.yaml + - helmrepository.yaml + - helmrelease.yaml # chart installs the DaemonSet / RBAC / SA / Service / ServiceMonitor + # default-deny baseline + the Prometheus scrape ingress + the apiserver/Prometheus egress. + # The scrape ingress is NOT optional decoration: without it the ServiceMonitor's targets are + # discovered and every scrape times out, leaving the TSDB empty and all three rules in alerts.yaml + # permanently quiet. See the header in that file for the measured Longhorn precedent. + - networkpolicy.yaml + # Three rules over kured's single metric, with a companion promtool unit test beside them. + - alerts.yaml diff --git a/platform/kured/namespace.yaml b/platform/kured/namespace.yaml new file mode 100644 index 0000000..c9b2c24 --- /dev/null +++ b/platform/kured/namespace.yaml @@ -0,0 +1,34 @@ +--- +# kured — namespace for the Kubernetes Reboot Daemon DaemonSet. +# +# PLATFORM INFRA, NOT AN APPLICATION NAMESPACE, and here that distinction is the difference between +# a working component and an admission rejection. kured runs `hostPID: true` with a `privileged: true` +# container (it `nsenter`s /proc/1/ns/mnt to run `/bin/systemctl reboot` on the host). Both are +# forbidden by platform/kyverno/policies/restrict-host-namespaces.yaml — but that policy, like every +# other policy in this repo, is scoped by +# `namespaceSelector: matchLabels: homelab.keithjackson.dev/app-namespace: "true"` +# and its own header states the carve-out: "platform components that do [need these] live in platform +# namespaces that do NOT carry the app-namespace label, so they are out of scope." +# Verified by reading the policies on this branch, not assumed: +# * restrict-host-namespaces.yaml rule `disallow-host-namespaces` (hostPID) +# * restrict-host-namespaces.yaml rule `disallow-privileged-containers` (privileged) +# +# We set the label EXPLICITLY to "false" rather than leaving it unset, following +# platform/intel-gpu-plugin/namespace.yaml and platform/vpn-egress: an unset label and a deliberate +# "false" are identical to the admission engine but not to a reviewer, and this component's +# correctness depends entirely on the answer. +# +# NO PodSecurity labels. There is no cluster-wide PSA baseline in this repo — `pod-security.*` +# appears only where a namespace opts IN (conformance/incluster, platform/kyverno/admission, +# platform/pm-agent), all at `enforce: restricted`. Adding them here would reject this DaemonSet for +# the same reasons Kyverno would. Leaving them off is the convention for a privileged platform +# namespace (platform/vpn-ingress/namespace.yaml sets none either). +# +# NO `gateway-access` label. kured serves no HTTP to users and attaches no HTTPRoute; its only +# listener is the :8080 metrics endpoint, reached in-cluster by Prometheus. See networkpolicy.yaml. +apiVersion: v1 +kind: Namespace +metadata: + name: kured + labels: + homelab.keithjackson.dev/app-namespace: "false" diff --git a/platform/kured/networkpolicy.yaml b/platform/kured/networkpolicy.yaml new file mode 100644 index 0000000..6961c74 --- /dev/null +++ b/platform/kured/networkpolicy.yaml @@ -0,0 +1,125 @@ +--- +# Default-deny baseline (R-NET-01): deny all ingress + egress, allow only DNS. `kured` is a PLATFORM +# namespace, so Kyverno's generator — scoped to `app-namespace=true` — does not produce this; we ship +# it explicitly, as crafty/headlamp/homepage/runbooks all do. +# +# THESE POLICIES DO APPLY TO kured, UNLIKE platform/vpn-ingress'. That component's identical-looking +# baseline is honest-but-inert because its DaemonSet runs `hostNetwork: true` and therefore has no +# Cilium endpoint for any policy to select. kured's chart defaults to `hostNetwork: false` and we +# keep that default (it needs the host PID namespace to reboot, not the host network stack), so it +# has a normal pod identity and every rule here is enforced. +# +# Cilium enforces the UNION of these policies (allow if ANY policy allows), so the CiliumNetworkPolicy +# below adds the kube-apiserver path on top of this deny-all floor. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: default-deny + namespace: kured + labels: + homelab.keithjackson.dev/baseline: default-deny +spec: + podSelector: {} + policyTypes: + - Ingress + - Egress + ingress: [] + egress: + - to: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: kube-system + ports: + - protocol: UDP + port: 53 + - protocol: TCP + port: 53 +--- +# ═══════════════════════════════════════════════════════════════════════════════════════════════ +# PROMETHEUS MUST REACH :8080, OR EVERY RULE IN alerts.yaml IS INERT. +# ═══════════════════════════════════════════════════════════════════════════════════════════════ +# This is not a theoretical hazard in this repo, it is a measured one. From the Longhorn entry in +# conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh: with the ServiceMonitor +# applied and no NetworkPolicy allowance, all three targets were DISCOVERED and every scrape failed +# with "context deadline exceeded" after 10.0s — "a drop, not a refusal" — and the TSDB stayed +# empty. Target discovery succeeding is therefore NOT evidence the scrape works, and the alerts +# would have been quiet rather than broken. KuredNotReporting is the standing detector for exactly +# this regression, and it can only work if this policy is correct. +# +# THE WHOLE `monitoring` NAMESPACE, NOT THE PROMETHEUS POD LABELS — following the same choice +# longhorn/networkpolicy-metrics.yaml made and for the same reason: the prometheus-operator owns +# those pod labels and rewrites them on upgrade (measured 2026-10-17, the pod carries +# `app.kubernetes.io/name: prometheus` plus five operator-managed labels including a +# controller-revision-hash), so selecting on them makes the scrape silently fail on a chart bump. +# The namespace holds only the monitoring stack. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: kured-allow-prometheus-scrape + namespace: kured + labels: + app.kubernetes.io/name: kured +spec: + podSelector: + matchLabels: + # Set by the chart's `kured.matchLabels` helper (_helpers.tpl:70-71) with release name `kured`. + app.kubernetes.io/name: kured + policyTypes: + - Ingress + ingress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: monitoring + ports: + - protocol: TCP + # The container port the chart names `metrics` (daemonset.yaml:211-213), from + # `configuration.metricsPort`, which we leave at the chart default of 8080 and mirror in + # the HelmRelease's `service.port`. + port: 8080 +--- +# Egress allows. kured talks to exactly two things and nothing else; both are required and the +# component is useless without either. +# +# 1. THE KUBE-APISERVER — the lock annotation on its own DaemonSet (`weave.works/kured-node-lock`), +# cordon/uncordon (`nodes` patch), the node annotations, listing pods, and creating +# `pods/eviction` objects during the drain. Reached via the `kube-apiserver` Cilium entity +# rather than an IP: on this cluster the API is fronted by a kube-vip VIP that moves between +# nodes on every reboot, so any CIDR-based rule here would break during the exact operation +# this component exists to perform. Same construct as platform/headlamp/networkpolicy.yaml. +# +# 2. PROMETHEUS, on 9090 — the alert gate. The HelmRelease sets +# `prometheusUrl: http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090` +# and `alertFilterMatchOnly: true`, so before taking the cluster lock kured asks Prometheus +# whether any of LonghornVolumeDegraded / LonghornVolumeFaulted / KubeNodeNotReady / +# NodeExporterDown is firing. IF THIS EGRESS IS BLOCKED, kured cannot evaluate the gate. +# It treats the query error as "do not proceed" and skips the reboot — which is the correct +# fail-safe direction, but it fails SILENTLY and indefinitely: reboots simply stop happening +# and nothing goes red except, eventually, KuredRebootPending. Do not remove this rule +# thinking it is only for metrics; metrics ingress is the policy above, this is the gate. +# +# NOTHING ELSE. No egress to the internet (kured pulls no content at runtime), no DNS beyond the +# baseline above, no Longhorn or etcd access — kured reads Longhorn's health THROUGH Prometheus +# alerts, never from Longhorn's own API, which is why that allow is absent here. +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: kured-allow-egress + namespace: kured + labels: + app.kubernetes.io/name: kured +spec: + endpointSelector: + matchLabels: + app.kubernetes.io/name: kured + egress: + - toEntities: + - kube-apiserver + - toEndpoints: + - matchLabels: + io.kubernetes.pod.namespace: monitoring + app.kubernetes.io/name: prometheus + toPorts: + - ports: + - port: "9090" + protocol: TCP diff --git a/renovate.json b/renovate.json index ac9e243..4402e14 100644 --- a/renovate.json +++ b/renovate.json @@ -1,6 +1,6 @@ { "$schema": "https://docs.renovatebot.com/renovate-schema.json", - "_comment": "Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). Renovate owns HELM CHART + DEPENDENCY bumps (opens PRs; allowed classes auto-merge on green CI). Flux image-automation owns per-service container IMAGE TAGS (the $imagepolicy setters). The two do NOT overlap: this config DISABLES image-tag updates so Renovate never fights Flux over a running version. Self-hosted against the bootstrap Gitea; platform: gitea set at runtime by the operator's renovate runner.", + "_comment": "Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). AS OF 2026-10-17 RENOVATE OWNS BOTH HALVES: Helm chart/dependency bumps AND per-service container image tags. It opens PRs for everything; allowed low-risk classes auto-merge on green CI. The ADR-0009 split that gave image tags to Flux image-automation is formally closed — see .gitea/workflows/renovate.yaml for the three measured reasons (it pushes commits rather than opening PRs, so it bypasses the CLAUDE.md §1 gate; `main` is protected with user_can_push:false and the flux-system GitRepository has no secretRef, so it cannot write anyway; and enabling it would need an in-cluster git WRITE credential for the repo that defines the cluster). image-automation has been commented out at platform/kustomization.yaml:52 since it was authored and the cluster holds zero ImagePolicy/ImageRepository/ImageUpdateAutomation objects, so nothing is being torn down. CONSEQUENCE: the `docker` and `kubernetes` managers, previously disabled ONLY to avoid fighting that controller, are now ENABLED — without them every plain `image: repo/name:tag` line in this repo was maintained by nothing at all, which is how the oauth2-proxy auth sidecars reached 7 minors behind the oauth2-proxy running in the same cluster. RUNNER: .gitea/workflows/renovate.yaml (scheduled daily); platform/endpoint are set there, not here.", "extends": [ "config:recommended", ":dependencyDashboard", @@ -15,13 +15,24 @@ "labels": ["dependencies", "automated-update"], "ignoreUnstable": true, "respectLatest": true, - "enabledManagers": ["flux", "kustomize", "helmv3", "regex", "github-actions"], + "enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "regex", "github-actions"], "flux": { "managerFilePatterns": ["/(^|/)platform/.+\\.ya?ml$/", "/(^|/)apps/.+\\.ya?ml$/"] }, "kustomize": { "managerFilePatterns": ["/(^|/)kustomization\\.ya?ml$/"] }, + "_kubernetes_comment": "ADDED 2026-10-17. The `kubernetes` manager reads plain `image:` fields out of Kubernetes manifests, and it is NOT auto-detected — it only looks at files matched here, which is why this block is as broad as it is. Scope covers every place this repo writes a container image: platform components, the rendered app waves, the ServiceDefinition catalog that generates them, the runbooks web server and the pm-agent job template. `catalog/` matters as much as `apps/`: apps/wave-*/*.yaml are GENERATED by catalog/render.py, so a bump applied only to the rendered output is reverted by the next render and would also fail ci/render-golden.sh. Renovate will raise BOTH, and both must land together.", + "kubernetes": { + "managerFilePatterns": [ + "/(^|/)platform/.+\\.ya?ml$/", + "/(^|/)platform-config/.+\\.ya?ml$/", + "/(^|/)apps/.+\\.ya?ml$/", + "/(^|/)catalog/.+\\.ya?ml$/", + "/(^|/)runbooks/serving/.+\\.ya?ml$/", + "/(^|/)infra/agent/.+\\.ya?ml$/" + ] + }, "customManagers": [ { "customType": "regex", @@ -60,14 +71,29 @@ "matchFileNames": ["ci/versions.env", "infra/devcontainer/versions.env"], "automerge": false, "addLabels": ["ci-tooling"] + }, + { + "description": "ADDED 2026-10-17. NEVER AUTO-MERGE THE BLAST-RADIUS COMPONENTS, not even a patch. The default rule above auto-merges minor/patch/digest on a green gate, which is right for an app image: a bad Sonarr bump inconveniences one pod. It is wrong for these five, because the gate is a STATIC check — it renders manifests and lints them, it does not boot a node, drain a volume or form an etcd quorum (vars.CLUSTER_AVAILABLE is unset, so L2/L3 conformance does not run in CI at all). So for anything whose failure mode is the CLUSTER rather than a service, a green gate is not evidence and a human must read the changelog. kured is here because it reboots control-plane nodes; longhorn because a storage-engine bump migrates live volume data; cilium because it is the CNI and a bad bump is a total partition (and its version is DUAL-PINNED in infra/ansible/roles/cilium/defaults/main.yml, which Renovate does not see, so the two would silently desync); cloudnative-pg because it drives Postgres switchovers; kube-prometheus-stack because ci/versions.env asserts an exact promtool/Prometheus lockstep and a chart bump that moves Prometheus fails the gate until that file moves with it.", + "matchPackageNames": [ + "kured", + "longhorn", + "cilium", + "cloudnative-pg", + "kube-prometheus-stack" + ], + "automerge": false, + "addLabels": ["needs-review", "blast-radius"] } ], + "_unmanageable_comment": "WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A FLOATING tag has no version to compare, so Renovate skips it silently — it does not pin it and does not warn. As of 2026-10-17 that is: plexinc/pms-docker:latest, lscr.io/linuxserver/jackett:latest, pihole/pihole (NO TAG AT ALL), quay.io/keycloak/keycloak (no tag), nextcloud:31-apache and docker:27-dind. Those must be pinned BY HAND to a concrete version before any automation can maintain them, and until then they drift on every pull with nothing recording what is running. Separately, the HelmRelease minor-locking ranges (9 of 12, e.g. `>=88.0.0 <89.0.0` on kube-prometheus-stack) cap Renovate at patches inside a line it has been told not to leave; raising those ceilings is deliberate per-component work and is NOT part of enabling the bot. Both lists are in decisions/2026-10-17-automated-upgrades.md.", "ignorePaths": [ "**/*.example", "**/*.sops.yaml", "conformance/**", - "platform/image-automation/per-service/**" + "platform/image-automation/per-service/**", + "platform/kyverno/tests/**" ], - "docker": { "enabled": false }, - "kubernetes": { "enabled": false } + "_ignorePaths_comment": "platform/kyverno/tests/** ADDED 2026-10-17: those Kyverno policy fixtures pin `alpine:latest` and a `conformance-runner:latest`, which are test inputs whose whole point is to be a fixed shape, not software to keep current. `conformance/**` was already ignored for the same reason; this closes the matching hole that enabling the `kubernetes` manager would otherwise have opened.", + "_docker_comment": "The `docker` manager (Dockerfiles, docker-compose) and the `kubernetes` manager above were BOTH previously `enabled: false`, solely so Renovate would not fight Flux image-automation over a running tag. That controller was never enabled and never will be here (see _comment), so the only thing those two flags were achieving was leaving every plain `image:` tag in the repo unmanaged. `docker` is left out of enabledManagers because this repo contains no Dockerfile that ships to the cluster; `kubernetes` is now ON and is the one that matters.", + "docker": { "enabled": true } } diff --git a/runbooks/kustomization.yaml b/runbooks/kustomization.yaml index bbed9f1..c6deba9 100644 --- a/runbooks/kustomization.yaml +++ b/runbooks/kustomization.yaml @@ -92,6 +92,9 @@ configMapGenerator: - node-io=node-io.md - node-memory=node-memory.md - node-memory-swap=node-memory-swap.md + # Covers all three platform-health.node-reboots rules (KuredRebootPending, + # KuredNodeCoverageGap, KuredNotReporting) from platform/kured/alerts.yaml. + - node-reboot-automation=node-reboot-automation.md # Linked from OAuth2ProxyGateUnobserved, and the diagnostic procedure for any gated app # that "is up but does not work" — the gate answers an unauthenticated XHR with a 302, # not a 401, so the failure arrives as an HTTP 200 carrying the wrong body. diff --git a/runbooks/node-reboot-automation.md b/runbooks/node-reboot-automation.md new file mode 100644 index 0000000..74f7392 --- /dev/null +++ b/runbooks/node-reboot-automation.md @@ -0,0 +1,254 @@ +# Node reboot automation (kured) + +**Alerts:** `KuredRebootPending`, `KuredNodeCoverageGap`, `KuredNotReporting` +**File:** `platform/kured/alerts.yaml` +**Expressions:** `max by (node) (kured_reboot_required) == 1` / `count(kured_reboot_required) < count(kube_node_info)` / `absent(kured_reboot_required)` + +## What this means here + +These three alerts cover the OS reboot half of the automated upgrade process. The design is two +pieces that only work together: + +- **`roles/unattended_upgrades`** (Ansible, on the host) installs Debian **security** updates + nightly. It never upgrades the kernel, never restarts k3s, and never reboots. When something it + installed needs a reboot, it leaves `/var/run/reboot-required` behind. +- **kured** (`platform/kured`, a DaemonSet) watches that file. When it exists, kured takes a + **cluster-wide lock**, cordons the node, drains it, reboots it, and uncordons it — then holds the + lock for a further 30 minutes before any other node may go. + +**kured does not decide whether to reboot.** On these nodes the sentinel is written almost +exclusively by a deliberate operator kernel upgrade +(`playbooks/node-baseline.yml --tags kernel`), because the kernel is blacklisted from unattended +upgrades. So a pending reboot normally means *a human already asked for this*. + +Which alert you have tells you which half is wrong: + +| alert | meaning | +|---|---| +| `KuredRebootPending` | The mechanism is alive and a reboot was offered, but it did not happen for >36h. **Something is blocking it.** | +| `KuredNodeCoverageGap` | kured is not running on every node. The uncovered node will never be rebooted, silently. | +| `KuredNotReporting` | kured is publishing nothing at all. Reboots are not happening, **and the other two alerts are inert.** Treat this as the most serious of the three. | + +**All three nodes are control-plane + etcd, so quorum is 2 of 3.** One node down is degraded but +up; two is a lost cluster. Everything below is written to protect that. `concurrency: 1` plus +`lockTtl: 0` means a node that hangs on reboot keeps the lock **forever** and no second node is ever +rebooted behind it. That is deliberate: the cycle stops and pages a human rather than stepping over +a corpse. + +**A reboot is not seamless.** Most workloads here are single-replica, so each reboot cycle takes a +brief outage of whatever runs on that node — including `coredns` (1 replica), which means a +cluster-wide DNS gap while it reschedules. See "Expected outage per cycle" below. + +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +picks a current-context per invocation, and that default has already changed once. Do not run two +`dev.sh` invocations at once; they share one container and interfere. + +Mutating kubectl (`apply`, `patch`, `annotate`, `uncordon`, `rollout`) and `flux reconcile` are +operator actions, not agent actions. Where this runbook gives a mutating command it is for an +operator, and it says so. + +## First commands + +Start here, in order. The first question is always *which half is broken*. + + # 1. Is kured running at all, and on how many nodes? + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds,pod -o wide' + + # 2. Which node is asking for a reboot, and who holds the lock? + # The lock is an annotation on kured's OWN DaemonSet. An empty/absent value means no holder. + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds kured \ + -o jsonpath="{.metadata.annotations}"; echo' + + # 3. Is a node cordoned, and is a reboot in progress? + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes \ + -o custom-columns=NAME:.metadata.name,SCHEDULABLE:.spec.unschedulable,TAINTS:.spec.taints' + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes \ + -o jsonpath="{range .items[*]}{.metadata.name}{\" \"}{.metadata.annotations.weave\.works/kured-reboot-in-progress}{\"\n\"}{end}"' + + # 4. What is kured itself saying? It logs its decision every period (1h). + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured logs ds/kured --tail=80' + + # 5. Is the gate blocking it? These are the ONLY alerts kured blocks on. + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n longhorn-system \ + get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness' \ + | grep -v healthy + + # 6. Does the node actually want a reboot? (read-only; SSH as k3s, no sudo needed) + ssh -o BatchMode=yes 192.168.2.10 'ls -la /var/run/reboot-required; cat /var/run/reboot-required.pkgs 2>/dev/null' + +## Likely causes, most likely first + +### 1. The Longhorn gate is holding it (the common `KuredRebootPending` cause) + +kured queries Prometheus before taking the lock and refuses to proceed while +`LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `KubeNodeNotReady` or `NodeExporterDown` is +firing. This is working as designed and it is the **most likely** cause. + +Why it lingers: there are 3 nodes, every volume has 3 replicas, and +`replica-soft-anti-affinity: false` means replicas may not share a node. So cordoning one node +leaves nowhere to put a replacement and **every volume goes degraded 2/3** for the duration. +Recovery is slow on purpose: `replica-replenishment-wait-interval` is 600s before a rebuild even +starts, and `concurrent-replica-rebuild-per-node-limit` is 1, so ~25 volumes rebuild one at a time. + +Check progress, then wait: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n longhorn-system \ + get volumes.longhorn.io -o json' | python3 -c 'import json,sys,collections; \ + v=json.load(sys.stdin)["items"]; print(collections.Counter(x["status"]["robustness"] for x in v))' + +If volumes are rebuilding, **do nothing**. If a volume is stuck `degraded` or `faulted` with no +rebuild progressing, that is a storage problem to fix first — see `/runbooks/longhorn-degraded`. +Rebooting another node while a volume is at 1/3 is how you lose data. + +### 2. The drain timed out against a PodDisruptionBudget + +Six PDBs currently allow **zero** voluntary disruptions, and two kinds matter: + +- **`longhorn-system/instance-manager-*`** (one per node). `minAvailable: 1` against a selector + pinned to a single node's single pod, so `disruptionsAllowed` is structurally **always 0**. + Longhorn's `longhorn-manager` *deletes* the PDB when its `node-drain-policy` + (`block-if-contains-last-replica`) judges the drain safe. The drain is **supposed** to wait here. +- **CNPG `-primary`** (`keycloak-db`, `nextcloud-db`, `semaphore-db`). Also structurally 0 + until the CNPG operator switches the primary to another instance. If all three primaries are on + the node being drained, three switchovers must happen. + +kured's `drainTimeout` is 15m: a genuinely stuck drain **aborts**, releases the lock, and is retried +next period rather than wedging forever. Repeated timeouts show in kured's logs. + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pdb -A -o wide' | awk '$NF==0 || NR==1' + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PRIMARY:.status.currentPrimary' + +If CNPG will not switch over, an **operator** can move the primary deliberately before the next +window rather than fighting the drain. + +### 3. A node hung on reboot and still holds the lock + +`k3s-prod-2` has done this before (2026-09-03: powered, off the LAN ~38 minutes, physical power +button required). `lockTtl: 0` means the lock does **not** expire, so the cycle stops here — +correctly, since the cluster is at 2/3 and must not go to 1/3. + +Symptoms: one node `NotReady` or absent, `KuredNodeCoverageGap` firing alongside +`KuredRebootPending`, and the lock annotation naming the missing node. + +**Operator recovery:** attend to the node physically. Once it is back and `Ready`, kured clears its +own lock on the next period. If the node is gone for good and the lock must be cleared by hand, +that is an operator action: + + # OPERATOR ONLY. Only when you have confirmed the named node is NOT mid-reboot. + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured annotate ds kured \ + weave.works/kured-node-lock- --overwrite' + +If the node was pruned mid-reboot and is left **cordoned** with no daemon to uncordon it: + + # OPERATOR ONLY. + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod uncordon ' + +### 4. Nothing can write the sentinel (so nothing ever reboots) + +If `KuredRebootPending` never fires and reboots never happen even after a kernel upgrade, suspect +the sentinel writer. On Debian, `/var/run/reboot-required` is created by a hook shipped in +**`update-notifier-common`** — and it was absent on all three nodes before this automation landed. +Without it kured is permanently, invisibly idle. + + ssh -o BatchMode=yes 192.168.2.10 'dpkg -l update-notifier-common needrestart unattended-upgrades | tail -4' + ssh -o BatchMode=yes 192.168.2.10 'ls /etc/kernel/postinst.d/ /etc/apt/apt.conf.d/ | sort' + +Fix by re-running the role, which asserts this explicitly: + + cd infra/ansible && ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify + +### 5. kured is not being scraped (`KuredNotReporting`) + +The chart defaults `service.create` and `metrics.create` to **false**, and the `kured` namespace +ships a default-deny NetworkPolicy. If either enablement regressed, or the Prometheus ingress allow +was dropped, targets are *discovered* and every scrape times out — the TSDB stays empty and all +three rules go quiet. Longhorn hit exactly this failure on this cluster. + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get svc,servicemonitor,networkpolicy,ciliumnetworkpolicy' + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n monitoring exec \ + sts/prometheus-kube-prometheus-stack-prometheus -c prometheus -- \ + wget -qO- "http://localhost:9090/api/v1/targets?state=active" ' | grep -o '"job":"kured[^,]*' | head + +### 6. A node is uncovered by the DaemonSet (`KuredNodeCoverageGap`) + +kured declares **no tolerations**, because no prod node is tainted. A newly provisioned or +re-provisioned node carrying a taint gets no kured pod and will never be rebooted. + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes \ + -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints' + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds kured \ + -o custom-columns=DESIRED:.status.desiredNumberScheduled,READY:.status.numberReady' + +The fix is a PR adding the needed toleration to `platform/kured/helmrelease.yaml` — not a `kubectl +patch`, which Flux reverts. + +## Expected outage per reboot cycle + +Stated plainly, because this process is **not** seamless. Every workload at 1 replica on the +draining node is down until it reschedules and becomes Ready elsewhere. Measured 2026-10-17: **45 +Deployments and 3 StatefulSets run a single replica.** The ones worth knowing about: + +- **`kube-system/coredns` — 1 replica.** A cluster-wide in-cluster DNS gap while it moves. This is + the single most impactful item and it is **not** fixed by this PR; raising coredns to 2–3 replicas + is recommended follow-up work. +- **`monitoring/prometheus-…-0`, `monitoring/loki-0`, `monitoring/alertmanager-…-0`** — all + single-replica StatefulSets on Longhorn RWO volumes, so they additionally wait for volume + detach/reattach. **You lose monitoring and alerting for part of the cycle**, including these very + alerts. +- **All three CNPG Postgres primaries** blip during switchover, which means Keycloak, Nextcloud and + Semaphore briefly lose their database. +- **`gitea`, `nextcloud`, `plex`, `jackett`, `qbittorrent`, `radarr`, `sonarr`, `seerr`, `crafty`, + `homepage`, `headlamp`, `semaphore`** — each down for its reschedule. +- **`unifi` (a KubeVirt VM)** has `evictionStrategy: LiveMigrate` but + `migrationMethod: BlockMigration`, so evicting it copies the whole root disk over the network. + Expect the drain of its node to be slow. +- **`kube-vip`** fails the control-plane VIP over on every node reboot. + +Multi-replica and therefore fine: `cilium-operator`, `oauth2-proxy`, `kyverno-admission-controller`, +the Longhorn CSI sidecars, `keycloak`, `pihole`, `runbooks`. + +## When to escalate or stop + +**STOP and do not intervene** if: + +- Longhorn volumes are `degraded` **and rebuilding**. This is the gate doing its job. Wait. +- A single node is mid-reboot and under 15 minutes have passed. + +**STOP and get a human to the hardware** if: + +- Any node is `NotReady` while another reboot is pending. **Never clear the kured lock in this + state** — that is the one action that can take etcd from 2/3 to 1/3 and lose the cluster. +- Two nodes are down. The cluster has no quorum; this runbook cannot help. Recover hardware first. + +**Escalate to a deliberate maintenance decision** if: + +- `KuredRebootPending` has been firing for over a week. The reboot is not going to happen on its + own; find the blocker above, or take the reboot manually during a watched window. +- A volume is `faulted`. Fix storage before any further reboot. + +**Turn the automation off** if it is misbehaving and you need the cluster stable. The honest way is +a PR, but for an emergency an operator may suspend the tier — this stops future reboots and leaves +running nodes untouched: + + # OPERATOR ONLY. Stops kured being reconciled; does NOT uncordon a node mid-cycle. + infra/devcontainer/dev.sh run bash -c 'flux --context k3s-prod suspend kustomization kured' + +Check afterwards whether a node was left cordoned (First commands, step 3) and uncordon it if so. +Resume with `flux resume kustomization kured`. Nothing in the graph depends on `kured`, so +suspending it stalls nothing else. + +--- + +Source of truth: `runbooks/node-reboot-automation.md`. Index: `/runbooks/` + +Records: `decisions/2026-10-17-automated-upgrades.md` (design + what was not verified), +`platform/kured/helmrelease.yaml` (every tuned value and why), +`infra/ansible/roles/unattended_upgrades/README.md` (the package half, and the kernel exclusion).