Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
  jakman85andClaude Opus 5
|
2f2e98aa19 |
No files matched your search
@@ -1,173 +0,0 @@
|
||||
# Renovate — the dependency/image update automation. SCHEDULED, self-hosted against this Gitea.
|
||||
#
|
||||
# ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# WHY THIS FILE EXISTS: renovate.json HAS BEEN IN THIS REPO AND HAS NEVER ONCE RUN.
|
||||
# ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# Measured 2026-10-17: `renovate.json` is present, complete and thoughtfully written — it declares
|
||||
# managers, schedules, automerge classes and grouping. Nothing executes it. There is no Renovate
|
||||
# workflow (this is the first), no Renovate CronJob or Deployment in the cluster
|
||||
# (`kubectl get cronjob,deploy,job -A | grep -i renovate` -> nothing), and no bot account doing it
|
||||
# from outside. Meanwhile SEVEN files carry comments asserting "Renovate maintains this pin"
|
||||
# (platform/homepage/deployment.yaml:40, runbooks/serving/deployment.yaml:87,
|
||||
# platform/crafty/deployment.yaml:63, platform/vpn-ingress/daemonset.yaml:259,
|
||||
# platform/keycloak/postgres-cluster.yaml:75, apps/wave-0/nextcloud-db.yaml:302, and
|
||||
# platform/headlamp/helmrepository.yaml:2). Those comments have been false for the life of the file.
|
||||
#
|
||||
# The cost is measurable. With nothing bumping anything: the six per-app oauth2-proxy auth sidecars
|
||||
# sit on v7.8.1 while the platform oauth2-proxy in the SAME cluster runs v7.15.5; Keycloak (the IdP)
|
||||
# runs 26.0.6, a November-2024 release, 8 minors behind; Velero runs a RELEASE CANDIDATE
|
||||
# (velero-plugin-for-aws v1.14.3-rc.2) with a stable available; the CNPG Postgres image is 16.4,
|
||||
# missing every 16.5+ security release. A pin nothing maintains is not a pin, it is a fossil.
|
||||
#
|
||||
# ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# WHY RENOVATE AND NOT FLUX IMAGE-AUTOMATION (the decision, and it reverses ADR-0009's split)
|
||||
# ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# ADR-0009 divided the work: Renovate owns Helm chart/dependency bumps, Flux image-automation owns
|
||||
# per-service container image TAGS, deliberately non-overlapping. The second half of that split
|
||||
# cannot be delivered on this repo's terms, for three measured reasons:
|
||||
#
|
||||
# 1. IT WOULD BYPASS THE MERGE GATE, WHICH CLAUDE.md §1 FORBIDS. ImageUpdateAutomation does not
|
||||
# open pull requests; it COMMITS AND PUSHES. Its default target is the source's own branch —
|
||||
# `main` — so it would write straight past the PR flow. It can be pointed at a side branch
|
||||
# instead, but then nothing opens the PR and bumps silently accumulate on a branch nobody
|
||||
# merges. Either way a human or a second bot has to close the loop.
|
||||
# 2. ON THIS CLUSTER IT CANNOT EVEN DO THAT. Measured 2026-10-17: `main` is protected with
|
||||
# `user_can_push: false`, and the `flux-system` GitRepository has NO `spec.secretRef` at all —
|
||||
# the source is anonymous and read-only. Enabling write-back means minting an in-cluster git
|
||||
# WRITE credential for the repository that defines the cluster, which is a materially larger
|
||||
# blast radius than a CI-scoped token.
|
||||
# 3. RENOVATE ALREADY OPENS PRs NATIVELY, so every change it proposes goes through the gate with
|
||||
# no extra machinery.
|
||||
#
|
||||
# So: Renovate owns BOTH halves, and Flux image-automation stays disabled. Note this is a change of
|
||||
# plan, not a change of state — image-automation has been commented out at
|
||||
# platform/kustomization.yaml:52 since it was authored, and the cluster has ZERO
|
||||
# ImageRepository/ImagePolicy/ImageUpdateAutomation objects (measured). Its two controllers ARE
|
||||
# installed and have idled for 43 days. Nothing is being torn down here; a path that was never
|
||||
# taken is being formally closed, and renovate.json's `docker`/`kubernetes` managers — disabled
|
||||
# purely to avoid fighting a controller that never ran — are switched on to fill the gap.
|
||||
#
|
||||
# ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# THIS WORKFLOW MUST NOT BE ABLE TO SUCCEED SILENTLY
|
||||
# ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# A SKIPPED Gitea job reports `success` (CLAUDE.md §2), so "green" is not evidence this ran. Two
|
||||
# guards:
|
||||
# * The preflight step FAILS HARD (exit 1) if RENOVATE_TOKEN is absent, rather than skipping.
|
||||
# A missing credential on a dependency bot is indistinguishable from "nothing to update", and
|
||||
# that is precisely how renovate.json stayed inert for months.
|
||||
# * The summary step records what ran into the job summary unconditionally.
|
||||
# What this still does NOT give you is an alert when the SCHEDULE stops firing. See the handoff
|
||||
# entry's "What I did NOT verify" — that gap is stated, not papered over.
|
||||
#
|
||||
# OPERATOR PREREQUISITE — ONE SECRET, NOT COMMITTED HERE:
|
||||
# `RENOVATE_TOKEN` — a Gitea access token for the bot account, scopes `write:repository` and
|
||||
# `read:user`. Set it as an Actions secret on this repository. Nothing in this repo mints,
|
||||
# stores or references its value; it is consumed by name only.
|
||||
# Until it exists, this workflow FAILS LOUDLY on every scheduled run. That is intended: a
|
||||
# visible red is the correct state for "automation configured but not credentialed".
|
||||
#
|
||||
# THE FIRST RUN WILL WANT TO OPEN A LOT OF PRs. ~30 dependencies are behind. renovate.json caps the
|
||||
# blast radius (`prConcurrentLimit: 5`, `prHourlyLimit: 2`), but run it manually with
|
||||
# `dry_run: true` FIRST and read the log before letting the schedule loose. That is what
|
||||
# workflow_dispatch's default is for.
|
||||
name: renovate
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# 03:15 UTC daily. renovate.json ALSO carries `"schedule": ["after 3am and before 6am"]` in
|
||||
# America/New_York, and that inner schedule is the authoritative one — Renovate re-checks it and
|
||||
# will decline to open PRs outside it. The two are deliberately both present: this cron decides
|
||||
# when the process RUNS, renovate.json decides when it may ACT. Keeping the cron slightly ahead
|
||||
# of the repo window means a run is always in flight when the window opens.
|
||||
- cron: "15 3 * * *"
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
dry_run:
|
||||
description: "Dry run (no PRs, no branches). Leave true for the first run."
|
||||
required: false
|
||||
default: "true"
|
||||
log_level:
|
||||
description: "Renovate LOG_LEVEL (info | debug)"
|
||||
required: false
|
||||
default: "info"
|
||||
|
||||
concurrency:
|
||||
# Never two Renovate runs at once: concurrent runs race on the same branches and can open
|
||||
# duplicate PRs. Not cancel-in-progress — a half-finished Renovate run can leave a branch pushed
|
||||
# with no PR, so let the first one finish.
|
||||
group: renovate
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
renovate:
|
||||
runs-on: ubuntu-latest
|
||||
# Renovate scans every manager across the whole repo; 40m is generous for this repo's size and
|
||||
# well under a runner's patience. Without a timeout a wedged run holds the `renovate`
|
||||
# concurrency group and silently blocks every subsequent scheduled run.
|
||||
timeout-minutes: 40
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Preflight - the token must exist, and its absence must be LOUD
|
||||
env:
|
||||
RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [ -z "${RENOVATE_TOKEN:-}" ]; then
|
||||
echo "::error::RENOVATE_TOKEN is not set on this repository." >&2
|
||||
cat >&2 <<'MSG'
|
||||
|
||||
Renovate cannot run without a Gitea access token, and this step fails rather than
|
||||
skipping ON PURPOSE: a skipped Gitea job reports `success`, so skipping here would make
|
||||
"dependency automation is not running" look exactly like "everything is up to date".
|
||||
That is the failure mode that let renovate.json sit inert in this repo for months while
|
||||
seven files claimed "Renovate maintains this pin".
|
||||
|
||||
TO FIX: create a Gitea access token for the bot account with scopes
|
||||
`write:repository` and `read:user`, then add it to this repository as the Actions
|
||||
secret RENOVATE_TOKEN. Do not commit it anywhere.
|
||||
MSG
|
||||
exit 1
|
||||
fi
|
||||
echo "RENOVATE_TOKEN is present (value not printed)."
|
||||
|
||||
- name: Run Renovate
|
||||
uses: renovatebot/[email protected]
|
||||
with:
|
||||
# Pinned, and Renovate maintains this pin itself: renovate.json's enabledManagers
|
||||
# includes `github-actions`, and `helpers:pinGitHubActionDigests` is in its `extends`.
|
||||
# This is the one dependency in the repo whose updater is its own updater.
|
||||
configurationFile: renovate.json
|
||||
token: ${{ secrets.RENOVATE_TOKEN }}
|
||||
env:
|
||||
# Self-hosted Gitea. `platform` + `endpoint` were the two values renovate.json's own
|
||||
# header said were "set at runtime by the operator's renovate runner" — this is that
|
||||
# runner, so they are set here rather than in the repo config, which keeps renovate.json
|
||||
# portable and free of this instance's URL.
|
||||
RENOVATE_PLATFORM: gitea
|
||||
RENOVATE_ENDPOINT: https://gitea.home.keithjackson.dev/api/v1
|
||||
RENOVATE_REPOSITORIES: Home-Network/home-network-k3s-impl
|
||||
# Renovate must never push to a protected branch; it works on its own branches and opens
|
||||
# PRs against this base. Stated explicitly rather than inherited, because this is the
|
||||
# property that keeps the whole mechanism inside CLAUDE.md §1's branch -> PR -> gate path.
|
||||
RENOVATE_BASE_BRANCHES: main
|
||||
RENOVATE_ONBOARDING: "false"
|
||||
RENOVATE_REQUIRE_CONFIG: "required"
|
||||
RENOVATE_DRY_RUN: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run == 'true' && 'full' || '' }}
|
||||
LOG_LEVEL: ${{ github.event_name == 'workflow_dispatch' && inputs.log_level || 'info' }}
|
||||
|
||||
- name: Summary
|
||||
if: always()
|
||||
run: |
|
||||
{
|
||||
echo "## Renovate"
|
||||
echo
|
||||
echo "- trigger: \`${{ github.event_name }}\`"
|
||||
echo "- dry run: \`${{ github.event_name == 'workflow_dispatch' && inputs.dry_run || 'false' }}\`"
|
||||
echo "- outcome: \`${{ job.status }}\`"
|
||||
echo
|
||||
echo "Renovate owns BOTH Helm chart bumps and container image tags here; Flux"
|
||||
echo "image-automation is deliberately disabled (platform/kustomization.yaml)."
|
||||
echo "Every change it proposes arrives as a pull request and must pass the merge gate."
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
@@ -1,98 +0,0 @@
|
||||
---
|
||||
# Flux Kustomization: kured (Kubernetes Reboot Daemon) — reconciles ./platform/kured.
|
||||
# Performs the cordon / drain / reboot / uncordon of a node that has a pending OS reboot, strictly
|
||||
# one node at a time, inside a nightly window, gated on Longhorn volume health. Design rationale,
|
||||
# every tuned value, and the honest list of what is NOT mitigated: platform/kured/helmrelease.yaml.
|
||||
#
|
||||
# WHY ITS OWN KUSTOMIZATION, AND NOT ONE LINE IN platform/kustomization.yaml
|
||||
#
|
||||
# The same argument the Cilium split-out recorded in platform/kustomization.yaml and intel-gpu-plugin
|
||||
# reused, except the consequence is worse here than for either of them:
|
||||
#
|
||||
# * `platform` is `wait: true` over twelve-plus components, so its Ready condition cannot tell you
|
||||
# "the reboot daemon is wedged" from "longhorn is slow".
|
||||
# * The only emergency valve would be `flux suspend kustomization platform` — freezing storage,
|
||||
# TLS, DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a DaemonSet.
|
||||
# * That tier's 10m interval / 5m timeout are called out there as WRONG for a 3-node DaemonSet
|
||||
# rollout. This is a 3-node DaemonSet rollout.
|
||||
# * And `platform` gates platform-config, which gates apps-wave-0 (wait:true), which gates
|
||||
# apps-wave-1. A reboot daemon that failed to go healthy inside that tier would stall EVERY
|
||||
# APPLICATION in the cluster on "dependency not ready". A component added to make maintenance
|
||||
# safer must not acquire the ability to cause an outage; that trade is the whole argument.
|
||||
#
|
||||
# dependsOn: [platform] — AND THIS ONE IS A REAL EDGE, NOT CONVENTION
|
||||
#
|
||||
# Not `cilium` (what intel-gpu-plugin uses) and not `platform-config` (what the leaf UIs use). It is
|
||||
# `platform`, for two independent reasons:
|
||||
#
|
||||
# 1. CRD ORDERING (F-MON1). platform/kured ships a PrometheusRule, and the chart generates a
|
||||
# ServiceMonitor. Both are CRs whose CRDs (monitoring.coreos.com) are installed by
|
||||
# kube-prometheus-stack INSIDE the `platform` tier. Flux dry-runs every object in a pass before
|
||||
# applying any of them, so a pass containing an unknown kind ABORTS — the failure that made the
|
||||
# CDI StorageProfiles a true deadlock on prod on 2026-09-05 (see
|
||||
# clusters/base/platform-config-cdi.yaml). Depending on `platform` guarantees the CRDs exist
|
||||
# before this pass is attempted.
|
||||
# 2. FUNCTIONAL PREREQUISITE. kured's reboot gate QUERIES Prometheus
|
||||
# (`prometheusUrl` in the HelmRelease). Prometheus lives in platform/monitoring. If kured ran
|
||||
# first it would get connection errors, treat the gate as unevaluable and skip every reboot —
|
||||
# the correct fail-safe direction, but silent. Longhorn, whose volume health IS the gate, is
|
||||
# also in that tier.
|
||||
#
|
||||
# It does NOT need platform-config: no Keycloak, no TLS, no split-horizon DNS, no Gateway, no
|
||||
# HTTPRoute. kured serves no HTTP to users.
|
||||
#
|
||||
# wait: true — A HEALTH SIGNAL, NOT A GATE. Following intel-gpu-plugin rather than the leaf-UI
|
||||
# template. A leaf UI is legitimately not-Ready for long stretches of a cold start while it waits on
|
||||
# Keycloak and DNS, so its health is uninformative; kured has no such dependency — it is a DaemonSet
|
||||
# that opens a socket and watches a file, so on healthy nodes it is Ready in seconds and not-Ready
|
||||
# genuinely means BROKEN. This turns `flux get kustomization kured` into a one-line answer to "is
|
||||
# reboot automation live on all three nodes", which is the question the operator must be able to
|
||||
# answer before trusting a kernel upgrade to it.
|
||||
#
|
||||
# NOTHING IN THE GRAPH dependsOn THIS, so wait:true gates nothing — the blast radius of an unhealthy
|
||||
# kured is this Kustomization's own Ready condition. Do not add `dependsOn: [kured]` anywhere: that
|
||||
# would hand a node-rebooting DaemonSet the power to stall whatever depends on it, which is precisely
|
||||
# what the placement above exists to prevent.
|
||||
#
|
||||
# prune: true is safe and deliberate. If this Kustomization is deleted, kured's DaemonSet goes with
|
||||
# it and reboots simply stop happening — nodes are left uncordoned and running. The one caveat is
|
||||
# that pruning mid-reboot would leave a node CORDONED with no daemon to uncordon it; that is a
|
||||
# `kubectl uncordon` away and is covered in runbooks/node-reboot-automation.md.
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: kured
|
||||
namespace: flux-system
|
||||
spec:
|
||||
# A HelmRelease plus five static manifests, nothing to re-render; kustomize-controller is
|
||||
# source-triggered, so this is only the drift-correction floor.
|
||||
interval: 10m
|
||||
retryInterval: 2m
|
||||
# 10m, not the 5m the `platform` tier uses: a 3-node DaemonSet rollout under `wait: true` can
|
||||
# exceed 5m on a cold start (image pull on three nodes), which is one of the reasons the Cilium
|
||||
# split-out gave for leaving that tier.
|
||||
timeout: 10m
|
||||
prune: true
|
||||
# See the header: a health SIGNAL for a component nothing depends on, not a gate.
|
||||
wait: true
|
||||
dependsOn:
|
||||
- name: platform # monitoring CRDs + Prometheus + Longhorn; see the header, this is load-bearing
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./platform/kured
|
||||
# REQUIRED, not boilerplate. platform/kured/alerts.yaml renders `${BASE_DOMAIN}` into the
|
||||
# `runbook_url` annotation of all three rules. Without this block that token becomes the EMPTY
|
||||
# STRING and the Kustomization still reports Ready — Flux's substitution fails OPEN — so every
|
||||
# alert would ship a dead link to `https:///runbooks/node-reboot-automation`. The gate stage
|
||||
# `flux-postbuild-substitution` asserts the converse (a rendered token implies this block), and
|
||||
# BASE_DOMAIN is declared in clusters/base/cluster-vars.yaml.
|
||||
#
|
||||
# Nothing else in this path contains `${...}` or a bare `$`: the HelmRelease's alertFilterRegexp is
|
||||
# a plain anchored alternation, and there are no shell snippets anywhere in the component — which
|
||||
# is deliberate, because envsubst would eat a braced shell variable and `$$` would trip the
|
||||
# dollar-collapse assertion in the same check.
|
||||
postBuild:
|
||||
substituteFrom:
|
||||
- kind: ConfigMap
|
||||
name: cluster-vars
|
||||
@@ -85,17 +85,4 @@ resources:
|
||||
# pm-agent-credentials.sops.yaml, so both overlays add a SOPS decryption patch for it (by
|
||||
# name). See clusters/base/pm-agent.yaml and platform/pm-agent/README.md.
|
||||
- pm-agent.yaml # Flux Kustomization -> ./platform/pm-agent (dependsOn apps-wave-0)
|
||||
# kured (Kubernetes Reboot Daemon) — its OWN Kustomization (dependsOn platform, wait:true), NOT a
|
||||
# line in platform/kustomization.yaml. Same reasoning as the Cilium and intel-gpu-plugin
|
||||
# split-outs, with a sharper consequence: `platform` gates platform-config and every apps wave, so
|
||||
# a wedged reboot DaemonSet placed in that wait:true tier could stall every application in the
|
||||
# cluster — a component added to make maintenance SAFER must not be able to cause an outage.
|
||||
# wait:true here is a HEALTH SIGNAL, not a gate: nothing dependsOn it, and nothing should.
|
||||
# The `platform` edge is load-bearing rather than conventional — it supplies the
|
||||
# monitoring.coreos.com CRDs this component's PrometheusRule/ServiceMonitor need (F-MON1), and
|
||||
# Prometheus itself, which kured QUERIES before every reboot to gate on Longhorn volume health.
|
||||
# It reboots a node only when /var/run/reboot-required exists, which on these nodes only a
|
||||
# deliberate operator kernel upgrade creates — the human gate is the sentinel, not the daemon.
|
||||
# See clusters/base/kured.yaml and platform/kured/helmrelease.yaml.
|
||||
- kured.yaml # Flux Kustomization -> ./platform/kured (dependsOn platform)
|
||||
- cluster-vars.yaml # overlay knob surface (base defaults)
|
||||
@@ -363,54 +363,6 @@ SCRAPE_OWNERS: list[dict] = [
|
||||
"why": "velero_* powers VeleroBackupFailure / PartialFailure / NoRecentBackup — the rules "
|
||||
"that failed to notice prod running with zero volume backups",
|
||||
},
|
||||
# ── ADDED 2026-10-17 (feat/automated-upgrades). A NEW OWNER, NOT A RELAXED ASSERTION. ──────
|
||||
# This entry is purely additive: it registers a new metric family and its enablement, and it
|
||||
# changes nothing about how any existing owner is evaluated. It makes the check STRICTER for the
|
||||
# component it covers, not looser — it adds two `require` patterns plus a `companion` obligation
|
||||
# that platform/kured must keep satisfying, so a future edit that disables kured's Service,
|
||||
# ServiceMonitor or NetworkPolicy FAILS the gate instead of silently re-inerting three alerts.
|
||||
#
|
||||
# WHY THE CHECK HAD TO BE TOUCHED AT ALL, rather than the manifest being wrong: assertion (3)
|
||||
# fails any alert over a metric no owner claims, and the escape hatch (KNOWN_UNWIRED_METRICS) is
|
||||
# SHRINK-ONLY, so declaring `kured_*` as accepted debt is not permitted and would be the wrong
|
||||
# answer anyway — the scrape genuinely IS wired here. The only two honest options were to
|
||||
# register the owner (this) or to ship the reboot automation with no alerting at all, which for a
|
||||
# component whose every failure mode is silent would have been the worse engineering choice.
|
||||
# `conformance/**` is a sensitive approval path precisely so that this reasoning gets read.
|
||||
{
|
||||
"owner": "kured: service.create + metrics.create in the HelmRelease",
|
||||
"prefixes": ("kured_",),
|
||||
"file": "platform/kured/helmrelease.yaml",
|
||||
# BOTH keys are required and the chart defaults BOTH to false. `metrics.create` makes the
|
||||
# ServiceMonitor; `service.create` makes the Service it selects. Enable only the monitor and
|
||||
# it selects nothing, so the single kured metric is never scraped — and a PrometheusRule over
|
||||
# a series no target produces is INERT, which for this component means its own liveness
|
||||
# alarm is the thing that breaks. Matched as two separate anchored blocks rather than one
|
||||
# pattern because they are separate top-level values keys, not nested.
|
||||
"require": [
|
||||
r"service:\s*\n\s*create:\s*true",
|
||||
r"metrics:\s*\n\s*create:\s*true",
|
||||
],
|
||||
"forbid": [],
|
||||
"why": "kured_reboot_required is the ONLY metric kured exposes (one GaugeVec, "
|
||||
"cmd/kured/main.go:92-96) and it powers all three platform-health.node-reboots "
|
||||
"rules. Those rules are the only way to learn that OS reboot automation has stalled: "
|
||||
"kured stays 1/1 Running while rebooting nothing, so a missed scrape here is "
|
||||
"indistinguishable from a cluster that needed no reboots",
|
||||
# The ServiceMonitor ALONE is not sufficient — the same NetworkPolicy trap Longhorn hit.
|
||||
"companion": {
|
||||
"file": "platform/kured/kustomization.yaml",
|
||||
"require": [r"^\s*-\s*networkpolicy\.yaml\s*$"],
|
||||
"why": "platform/kured/networkpolicy.yaml ships a default-deny baseline (the kured "
|
||||
"namespace is a platform namespace, so Kyverno's app-namespace-scoped generator "
|
||||
"does not cover it) AND the ingress allow that lets Prometheus reach :8080 "
|
||||
"through it. Drop the networkpolicy from this kustomization and the deny-all "
|
||||
"goes too, so this currently fails SAFE — but reinstating the baseline without "
|
||||
"the allow reproduces Longhorn's measured failure exactly: targets DISCOVERED, "
|
||||
"every scrape 'context deadline exceeded', TSDB empty, all three rules quiet. "
|
||||
"Keeping the two in one file makes that pair inseparable",
|
||||
},
|
||||
},
|
||||
]
|
||||
|
||||
# `up` is produced by EVERY scrape target, so no owner can meaningfully claim it and assertion (1)
|
||||
|
||||
@@ -1,300 +0,0 @@
|
||||
---
|
||||
date: 2026-10-17
|
||||
branch: feat/automated-upgrades
|
||||
title: Automated upgrades — kured + security-only unattended-upgrades, and Renovate finally gets a runner
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Two halves of one process. Neither is useful alone and they must land together.
|
||||
|
||||
### (a) OS upgrades and reboots
|
||||
|
||||
**`platform/kured/`** — the Kubernetes Reboot Daemon, chart `6.1.0` / app `1.23.0`, in its own Flux
|
||||
Kustomization (`clusters/base/kured.yaml`, `dependsOn: [platform]`, `wait: true`), registered in
|
||||
`clusters/base/kustomization.yaml`. Not a line in `platform/kustomization.yaml`: that tier is
|
||||
`wait: true` and gates `platform-config` → every apps wave, so a wedged reboot daemon placed there
|
||||
could stall every application in the cluster. A component added to make maintenance safer must not
|
||||
be able to cause an outage.
|
||||
|
||||
**`infra/ansible/roles/unattended_upgrades/`** + `playbooks/unattended-upgrades.yml`, enabled for
|
||||
prod in `inventory/group_vars/k3s_prod/vars.yml`. A **new** role rather than an extension of
|
||||
`node_baseline` — different lifecycle, and other open branches touch that role.
|
||||
|
||||
**The design, and how it answers this repo's two standing objections to reboot automation.**
|
||||
`roles/node_baseline/tasks/main.yml:61-67` says "IT DOES NOT REBOOT" (k3s-prod-2 hung on a warm
|
||||
reboot 2026-09-03; physical power button required), and `docs/plans/cluster-upgrade-plan.md:291-299`
|
||||
rejected system-upgrade-controller for "automating away the human gate between nodes". Both still
|
||||
stand. **The human gate is the sentinel.** unattended-upgrades is security-only with the kernel
|
||||
blacklisted, so it essentially never produces `/var/run/reboot-required`; that file appears when an
|
||||
operator deliberately runs `node-baseline.yml --tags kernel`. kured then performs only the
|
||||
*mechanical* half the operator would otherwise type by hand — including the ordering rule
|
||||
`node_baseline`'s own output has to shout ("NEVER node 2 first"), which the operator keeps by
|
||||
running the kernel play with `--limit <host>` one node at a time. `cluster-upgrade-plan.md:329-330`
|
||||
names the condition under which its own rejection reverses (fast disks), and prod is that cluster.
|
||||
|
||||
**Quorum: concurrent reboots are structurally impossible.** All three nodes are control-plane+etcd
|
||||
(quorum 2/3). `concurrency: 1` plus kured's single cluster-wide lock annotation on its own DaemonSet
|
||||
means a node reboots only while holding that lock, enforced by API-server optimistic concurrency
|
||||
rather than by timing. **`lockTtl: 0`** (expiry disabled) is the deliberate choice: a TTL exists so
|
||||
reboots continue after the lock holder dies, which on 3 etcd nodes is exactly the catastrophe — a
|
||||
hung node's lock expiring would let a second node take the cluster to 1/3. So a hang stops the cycle
|
||||
forever and pages a human. `lockReleaseDelay: 30m` holds the lock after a node returns.
|
||||
|
||||
**The Longhorn gate.** Measured: 25 volumes, all `numberOfReplicas: 3`, one replica per node, with
|
||||
`replica-soft-anti-affinity: false` and `disable-scheduling-on-cordoned-node: true`. So every drain
|
||||
takes **all 25 volumes to degraded 2/3**, and recovery is serialized —
|
||||
`replica-replenishment-wait-interval: 600`, `concurrent-replica-rebuild-per-node-limit: 1`. Reboot
|
||||
the next node too early and volumes reach 1/3, where `node-drain-policy:
|
||||
block-if-contains-last-replica` can block a drain forever. The inter-node gate is therefore **volume
|
||||
robustness, not pod readiness**, and it is doubled: `lockReleaseDelay: 30m` (unconditional, covers
|
||||
the 600s wait) plus kured's Prometheus alert gate (semantic, covers a rebuild longer than 30m).
|
||||
|
||||
**`alertFilterMatchOnly: true` is load-bearing.** The flag inverts the regexp's meaning: `false`
|
||||
(chart default) makes it an *ignore* list and kured blocks on **any** firing alert. Measured
|
||||
2026-10-17, five alerts were firing with no end in sight (`CNPGPrimariesCoLocated`,
|
||||
`KubeDeploymentReplicasMismatch`, `KubeDeploymentRolloutStuck`, `KubePodNotReady`,
|
||||
`OAuth2ProxyGateUnobserved` — the last three all tracing to `pm-agent`, ImagePullBackOff on a
|
||||
placeholder digest for 12 days). Under the default, kured would have been blocked on day one and
|
||||
never rebooted anything, indistinguishable from "no reboots needed". The allow-list is anchored:
|
||||
`^(LonghornVolumeDegraded|LonghornVolumeFaulted|KubeNodeNotReady|NodeExporterDown)$`.
|
||||
`NodeEtcdRequestsSlow` and the `NodeIOPressure*` family are **deliberately excluded** despite being
|
||||
exactly what you'd want to pause for — prod-1 has a documented etcd WAL I/O starvation history and
|
||||
including them risks a permanent wedge that looks like safety.
|
||||
|
||||
**The kernel floor (the highest-risk item), handled with two independent exclusions.** Measured: all
|
||||
three nodes run `7.1.8+deb13-amd64` (`7.1.8-1~bpo13+1`, the backports metapackage), far above the
|
||||
6.13 memattr floor — **but `linux-image-6.12.107` and `linux-image-6.12.94` are also still
|
||||
installed, below the floor, bootable, and `apt-mark showhold` is empty on every node.** So: (1) the
|
||||
allowed-origins list is `label=Debian-Security` only, and the running kernel comes from
|
||||
`o=Debian Backports` (pin 100, the `NotAutomatic`+`ButAutomaticUpgrades` signature), so u-u cannot
|
||||
*see* a kernel upgrade; (2) `^linux-` is blacklisted anyway, so widening origins later — the most
|
||||
likely future edit — cannot re-arm the hazard. The role **asserts both against `apt-config dump`**,
|
||||
apt's own parse of the whole directory, rather than trusting the file it just wrote. Also blacklisted:
|
||||
`intel-microcode`, `amd64-microcode`, `grub-*`. `Remove-Unused-Kernel-Packages` is **false** against
|
||||
the Debian default, because those older kernels are the documented rollback path. None of this stops
|
||||
the kernel moving — the operator's deliberate Ansible run still does that, unchanged.
|
||||
|
||||
**The silent-no-op trap this closes.** Measured: `update-notifier-common`, `needrestart` and
|
||||
`unattended-upgrades` were **absent on all three nodes**, so *nothing could create
|
||||
`/var/run/reboot-required`*. kured deployed alone would have been permanently idle while 1/1 Running
|
||||
and reporting `kured_reboot_required=0` — green forever. The role installs the sentinel writer and
|
||||
**asserts a hook exists** by discovery (not a hard-coded path), failing loudly if not.
|
||||
`needrestart` runs in auto mode with `k3s.service` blacklisted — auto-restart is what makes a
|
||||
security-only policy effective, but restarting the control plane unattended is worse than a delayed
|
||||
library fix. (k3s is not an apt package at all here: no `k3s`, no `containerd` in dpkg.)
|
||||
|
||||
**Alerts + runbook.** `platform/kured/alerts.yaml` — three rules over kured's *only* metric
|
||||
(`kured_reboot_required`, one GaugeVec, read from `cmd/kured/main.go:92-96`) plus `kube_node_info`:
|
||||
`KuredNotReporting` (`absent()` — the pipeline is dead and the other two are inert),
|
||||
`KuredNodeCoverageGap` (`count(kured) < count(nodes)` — a node silently uncovered, which `absent()`
|
||||
cannot see), `KuredRebootPending` (`for: 36h` — a reboot was offered and blocked). Companion
|
||||
promtool suite `alerts.promtool-test.yaml` pins all three both ways, including that the rule
|
||||
**stops** when a reboot succeeds. `runbooks/node-reboot-automation.md` registered in
|
||||
`runbooks/kustomization.yaml`.
|
||||
|
||||
**The window is DAILY 03:00-06:00, and that choice is driven by the alert.** A weekly window means a
|
||||
pending reboot legitimately waits 6 days, forcing `for: 8d`+ — longer than this cluster's 7d
|
||||
Prometheus retention and reset by any restart past the 1h for-outage-tolerance. Daily caps the
|
||||
legitimate wait under 24h and makes `for: 36h` both robust and meaningful.
|
||||
|
||||
### (b) Image / dependency automation
|
||||
|
||||
**Renovate was aspirational. It now has a runner.** `.gitea/workflows/renovate.yaml` — scheduled
|
||||
daily 03:15 UTC, `workflow_dispatch` with `dry_run` defaulting to **true** for a safe first run.
|
||||
Preflight **fails hard (exit 1)** if `RENOVATE_TOKEN` is missing rather than skipping, because a
|
||||
skipped Gitea job reports `success` and that is precisely how `renovate.json` stayed inert.
|
||||
|
||||
**`renovate.json`: Renovate now owns both halves, and Flux image-automation is formally closed.**
|
||||
ADR-0009 gave container image tags to Flux image-automation; that cannot be delivered on this
|
||||
repo's terms. (1) `ImageUpdateAutomation` **pushes commits, it does not open PRs**, so it bypasses
|
||||
the CLAUDE.md §1 gate; pointed at a side branch instead, nothing opens the PR. (2) It cannot work
|
||||
here anyway: `main` is protected `user_can_push: false` and the `flux-system` GitRepository has **no
|
||||
`spec.secretRef`** — the source is anonymous and read-only. (3) Enabling it means an in-cluster git
|
||||
**write** credential for the repo that defines the cluster. So `docker`/`kubernetes` — disabled
|
||||
*only* to avoid fighting that controller — are enabled, with `kubernetes.managerFilePatterns`
|
||||
covering `platform/`, `platform-config/`, `apps/`, **`catalog/`** (apps are generated from it; a bump
|
||||
only to the rendered output is reverted by the next render and fails `ci/render-golden.sh`),
|
||||
`runbooks/serving/` and `infra/agent/`. Added a no-automerge `blast-radius` rule for `kured`,
|
||||
`longhorn`, `cilium`, `cloudnative-pg`, `kube-prometheus-stack`, because the gate is static and does
|
||||
not boot a node or form a quorum. `platform/kyverno/tests/**` added to `ignorePaths`.
|
||||
|
||||
### Sensitive paths — this PR needs a non-author approving review
|
||||
|
||||
It touches **`clusters/**`**, **`conformance/**`** and **`.gitea/**`**, all in
|
||||
`infra/gitea/approval-policy.json`. It will **not** merge on a green gate alone. Per CLAUDE.md §1 I
|
||||
have **not** posted `/approve-override` and must not — that is the self-approval the stage exists to
|
||||
prevent.
|
||||
|
||||
**The `conformance/` edit, justified.** `+34/-0` in
|
||||
`alert-metrics-have-scrape-targets/check.sh`: one new `SCRAPE_OWNERS` entry claiming `kured_`. It
|
||||
**weakens nothing** — no existing owner's evaluation changes — and it *tightens* the check for my
|
||||
own component, adding two `require` patterns (`service.create: true`, `metrics.create: true`; the
|
||||
chart defaults **both** to false) plus a `companion` requirement that `networkpolicy.yaml` stay in
|
||||
the kustomization. It had to be touched: assertion (3) fails any alert over an unclaimed metric, and
|
||||
the escape hatch `KNOWN_UNWIRED_METRICS` is **shrink-only**, so declaring `kured_*` as debt was not
|
||||
permitted and would have been wrong anyway. The alternative was shipping reboot automation with no
|
||||
alerting, which for a component whose every failure mode is silent is the worse choice. The entry
|
||||
carries a dated comment saying all of this.
|
||||
|
||||
## What I verified
|
||||
|
||||
Measured against the live cluster and nodes, read-only, on 2026-10-17.
|
||||
|
||||
- **Flux image-automation controllers ARE installed and idle.** `-n flux-system get deploy` shows
|
||||
`image-automation-controller` and `image-reflector-controller` both `1/1`, age **43d**; all three
|
||||
`image.toolkit.fluxcd.io` CRDs exist. And **zero** `ImageRepository`/`ImagePolicy`/
|
||||
`ImageUpdateAutomation` objects — "No resources found" for each. The controllers have run for 43
|
||||
days with nothing to do. `platform/kustomization.yaml:52` has the component commented out.
|
||||
- **Renovate is aspirational, confirmed three ways**: no workflow in `.gitea/` or `ci/` (grep),
|
||||
no Renovate CronJob/Deployment/Job in any namespace, and `renovate.json` itself disabled the
|
||||
`docker`+`kubernetes` managers — so every plain `image:` tag was maintained by **nothing**, while
|
||||
**seven files** claim "Renovate maintains this pin".
|
||||
- **`flux-system` GitRepository has no `secretRef`** (empty), URL is the in-cluster Gitea, and its
|
||||
artifact revision was `main@sha1:1a0b208…`, matching my base.
|
||||
- **Nodes**: 3× Debian 13 trixie, all `control-plane,etcd`, k3s `v1.36.4+k3s1`, kernel
|
||||
`7.1.8+deb13-amd64`. `apt-mark showhold` **empty**; `/etc/apt/preferences.d/` **empty**;
|
||||
`/var/run/reboot-required` absent; `unattended-upgrades`/`update-notifier-common`/`needrestart`
|
||||
all **absent**; `linux-image-6.12.107` and `6.12.94` installed alongside the backports kernel;
|
||||
no `k3s`/`containerd` apt package; `apt-cache policy` origins and pins as quoted above.
|
||||
- **No node taints** on any of the three (so no tolerations needed), and `system-node-critical`
|
||||
exists.
|
||||
- **Longhorn**: 1.12.1; `node-drain-policy: block-if-contains-last-replica`;
|
||||
`replica-soft-anti-affinity: false`; `disable-scheduling-on-cordoned-node: true`;
|
||||
`replica-replenishment-wait-interval: 600`; `concurrent-replica-rebuild-per-node-limit: 1`;
|
||||
25 volumes all 3-replica/healthy; 25 replicas per node.
|
||||
- **6 PDBs with 0 allowed disruptions**: 3× `longhorn-system/instance-manager-*` (structurally 0 —
|
||||
single-node selector, `minAvailable: 1`) and 3× CNPG `*-primary`. All three Postgres primaries
|
||||
were on `k3s-prod-1`.
|
||||
- **kured drain flags do not exist.** Read `cmd/kured/main.go:438-449`: `Force`,
|
||||
`DeleteEmptyDirData` and `IgnoreAllDaemonSets` are **hardcoded true**. Passing
|
||||
`--ignore-daemonsets`/`--delete-emptydir-data`/`--force` as extraArgs would CrashLoop the
|
||||
DaemonSet. All three are genuinely needed (12 DaemonSets, 135 emptyDir pods measured).
|
||||
- **kured exposes exactly one metric** — the only `MustRegister` in the project.
|
||||
- **Prometheus selectors are all `{}`** (`serviceMonitorSelector`, `serviceMonitorNamespaceSelector`,
|
||||
`ruleSelector`, `ruleNamespaceSelector`), so a `PrometheusRule`/`ServiceMonitor` in a new namespace
|
||||
is picked up with no label plumbing.
|
||||
- **`count(kube_node_info) == 3`** and `count(longhorn_volume_robustness) == 100`, queried through a
|
||||
read-only port-forward. The 5 chronically-firing alerts were read from `/api/v1/alerts`.
|
||||
- **`LonghornVolumeDegraded`** is `longhorn_volume_robustness{state="degraded"} == 1`, `for: 10m` —
|
||||
shorter than Longhorn's own 600s replenishment wait, so degradation reliably outlasts the latch.
|
||||
- **kured chart 6.1.0 → appVersion 1.23.0** is the newest of 80 entries in the chart index.
|
||||
- **Kyverno/PSA will not block it**: every policy is scoped
|
||||
`app-namespace: "true"`; the namespace sets it explicitly `"false"`, and
|
||||
`restrict-host-namespaces.yaml`'s own header states that carve-out. No cluster-wide PSA baseline.
|
||||
- **`renovate.json` parses** as valid JSON after my edits (`json.load`).
|
||||
- **kured is not installed** and the string `kured` appears nowhere in the repo — greenfield.
|
||||
|
||||
### CORRECTION 2026-10-18 — the first real run FAILED, and the sentinel design was wrong
|
||||
|
||||
`ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod` failed on all three
|
||||
nodes at the package task:
|
||||
|
||||
```
|
||||
fatal: [k3s-prod-1]: FAILED! => "No package matching 'update-notifier-common' is available"
|
||||
```
|
||||
|
||||
**`update-notifier-common` is an UBUNTU package and does not exist in Debian.** Measured on trixie:
|
||||
`apt-cache policy update-notifier-common` → `Candidate: (none)`. The original entry asserted this
|
||||
package "owns the hook that creates /var/run/reboot-required" — true on Ubuntu, false here, and the
|
||||
role could never have worked. This was the exact risk the first version of this entry flagged ("the
|
||||
`find` patterns are a guess at which package ships what") landing in a worse place than predicted:
|
||||
not a wrong glob, a nonexistent package.
|
||||
|
||||
Three further measurements settled what to do instead:
|
||||
|
||||
* `/usr/share/update-notifier/notify-reboot-required` is **absent**.
|
||||
* **No `linux-image-*` postinst references `notify-reboot-required`** — invoking it is an Ubuntu
|
||||
patch, so merely placing a script at that path would never be called.
|
||||
* No hook under `/etc/apt/apt.conf.d` or `/etc/kernel/` mentions `reboot-required`.
|
||||
|
||||
So Debian trixie has **no stock mechanism at all**, and the role now ships the writer itself:
|
||||
`/usr/local/sbin/k3s-notify-reboot-required` (compares the running kernel against the newest
|
||||
installed via `linux-version`, from `linux-base`, which every `linux-image` depends on — measured
|
||||
`linux-base 4.12.1` present) plus a `DPkg::Post-Invoke` hook at
|
||||
`/etc/apt/apt.conf.d/99-k3s-reboot-required`. `/run` is a tmpfs (measured), so the sentinel
|
||||
self-clears on boot and needs no cleanup task.
|
||||
|
||||
The verify assert changed shape too: it was a `find` glob over `/etc/kernel/postinst.d` and
|
||||
`/etc/apt/apt.conf.d` for `*update-notifier*` / `*unattended-upgrades*`. That glob was **weaker
|
||||
than it looked** — it would have matched Debian's own `unattended-upgrades` apt conffile and
|
||||
reported "sentinel writer present" while no writer existed. It now stats the two exact paths this
|
||||
role owns and requires the script to be executable.
|
||||
|
||||
**Newly verified (this correction):** the sentinel script was executed on prod-1 as the
|
||||
unprivileged `k3s` user under `sh -x`. It resolved `running=7.1.8+deb13-amd64`,
|
||||
`newest=7.1.8+deb13-amd64`, took the equal branch, exited 0 and wrote nothing — correct, since the
|
||||
running kernel *is* the newest. `sh -n` and `bash -n` clean.
|
||||
|
||||
**Still NOT verified after this correction:** the *writing* half. Because running == newest on all
|
||||
three nodes, the `touch` path has never executed, so I have not seen the sentinel actually created,
|
||||
nor seen kured react to it. The honest test is the operator's next kernel upgrade. Also unverified:
|
||||
whether a `DPkg::Post-Invoke` hook fires in the `unattended-upgrades` code path specifically (it
|
||||
runs dpkg, so it should, but that is inference) — and whether `linux-version sort --reverse` orders
|
||||
a mixed backports/stable kernel set the way I assume when a 6.12 and a 7.1 are both installed. It
|
||||
produced the right answer with the current set; a future set could differ.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
The honest list. Most of this needs an actual upgrade cycle, which I did not and could not run.
|
||||
|
||||
- **I never ran an OS upgrade, never rebooted a node, and never watched kured take a lock.** No part
|
||||
of the runtime behaviour of either half has been observed. `sudo` needs a password I do not have,
|
||||
so I could not run `apt`, `unattended-upgrade --dry-run`, or install anything. **Everything about
|
||||
(a) working is inference from configuration and measured cluster state, not observation.**
|
||||
- **The Ansible role has never been executed.** Not syntax-checked by `ansible-playbook --syntax-check`
|
||||
either (no inventory access from the gate). Its asserts, its `apt-config dump` parsing, and the
|
||||
sentinel-hook discovery are all unproven. The `find` patterns for the sentinel hook are a *guess at
|
||||
which package ships what* — I could not read `update-notifier-common`'s file list (packages.debian.org
|
||||
returned no such package for that suite/arch), which is exactly why the role discovers rather than
|
||||
hard-codes. **If no hook matches, the role fails loudly — but it may fail on a correct node.**
|
||||
- **Whether CNPG actually switches the primary in response to a drain-induced eviction.** All three
|
||||
`*-primary` PDBs are structurally 0-disruption and `.spec.nodeMaintenanceWindow` is unset on all
|
||||
three clusters. If the operator does not switch over, draining `k3s-prod-1` blocks on three PDBs at
|
||||
once and hits `drainTimeout`. **This is the most likely way the first real reboot fails.** It
|
||||
should be tested with one manual, watched `kubectl drain` before trusting the timer.
|
||||
- **Whether `drainTimeout: 15m` is long enough.** Unmeasured. The `unifi` VMI has
|
||||
`evictionStrategy: LiveMigrate` with `migrationMethod: BlockMigration`, i.e. a full root-disk copy
|
||||
over the network on every `k3s-prod-1` drain. I do not know how long that takes.
|
||||
- **Whether the alert gate actually blocks as intended.** I never saw kured query Prometheus. The
|
||||
assumption that it treats a query error as "do not proceed" is from its documented behaviour, not
|
||||
observed. The NetworkPolicy allowing kured→Prometheus:9090 and Prometheus→kured:8080 is
|
||||
**unproven**; if either is wrong the symptom is silence, which is why `KuredNotReporting` exists —
|
||||
but that alert is itself unproven against a live scrape.
|
||||
- **The alert expressions have not run against real `kured_*` data**, because the metric does not
|
||||
exist in this cluster yet. They pass promtool unit tests against synthetic series whose shape I
|
||||
read from kured's source. `absent()` and the `count() < count()` comparison are the two most
|
||||
likely to behave differently against real label sets.
|
||||
- **`enabledManagers: ["regex"]` may be the wrong name.** Modern Renovate expects `custom.regex` for
|
||||
`customManagers`. I left the pre-existing value alone rather than guess, but **the inline
|
||||
`# renovate:` markers may simply not run.** Unverifiable without executing Renovate. Two markers
|
||||
are additionally known-broken independent of this: `external-snapshotter`'s sits two comment lines
|
||||
above its `newTag`, and Keycloak's `kustomization.yaml` has no marker at all despite a comment
|
||||
claiming Renovate maintains it.
|
||||
- **Renovate has still never run.** I added the runner; `RENOVATE_TOKEN` does not exist yet, so the
|
||||
first scheduled run **will fail** (by design). Nothing about the config is proven correct until a
|
||||
`dry_run: true` dispatch is read by a human. Expect it to want to open ~30 PRs.
|
||||
- **No alert for "automatic patching silently stopped".** u-u's success lives in
|
||||
`/var/log/unattended-upgrades/` on each node and nothing reads it. Likewise **no alert for the
|
||||
Renovate schedule ceasing to fire**. Both are real gaps I did not close.
|
||||
- **`coredns` is 1 replica and I did not fix it.** It is k3s-managed, out of this PR's scope, and it
|
||||
means every `k3s-prod-1` reboot causes a cluster-wide in-cluster DNS gap. Recommended follow-up.
|
||||
- **kured cannot honour "NEVER node 2 first".** It has no ordering control. The mitigation — run the
|
||||
kernel play with `--limit` one host at a time — is a *procedure*, documented in the runbook, not an
|
||||
enforced property. If the operator flags all three nodes at once, kured picks the order.
|
||||
- **kured cannot press a power button.** The 2026-09-03 warm-reboot hang is unmitigated. Bounded, not
|
||||
prevented: quorum holds at 2/3, the lock is held forever so nothing cascades, and
|
||||
`KuredRebootPending` eventually fires.
|
||||
- **Upstream version claims are a snapshot, partly from search results**, not all from authoritative
|
||||
release APIs. I could not determine latest for versitygw, crafty, nginx-unprivileged, act_runner,
|
||||
`virt-template-*`, the linuxserver wireguard date-tag, or the exact latest PostgreSQL 16 patch
|
||||
(sources conflicted 16.10 vs 16.15). **The gitea `1.24.3 → v28.1.0` gap is almost certainly an
|
||||
upstream versioning-scheme change, not 26 majors — do not act on that framing.**
|
||||
- **I did not verify Renovate's `kustomize` manager still processes `images:`/`newTag`** now that
|
||||
`docker` is enabled, which decides whether `external-snapshotter` and `intel-gpu-plugin` are
|
||||
managed at all.
|
||||
- **I did not raise any HelmRelease version ceiling**, so the 9 minor-locking ranges still cap
|
||||
Renovate at patches. Bumping `kube-prometheus-stack` in particular will fail the gate until
|
||||
`ci/versions.env`'s exact promtool/Prometheus `3.14.0` lockstep moves with it.
|
||||
- **L2/L3 conformance did not run** (`vars.CLUSTER_AVAILABLE` unset), so nothing here is proven
|
||||
against a cluster by CI either.
|
||||
@@ -0,0 +1,106 @@
|
||||
---
|
||||
date: 2026-10-17
|
||||
branch: feat/graceful-cluster-shutdown
|
||||
title: "Graceful node shutdown was never configured, and scaling a StatefulSet deleted its PVC"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
All three nodes had to be powered off to move them onto a UPS. There was **no procedure for it**:
|
||||
`infra/cluster.sh stop` is `virsh`-based and only ever applied to the decommissioned dev cluster, so
|
||||
the whole sequence was improvised live. Two defects came out of doing it.
|
||||
|
||||
**1. Kubelet graceful node shutdown was unconfigured on every prod node.** No `shutdownGracePeriod`
|
||||
anywhere in `infra/ansible/`, so `shutdown -h` stopped k3s, containerd went with it, and every pod
|
||||
was SIGKILLed where it stood — 9 CloudNativePG instances mid-checkpoint, the UniFi VM, and 25
|
||||
attached Longhorn volumes. `roles/k3s_server` now writes `kubelet-arg` into
|
||||
`/etc/rancher/k3s/config.yaml` with `shutdown-grace-period=90s` and
|
||||
`shutdown-grace-period-critical-pods=20s`.
|
||||
|
||||
The half that is easy to miss and silently defeats the other half: **graceful node shutdown is
|
||||
enforced by logind, not by the kubelet.** The kubelet holds an inhibitor *delay* lock and systemd
|
||||
waits at most `InhibitDelayMaxSec` — **default 5 seconds** — before proceeding regardless. A grace
|
||||
period above that cap is truncated with no error, so the feature looks configured and still
|
||||
hard-kills pods. The role now ships a `logind.conf.d` drop-in setting it to 120, and **asserts**
|
||||
`k3s_logind_inhibit_delay_max_sec >= k3s_kubelet_shutdown_grace_period` rather than documenting the
|
||||
relationship, because the failure is invisible.
|
||||
|
||||
**2. Scaling a StatefulSet to 0 DELETED its PVC and Longhorn volume.** `monitoring/loki` carries
|
||||
`persistentVolumeClaimRetentionPolicy: {whenDeleted: Delete, whenScaled: Delete}`, so the quiesce's
|
||||
`scale --replicas=0` destroyed `storage-loki-0` outright. Loki stores with `type: filesystem`, so
|
||||
that PVC was the only copy of its log data. This is the defect in this change set that actually
|
||||
destroyed something, and it was mine.
|
||||
|
||||
New `infra/maintenance/quiesce.sh` (`quiesce` | `resume` | `status`) encodes the sequence that
|
||||
worked, and **refuses to scale any StatefulSet whose `whenScaled` is `Delete`** — it leaves it
|
||||
running, accepts that its volume hard-detaches at poweroff, and prints which volumes to expect
|
||||
`degraded`. A rebuild is a vastly better outcome than a deletion. Workload discovery is computed
|
||||
from the live cluster (every Deployment/StatefulSet whose pods mount a PVC, Longhorn or TrueNAS NFS)
|
||||
rather than hardcoded, so a newly onboarded service is covered without anyone remembering to add it.
|
||||
|
||||
`runbooks/cluster-maintenance-shutdown.md` (new, registered in `runbooks/kustomization.yaml`) is the
|
||||
operator page: shutdown order, the TrueNAS-after-nodes constraint, power-on order, the
|
||||
un-hibernate-Postgres step Flux will not do for you, and a "what looks broken but is fine" section
|
||||
separating the `4/19`-Kustomizations-Ready cascade, the degraded-and-rebuilding volume and the
|
||||
self-recovering `keycloak-operator` from the one thing that genuinely needs a human (a stuck
|
||||
`oauth2-proxy` replica).
|
||||
|
||||
## What I verified
|
||||
|
||||
* **Graceful shutdown was genuinely absent**, not merely undocumented: `grep -rn
|
||||
'shutdownGracePeriod|graceful-node-shutdown'` over `infra/ansible/` returns nothing, and no
|
||||
`logind` or `InhibitDelay` setting exists anywhere in the tree.
|
||||
* **The whole shutdown/restart cycle was executed end to end on prod 2026-10-17** and the cluster
|
||||
came back: 3/3 nodes Ready with etcd `healthz` ok, 19/19 Flux Kustomizations Ready,
|
||||
`platform-config`'s `lastAppliedRevision` equal to the GitRepository artifact revision, all 3
|
||||
CNPG clusters `healthy` with primaries elected, the UniFi VM `Running`, PiHole resolving
|
||||
(`home.keithjackson.dev` → `192.168.2.20` via `192.168.2.23`), and the Gateway serving
|
||||
`200/200/401` for apex/gitea/plex.
|
||||
* **The PVC-deletion mechanism is confirmed, not inferred:** `kubectl get sts loki -o
|
||||
jsonpath={.spec.persistentVolumeClaimRetentionPolicy}` returns
|
||||
`{"whenDeleted":"Delete","whenScaled":"Delete"}`, the PVC was `NotFound` after the cycle, and the
|
||||
Longhorn volume count had gone 25 → 24.
|
||||
* **The damage was repaired and verified:** `scale sts loki --replicas=1` (restoring the replica
|
||||
count the chart declares) recreated `storage-loki-0` as a fresh 50Gi Bound PVC, `loki-0` reached
|
||||
`2/2 Running`, and the volume count returned to 25.
|
||||
* **The guard works against the live cluster**, tested both ways: `sts_deletes_pvc_on_scale
|
||||
monitoring loki` returns true (would be skipped) and `pihole pihole` returns false (safe to
|
||||
scale).
|
||||
* **The script's read-only paths run clean on prod:** `status` reports correctly, and the discovery
|
||||
function returns the 13 workloads currently holding a PVC.
|
||||
* `bash -n` clean; the two edited Ansible YAML files parse.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
* **The kubelet change has NEVER BEEN APPLIED OR TESTED.** This is the significant gap. I did not
|
||||
run the role, did not restart k3s, and have **not** observed a node actually drain on
|
||||
`shutdown -h`. Everything about it is reasoning from kubelet/systemd semantics plus the measured
|
||||
absence of the setting. Applying it needs a **rolling** k3s restart (`serial: 1` — kube-vip keeps
|
||||
the API up, etcd holds at 2/3), which is an operator action (CLAUDE.md §5). **Verify by draining
|
||||
one node for real:** apply, then `shutdown -h now` on a single node and watch whether pods
|
||||
terminate gracefully instead of vanishing.
|
||||
* **Whether 90s/20s are the right numbers.** Chosen to comfortably cover a Postgres fast shutdown
|
||||
and a Longhorn flush; not measured against either. The budget is spent on every reboot, so if a
|
||||
node takes noticeably longer to go down, this is why.
|
||||
* **Whether k3s accepts `kubelet-arg` in `config.yaml` as written.** The file already carries
|
||||
`kube-apiserver-arg` as a list and `kubelet-arg` is the documented sibling, but I did not render
|
||||
or apply it, so a typo would surface only at k3s restart — and a bad `config.yaml` is exactly the
|
||||
kind of thing that makes k3s fail to come up. **Review this one before applying.**
|
||||
* **The `assert` task's `regex_replace` on the grace period** assumes a plain `90s`-style value; a
|
||||
value expressed as `1m30s` would parse to the nonsense integer `130`. Fine for the committed
|
||||
defaults, wrong if someone writes a compound duration.
|
||||
* **`resume`'s Flux-reconcile wait has never run**, and neither has `quiesce` as a script — the
|
||||
2026-10-17 cycle was done with the equivalent commands by hand, so the script encodes a sequence
|
||||
that worked but has not itself been executed end to end. Only `status` and the discovery and guard
|
||||
functions have run.
|
||||
* **Loki's lost log window is gone for good.** Up to 72h of logs (its `retention_period`), destroyed
|
||||
by my scale-down. Nothing recovers it; Velero's schedules cover the namespace but the PVC was
|
||||
deleted, not corrupted, and I did not attempt a restore because the data self-expires anyway.
|
||||
* **No other `whenScaled: Delete` StatefulSet was audited.** The guard now catches them generically,
|
||||
but I did not enumerate which other workloads carry the policy, so I cannot say loki was the only
|
||||
one at risk. The runbook gives the one-liner to list them.
|
||||
* **TrueNAS and the router are entirely outside this change.** TrueNAS is not in the Ansible
|
||||
inventory and refuses SSH, so its shutdown stays a web-UI step with no automation and no check.
|
||||
* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so CI proves the L0/L1
|
||||
offline subset only.
|
||||
* I did not re-measure branch protection or runner state (CLAUDE.md §2b).
|
||||
@@ -1,99 +0,0 @@
|
||||
---
|
||||
date: 2026-10-17
|
||||
branch: fix/oauth2-proxy-oidc-startup-race
|
||||
title: "oauth2-proxy dies on a cold start because OIDC discovery runs once, before Keycloak serves"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Exposed by the 2026-10-17 whole-cluster shutdown for the UPS move. oauth2-proxy performs OIDC
|
||||
discovery **once, at startup**, and `os.Exit(1)`s if it fails. On a cold boot Keycloak is not
|
||||
serving yet, so the shared Gateway answers 503 and the proxy dies:
|
||||
|
||||
```
|
||||
[provider.go:55] Performing OIDC Discovery...
|
||||
[main.go:73] ERROR: Failed to initialise OAuth2 Proxy: error initialising provider: ...
|
||||
failed to discover OIDC configuration: unexpected status "503": no healthy upstream
|
||||
```
|
||||
|
||||
It then sits in **exponential CrashLoopBackOff long after Keycloak is healthy** — one of the two
|
||||
replicas was still down minutes later and only a manual `kubectl delete pod` cleared it. There is no
|
||||
retry flag; the process does not survive a cold start. A readinessProbe cannot help (the container
|
||||
is already dead), and CrashLoopBackOff backoff is not tunable per-pod.
|
||||
|
||||
`platform/oauth2-proxy/helmrelease.yaml` gains an `extraInitContainers` gate (`wait-for-idp`) that
|
||||
polls Keycloak's discovery doc until it answers, then lets the proxy start. A pod in `Init` does not
|
||||
accrue CrashLoopBackOff backoff and retries promptly, which is the point.
|
||||
|
||||
**Bounded, and fails open.** After ~5 minutes it gives up and starts the proxy anyway, so the worst
|
||||
case is exactly today's behaviour and never something new. An unbounded wait would trade a CrashLoop
|
||||
for a pod wedged in `Init` forever, which is not an improvement.
|
||||
|
||||
## Why this shape and not the alternatives
|
||||
|
||||
* **`extraInitContainers` is the chart's own documented hook for precisely this.** Its commented
|
||||
example in `values.yaml` is literally `- name: wait-for-idp`. The key was **confirmed against the
|
||||
pinned chart** (`helm show values oauth2-proxy/oauth2-proxy --version 10.7.1` → `extraInitContainers: []`
|
||||
at line 428), not assumed — a wrong Helm values key is silently ignored, which is the
|
||||
"reviewed, merged, apparently done, still broken" shape this repo keeps getting bitten by.
|
||||
* **No new base image.** The chart already runs an `alpine` init container for its built-in
|
||||
`initContainers.waitForRedis` gate, so alpine is a base this chart uses regardless. Pinned to
|
||||
`3.22` (verified to exist upstream) rather than the chart's own `latest`.
|
||||
* **Flux `postRenderers` rejected:** it would work independently of the chart schema, but nothing in
|
||||
this tree uses postRenderers, so it would introduce a mechanism with no precedent for a problem
|
||||
the chart already has a hook for.
|
||||
* **`--skip-oidc-discovery` rejected:** it removes the startup fetch entirely, which is arguably
|
||||
more elegant, but it means hardcoding four Keycloak endpoints and I could not verify offline that
|
||||
oauth2-proxy then fetches JWKS lazily rather than at startup. oauth2-proxy gates SSO for PiHole,
|
||||
the \*arr apps, qBittorrent, Homepage, Prometheus, Alertmanager and Hubble — a wrong guess locks
|
||||
the operator out of all of them, so the reversible, additive option wins.
|
||||
|
||||
The `$$`-free convention matters here: shell variables are written bare (`$i`, `$url`, `$(( ))`)
|
||||
with **no braced form anywhere**, because `platform` declares `postBuild.substituteFrom cluster-vars`
|
||||
and envsubst consumes braced expansions — a braced shell variable renders to the empty string and
|
||||
silently breaks the loop. `${BASE_DOMAIN}` is braced on purpose. Same hazard and same convention as
|
||||
the gitea and nextcloud OIDC bootstrap hooks, and `ci: flux-postbuild-substitution` enforces it.
|
||||
|
||||
## What I verified
|
||||
|
||||
* **The failure is measured, not theorised.** The log above is from the live pod
|
||||
(`oauth2-proxy-5d6c74876b-bzb2p`, `lastState.terminated: {reason: Error, exitCode: 1}`), captured
|
||||
with `--previous` after the cold boot. The surviving replica's log shows the same startup
|
||||
succeeding once Keycloak was up, which is what identifies this as a race rather than a config
|
||||
fault.
|
||||
* **Deleting the pod is a sufficient workaround** — done on prod 2026-10-17, the replacement came up
|
||||
clean and both replicas have been healthy since.
|
||||
* **The chart key is real**, at the pinned version range: `helm search repo` resolves
|
||||
`>=10.0.0 <11.0.0` to chart `10.7.1` (app `7.15.5`), whose values carry `extraInitContainers: []`,
|
||||
`extraContainers: []` and the built-in `initContainers.waitForRedis` (which uses `alpine`).
|
||||
* **`docker.io/library/alpine:3.22` exists** — confirmed against the registry tag list rather than
|
||||
assumed from memory.
|
||||
* The edited HelmRelease parses as YAML.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
* **UNAPPLIED. The init container has never run.** I did not reconcile the HelmRelease, so I have not
|
||||
seen `wait-for-idp` execute, exit 0, or gate the proxy. Applying it rolls both oauth2-proxy
|
||||
replicas, which is a mutating action (CLAUDE.md §5) — and it rolls the component that gates SSO for
|
||||
seven apps, so **do it when you can watch it**, not unattended.
|
||||
* **I did not re-create the cold-start race to prove the fix.** That needs Keycloak down while
|
||||
oauth2-proxy starts; the honest test is the next whole-cluster boot. Until then this is reasoned
|
||||
from the measured failure, not demonstrated against it.
|
||||
* **busybox `wget`'s TLS behaviour here is assumed, not tested.** The claim is that alpine's
|
||||
`ca-certificates-bundle` lets it validate the Gateway's LE-prod certificate, so a genuine TLS
|
||||
failure exits non-zero rather than passing falsely. If that is wrong in either direction the gate
|
||||
degrades safely — a never-succeeding probe just burns the 5-minute budget and then starts the proxy
|
||||
anyway — but the fast path would be silently dead and this would be doing nothing. **Check the
|
||||
init container's log on first apply**: it should print "Keycloak discovery reachable" within
|
||||
seconds on a warm cluster, not the give-up message.
|
||||
* **Whether a pod can reach the Gateway hostname at all from this namespace is inherited, not
|
||||
re-proven.** The main container does exactly this fetch today and succeeds, so the path is known
|
||||
good — but note the standing trap that an arbitrary pod gets 403 from every Gateway hostname
|
||||
(`decisions/2026-10-04-plex-advertise-plexdirect.md`); oauth2-proxy is exempt only because
|
||||
Keycloak's ingress policy admits it. An init container in the same pod inherits that identity, so
|
||||
this should hold — unverified.
|
||||
* **60 attempts x 5s is not tuned.** It is "comfortably longer than Keycloak's observed cold start",
|
||||
not a measured bound.
|
||||
* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline
|
||||
subset only.
|
||||
* I did not re-measure branch protection or runner state (CLAUDE.md §2b).
|
||||
@@ -420,34 +420,3 @@ flux_bootstrap_env_path: "clusters/prod"
|
||||
# rather than leaving an inert drop-in behind, so this group-wide `true` is still correct
|
||||
# for it.
|
||||
etcd_io_priority_enabled: true
|
||||
|
||||
# --- automatic security patching (roles/unattended_upgrades) -------------------------
|
||||
# TURNS ON roles/unattended_upgrades FOR PROD ONLY. Role default is false. Same seam as
|
||||
# node_baseline_backports_kernel and etcd_io_priority_enabled above.
|
||||
#
|
||||
# WHAT WAS THERE BEFORE: nothing at all. Measured on all three prod nodes 2026-10-17 —
|
||||
# `unattended-upgrades`, `update-notifier-common` and `needrestart` were ALL absent,
|
||||
# `apt-mark showhold` was empty on every node, /etc/apt/preferences.d/ was empty, and
|
||||
# nothing on the node could create /var/run/reboot-required. These boxes have had ZERO
|
||||
# automatic patching since provisioning, and no way to signal that a reboot was needed.
|
||||
#
|
||||
# WHAT ENABLING THIS DOES *NOT* DO: nothing, until the operator runs
|
||||
# playbooks/unattended-upgrades.yml. It is not applied by site.yml (deliberately — see
|
||||
# that playbook's header). And once run, it still never upgrades the KERNEL, never
|
||||
# restarts k3s, and never reboots:
|
||||
# * kernel — blacklisted `^linux-`, AND the trixie-backports archive the running
|
||||
# kernel comes from (o=Debian Backports, measured) is not an allowed origin. Two
|
||||
# independent exclusions, because this is the EFI-memattr constraint: these HP minis
|
||||
# need >= 6.13 or they can boot with a fraction of their RAM
|
||||
# (PROD-PROVISIONING.md §0b), and 6.12 kernels are still installed and bootable here.
|
||||
# The kernel continues to move ONLY under a deliberate
|
||||
# `node-baseline.yml --tags kernel` run, exactly as before.
|
||||
# * k3s — needrestart blacklist. k3s is not even an apt package on these nodes
|
||||
# (measured: no `k3s`, no `containerd` in dpkg), so the control plane and runtime are
|
||||
# outside apt's reach regardless.
|
||||
# * reboots — Automatic-Reboot "false", asserted by the role against apt's own
|
||||
# effective config. Reboots belong to kured (platform/kured): one node at a time,
|
||||
# under a cluster-wide lock, inside 03:00-06:00, after cordon + drain. Three
|
||||
# control-plane nodes with etcd quorum 2/3 must never reboot themselves
|
||||
# independently, which is what Automatic-Reboot "true" would mean.
|
||||
unattended_upgrades_enabled: true
|
||||
@@ -1,36 +0,0 @@
|
||||
---
|
||||
# Automatic Debian SECURITY patching for the k3s nodes — the package half of the automated upgrade
|
||||
# process. The reboot half is kured (platform/kured), reconciled by Flux; the two are designed
|
||||
# together and neither is useful alone. Full rationale for every decision:
|
||||
# roles/unattended_upgrades/defaults/main.yml and that role's README.md.
|
||||
#
|
||||
# TARGET (R-ENV-02): `k3s_hosts` defaults to dev's `k3s_servers`; prod is
|
||||
# ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod
|
||||
# Same pattern node-baseline.yml and node-health.yml established — `hosts:` is templated before
|
||||
# group_vars exist, so the target is an extra var, never a group_vars value.
|
||||
#
|
||||
# The role is a NO-OP unless `unattended_upgrades_enabled` is true, which is set for prod only in
|
||||
# inventory/group_vars/k3s_prod/vars.yml. Running this against dev changes nothing.
|
||||
#
|
||||
# DELIBERATELY NOT IN site.yml. site.yml is the bring-up chain (node-baseline -> k3s ->
|
||||
# etcd-io-priority -> kube-vip -> cilium -> verify -> flux-bootstrap) and is "not run against prod
|
||||
# casually". Automatic patching is not a prerequisite for anything in that chain and nothing in the
|
||||
# chain depends on it, so adding it there would only widen what a cluster rebuild touches. It is
|
||||
# applied on its own, once, and re-applied when its configuration changes.
|
||||
#
|
||||
# SAFE TO RUN ON A LIVE CLUSTER, AT ANY TIME. This playbook installs and configures the upgrader. It
|
||||
# never runs `apt upgrade`, never restarts k3s, and never reboots — the role asserts the last of
|
||||
# those against apt's own effective config rather than trusting the file it just wrote. The first
|
||||
# unattended upgrade happens at the next timer firing (04:00 +/-60m), and any reboot that upgrade
|
||||
# implies is kured's to schedule, one node at a time, inside 03:00-06:00.
|
||||
#
|
||||
# TAGS. `--tags verify` re-runs only the read-only assertions and the per-node summary, which is the
|
||||
# way to answer "is automatic patching actually working on these boxes, and is anything waiting on a
|
||||
# reboot" without rewriting any configuration:
|
||||
# ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify
|
||||
- name: Automatic security upgrades on the k3s nodes (kernel excluded; reboots delegated to kured)
|
||||
hosts: "{{ k3s_hosts | default('k3s_servers') }}"
|
||||
become: true
|
||||
gather_facts: true
|
||||
roles:
|
||||
- unattended_upgrades
|
||||
@@ -107,3 +107,31 @@ k3s_base_domain: "home.keithjackson.dev"
|
||||
|
||||
# Derived, so the realm path is stated once. Override k3s_base_domain, not this.
|
||||
k3s_oidc_issuer_url: "https://keycloak.{{ k3s_base_domain }}/realms/homelab"
|
||||
|
||||
# --- Graceful node shutdown (kubelet) ----------------------------------------------------------
|
||||
# MEASURED 2026-10-17, during the UPS move: graceful node shutdown was NOT configured on any prod
|
||||
# node, so `shutdown -h` did NOT drain pods. systemd stops k3s, containerd goes with it, and every
|
||||
# pod is SIGKILLed where it stands -- including 9 CloudNativePG instances mid-checkpoint, the UniFi
|
||||
# VM, and 25 attached Longhorn volumes. The whole-cluster shutdown that day had to quiesce
|
||||
# everything BY HAND first (runbooks/cluster-maintenance-shutdown.md), and the one volume that did
|
||||
# not finish detaching came back `degraded` and had to rebuild.
|
||||
#
|
||||
# With these set, the kubelet takes a systemd inhibitor lock and, on shutdown, evicts ordinary pods
|
||||
# first and critical ones last, so a plain `shutdown -h`/reboot drains the node. That makes the
|
||||
# manual scale-down in the runbook a belt-and-braces step for a PLANNED whole-cluster shutdown
|
||||
# rather than the only thing standing between a reboot and a hard kill.
|
||||
#
|
||||
# 90s/20s, not larger: the budget is spent on every reboot and node upgrade, and the kubelet waits
|
||||
# the FULL ordinary period before it starts on critical pods. Postgres does a fast shutdown in
|
||||
# well under 90s here, and Longhorn needs only enough time to flush and detach.
|
||||
k3s_kubelet_shutdown_grace_period: "90s"
|
||||
k3s_kubelet_shutdown_grace_period_critical_pods: "20s"
|
||||
|
||||
# *** THE HALF THAT IS EASY TO MISS, AND SILENTLY BREAKS THE OTHER HALF. ***
|
||||
# Graceful node shutdown works by the kubelet holding a logind INHIBITOR DELAY lock. systemd caps
|
||||
# how long any inhibitor may delay shutdown at `InhibitDelayMaxSec`, which DEFAULTS TO 5 SECONDS.
|
||||
# If that cap is below the grace period above, systemd proceeds anyway and the drain is truncated
|
||||
# to ~5s -- i.e. the feature appears configured, reports no error, and still hard-kills pods. So
|
||||
# this MUST be >= k3s_kubelet_shutdown_grace_period; the role asserts it rather than trusting it.
|
||||
# Set a little above the kubelet's total (90s + 20s) to leave headroom.
|
||||
k3s_logind_inhibit_delay_max_sec: 120
|
||||
@@ -15,3 +15,12 @@
|
||||
port: "{{ k3s_api_port }}"
|
||||
timeout: 180
|
||||
listen: Restart k3s
|
||||
|
||||
# logind re-reads InhibitDelayMaxSec only on restart. Restarting logind does NOT log users out or
|
||||
# disturb k3s -- it manages login sessions and inhibitor locks, not workloads. The kubelet picks up
|
||||
# the new cap when it next takes its lock, so no k3s restart is needed for THIS half.
|
||||
- name: Restart logind
|
||||
ansible.builtin.systemd_service:
|
||||
name: systemd-logind
|
||||
state: restarted
|
||||
listen: Restart logind
|
||||
@@ -86,6 +86,16 @@
|
||||
# k3s_base_domain (defaults/main.yml), whose [parity] contract with the cluster's
|
||||
# BASE_DOMAIN is documented there. Dev's default renders the value it already runs, so
|
||||
# config.yaml is unchanged and the Restart handler does not fire.
|
||||
# ── Graceful node shutdown ──────────────────────────────────────────────────────────────
|
||||
# Without these the kubelet takes no systemd inhibitor lock, so `shutdown -h` stops
|
||||
# containerd and SIGKILLs every pod where it stands -- MEASURED on all three prod nodes
|
||||
# 2026-10-17 (defaults/main.yml has the detail). The companion logind drop-in below raises
|
||||
# InhibitDelayMaxSec, WITHOUT WHICH systemd caps the drain at ~5s and these values are
|
||||
# quietly ignored.
|
||||
kubelet-arg:
|
||||
- "shutdown-grace-period={{ k3s_kubelet_shutdown_grace_period }}"
|
||||
- "shutdown-grace-period-critical-pods={{ k3s_kubelet_shutdown_grace_period_critical_pods }}"
|
||||
|
||||
kube-apiserver-arg:
|
||||
- "oidc-issuer-url={{ k3s_oidc_issuer_url }}"
|
||||
- "oidc-client-id=kubernetes"
|
||||
@@ -183,3 +193,46 @@
|
||||
host: "127.0.0.1"
|
||||
port: "{{ k3s_api_port }}"
|
||||
timeout: 180
|
||||
|
||||
# ── Graceful node shutdown: the systemd half — tag `shutdown` ──────────────────────────────────
|
||||
# The kubelet's shutdown-grace-period is ENFORCED BY logind, not by the kubelet: the kubelet holds
|
||||
# an inhibitor DELAY lock, and systemd will wait at most `InhibitDelayMaxSec` for it (default 5s)
|
||||
# before proceeding regardless. A grace period above that cap is therefore silently truncated --
|
||||
# the feature looks configured, logs no error, and still hard-kills pods. Assert the relationship
|
||||
# instead of documenting it, because the failure is invisible.
|
||||
- name: Assert the logind inhibitor cap can actually cover the kubelet grace period
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- (k3s_logind_inhibit_delay_max_sec | int) >= (k3s_kubelet_shutdown_grace_period | regex_replace('[^0-9]', '') | int)
|
||||
fail_msg: >-
|
||||
k3s_logind_inhibit_delay_max_sec ({{ k3s_logind_inhibit_delay_max_sec }}s) is below
|
||||
k3s_kubelet_shutdown_grace_period ({{ k3s_kubelet_shutdown_grace_period }}). systemd would
|
||||
stop waiting for the kubelet's inhibitor lock before the drain finished, so pods would be
|
||||
SIGKILLed anyway and graceful node shutdown would be configured but ineffective.
|
||||
quiet: true
|
||||
tags: [shutdown]
|
||||
|
||||
- name: Ensure /etc/systemd/logind.conf.d exists
|
||||
ansible.builtin.file:
|
||||
path: /etc/systemd/logind.conf.d
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
tags: [shutdown]
|
||||
|
||||
# A drop-in, not an edit of logind.conf: the distro file stays pristine and `state: absent` here
|
||||
# would fully revert the change.
|
||||
- name: Raise the logind inhibitor delay cap so the kubelet can finish draining
|
||||
ansible.builtin.copy:
|
||||
dest: /etc/systemd/logind.conf.d/10-k3s-graceful-shutdown.conf
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: |
|
||||
# Managed by Ansible (roles/k3s_server). See defaults/main.yml.
|
||||
# MUST stay >= kubelet shutdown-grace-period, or the drain is truncated to this value.
|
||||
[Login]
|
||||
InhibitDelayMaxSec={{ k3s_logind_inhibit_delay_max_sec }}
|
||||
notify: Restart logind
|
||||
tags: [shutdown]
|
||||
@@ -1,195 +0,0 @@
|
||||
# roles/unattended_upgrades
|
||||
|
||||
Automatic Debian **security** patching on the k3s nodes. The kernel is excluded, k3s is never
|
||||
restarted, and reboots are delegated to **kured** (`platform/kured`).
|
||||
|
||||
- **Playbook:** `playbooks/unattended-upgrades.yml`
|
||||
- **Enabled by:** `unattended_upgrades_enabled` — role default `false`, set `true` for prod in
|
||||
`inventory/group_vars/k3s_prod/vars.yml`
|
||||
- **Companion component:** `platform/kured` (the reboot half). **Neither is useful alone** — see
|
||||
"The two halves" below.
|
||||
- **Every decision and its reasoning:** `defaults/main.yml`. That file is the argument; this one is
|
||||
the orientation.
|
||||
|
||||
## The two halves, and why they ship together
|
||||
|
||||
| | what it does | what it must NOT do |
|
||||
|---|---|---|
|
||||
| **this role** (Ansible, host) | installs security updates continuously; leaves `/var/run/reboot-required` behind when one needs a reboot | upgrade the kernel; restart k3s; reboot |
|
||||
| **kured** (Flux, cluster) | cordons, drains, reboots and uncordons **one node at a time** when that file exists | decide *whether* a reboot is needed |
|
||||
|
||||
Deploying either alone produces a mechanism that looks healthy and does nothing:
|
||||
|
||||
- **kured alone** watches a file that **nothing on Debian ever creates**. Measured 2026-10-18,
|
||||
after a first attempt to install Ubuntu's `update-notifier-common` failed on all three nodes
|
||||
with *"No package matching 'update-notifier-common' is available"*:
|
||||
- `apt-cache policy update-notifier-common` → `Candidate: (none)` — the package does not exist
|
||||
in Debian; it is Ubuntu-only.
|
||||
- `/usr/share/update-notifier/notify-reboot-required` → absent.
|
||||
- **No `linux-image-*` postinst references `notify-reboot-required`** — calling it is an Ubuntu
|
||||
patch, so even dropping a script at that path would never be invoked.
|
||||
- No hook under `/etc/apt/apt.conf.d` or `/etc/kernel/` mentions `reboot-required`.
|
||||
|
||||
So **this role ships the writer itself** (`/usr/local/sbin/k3s-notify-reboot-required` plus a
|
||||
`DPkg::Post-Invoke` hook) rather than installing a package. Without it kured would sit at 1/1
|
||||
Running reporting `kured_reboot_required=0` forever.
|
||||
- **this role alone** installs patches that a long-running process never picks up, and when a
|
||||
reboot *is* needed there is nothing safe to take it. `Automatic-Reboot "true"` is not the
|
||||
answer: three control-plane nodes with etcd quorum 2/3, each rebooting at its own time, is how
|
||||
the cluster is lost.
|
||||
|
||||
## What it was replacing: nothing
|
||||
|
||||
Measured on all three prod nodes, 2026-10-17:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| `unattended-upgrades` | **not installed** |
|
||||
| `update-notifier-common` | **not installable** — Ubuntu-only, `Candidate: (none)` on trixie |
|
||||
| `needrestart` | **not installed** |
|
||||
| `apt-mark showhold` | **empty** — no package held on any node |
|
||||
| `/etc/apt/preferences.d/` | **empty** — no apt pin declares the backports-kernel intent |
|
||||
| `/var/run/reboot-required` | absent, and uncreatable |
|
||||
|
||||
These nodes had received **no automatic patching since provisioning**.
|
||||
|
||||
## The three decisions
|
||||
|
||||
### 1. Security only
|
||||
|
||||
Allowed origins are `label=Debian-Security` and nothing else. `trixie-updates` is excluded
|
||||
deliberately (not overlooked) — it carries non-security fixes that are useful but are the archive
|
||||
through which an unreviewed change would most plausibly arrive on a control-plane node. The knob to
|
||||
add it is `unattended_upgrades_include_updates`.
|
||||
|
||||
### 2. The kernel is excluded twice over
|
||||
|
||||
**This is the highest-risk part of the change.** These HP EliteDesk 800 G5 minis publish a corrupt
|
||||
EFI Memory Attributes Table on R21 firmware; Linux `< 6.13` does not validate it before reserving
|
||||
memory from its contents, and `k3s-prod-2` twice booted with **2.6 GiB of 16 GB**. Upstream
|
||||
`8fbe4c49c0cc` (first in v6.13) is the only fix — there is no kernel cmdline workaround. Full
|
||||
write-up: `infra/PROD-PROVISIONING.md` §0b and `roles/node_baseline/defaults/main.yml:21-51`.
|
||||
|
||||
Measured 2026-10-17: all three nodes run `7.1.8+deb13-amd64` (`7.1.8-1~bpo13+1`, i.e. the backports
|
||||
metapackage), comfortably above the floor — **but `linux-image-6.12.107+deb13-amd64` and
|
||||
`linux-image-6.12.94+deb13-amd64` are also still installed, below the floor, bootable, and held by
|
||||
nothing.**
|
||||
|
||||
So an unattended process must not be able to touch the kernel at all:
|
||||
|
||||
1. **Origin exclusion (structural).** The running kernel comes from `o=Debian Backports` (measured
|
||||
via `apt-cache policy`, pin priority 100 — the `NotAutomatic` + `ButAutomaticUpgrades`
|
||||
signature). That origin is not in `Origins-Pattern`, so unattended-upgrades cannot see a kernel
|
||||
upgrade at all.
|
||||
2. **Package blacklist (belt and braces).** `^linux-` is blacklisted, so even if the origin list is
|
||||
later widened — the most likely future edit to this role, and the one that would silently re-arm
|
||||
the hazard — the kernel is still untouchable.
|
||||
|
||||
The role **asserts both** against `apt-config dump` (apt's own parse of the whole
|
||||
`/etc/apt/apt.conf.d/` directory) rather than trusting the file it just wrote.
|
||||
|
||||
**This does not stop the kernel moving.** The operator still advances it deliberately with
|
||||
|
||||
```
|
||||
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel --limit <host>
|
||||
```
|
||||
|
||||
which is the only thing that maintains the backports source, the metapackage and the on-node
|
||||
provenance stamp. `node_baseline`'s own note that `apt upgrade` keeps the backports kernel current
|
||||
still holds for an operator-run apt. What changed is that a **new unattended actor** is forbidden
|
||||
from it.
|
||||
|
||||
Also blacklisted, for the same "operator-only" reason: `intel-microcode`, `amd64-microcode`,
|
||||
`grub-*`. Microcode and bootloader changes take effect on reboot and alter boot behaviour on the
|
||||
very firmware that is already misbehaving here.
|
||||
|
||||
`Remove-Unused-Kernel-Packages` is **false**, against the Debian default: those older kernels are
|
||||
the documented rollback path, and an unattended process pruning them could remove the fallback an
|
||||
operator needs in front of a box that will not boot.
|
||||
|
||||
### 3. needrestart restarts services — but never k3s
|
||||
|
||||
A security-only policy that restarts nothing patches bytes on disk that no running process reads.
|
||||
`needrestart` in mode `a` restarts services still mapped to a deleted library, which is what makes
|
||||
the policy actually effective.
|
||||
|
||||
`k3s.service` is blacklisted. Restarting the Kubernetes control plane unattended at 04:00 is a
|
||||
worse outcome than a delayed library fix; that fix lands at the next kured reboot instead. Measured
|
||||
2026-10-17, `k3s.service` is the **only** matching unit — k3s bundles its own containerd and
|
||||
kubelet, and neither `containerd` nor `containerd.io` is an apt package on these nodes — so this
|
||||
one entry is complete today.
|
||||
|
||||
## Schedule
|
||||
|
||||
`apt-daily` at 03:30 and `apt-daily-upgrade` at 04:00, both `±60m` randomised, via systemd drop-ins.
|
||||
Chosen to land **inside** kured's 03:00–06:00 reboot window so that a sentinel written by an upgrade
|
||||
can be acted on the same night rather than ~24h later.
|
||||
|
||||
Two traps the role handles explicitly:
|
||||
|
||||
- `/etc/apt/apt.conf.d/20auto-upgrades` must set the four `APT::Periodic::*` counters. Without
|
||||
them the apt-daily timers fire and **do nothing** — installed, configured, green, and inert.
|
||||
- The timer drop-ins clear `OnCalendar=` before setting it. systemd **accumulates** `OnCalendar`
|
||||
entries across drop-ins, so omitting the reset keeps the shipped schedule *in addition* to ours.
|
||||
|
||||
## Why a new role rather than extending `node_baseline`
|
||||
|
||||
`node_baseline` is the repo's only other apt surface and would have been a defensible home. Two
|
||||
reasons against:
|
||||
|
||||
1. **Single responsibility.** `node_baseline` is the *first-boot floor* — it runs before k3s exists
|
||||
and is the thing that makes a node installable. Continuous patching policy is a different
|
||||
lifecycle with a different blast radius, and it wants to be runnable (and re-runnable) without
|
||||
going anywhere near that role's data-disk tasks.
|
||||
2. **Concurrent work.** Other open branches touch `roles/node_baseline/`. A new role has nothing to
|
||||
merge.
|
||||
|
||||
Nothing here duplicates `node_baseline`: it does not add apt sources, does not touch the kernel, and
|
||||
does not manage disks.
|
||||
|
||||
## Post-merge verification
|
||||
|
||||
Running the role is safe on a live cluster at any time — it installs and configures the upgrader, and
|
||||
never runs `apt upgrade`, restarts k3s, or reboots.
|
||||
|
||||
```bash
|
||||
# Apply (prod). Idempotent; re-runnable.
|
||||
cd infra/ansible && ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod
|
||||
|
||||
# Re-run ONLY the read-only assertions + the per-node pending-reboot summary.
|
||||
cd infra/ansible && ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify
|
||||
```
|
||||
|
||||
Then, on a node, confirm the three properties that matter (all read-only, no sudo needed for the
|
||||
first two):
|
||||
|
||||
```bash
|
||||
ssh [email protected] 'apt-config dump | grep -iE "unattended-upgrade::(automatic-reboot|package-blacklist|origins)"'
|
||||
ssh [email protected] 'systemctl list-timers apt-daily-upgrade.timer --no-pager'
|
||||
ssh [email protected] 'ls -la /var/run/reboot-required* 2>&1 | head -2
|
||||
ls -l /usr/local/sbin/k3s-notify-reboot-required /etc/apt/apt.conf.d/99-k3s-reboot-required 2>&1
|
||||
dpkg -l needrestart unattended-upgrades | tail -3'
|
||||
```
|
||||
|
||||
Expect: `Automatic-Reboot "false"`, a blacklist containing `^linux-`, an `Origins-Pattern`
|
||||
containing `Debian-Security` and **not** `Backports`, a next-elapse around 04:00, and all three
|
||||
packages `ii`.
|
||||
|
||||
A dry run of what the upgrader would actually do (needs root, so this one is an operator step):
|
||||
|
||||
```bash
|
||||
sudo unattended-upgrade --dry-run --debug
|
||||
```
|
||||
|
||||
The kernel must appear in its blacklist output, not its upgrade list.
|
||||
|
||||
## What this role cannot tell you
|
||||
|
||||
- **That a security upgrade has ever succeeded.** That is in
|
||||
`/var/log/unattended-upgrades/unattended-upgrades.log` on each node, and nothing in this repo or
|
||||
in Prometheus reads it. There is no alert for "automatic patching has silently stopped working".
|
||||
That is a real, known gap — see the handoff entry
|
||||
`decisions/2026-10-17-automated-upgrades.md`.
|
||||
- **That a reboot actually happened.** That is kured's half, and it *is* alerted on
|
||||
(`KuredRebootPending`, `KuredNodeCoverageGap`, `KuredNotReporting` in
|
||||
`platform/kured/alerts.yaml`).
|
||||
@@ -1,199 +0,0 @@
|
||||
---
|
||||
# roles/unattended_upgrades — automatic Debian SECURITY patching on the k3s nodes, with the kernel
|
||||
# deliberately excluded and with reboots delegated to kured (platform/kured).
|
||||
#
|
||||
# OFF BY DEFAULT, ON FOR PROD. Same seam as node_baseline_backports_kernel / etcd_io_priority_enabled:
|
||||
# a plain boolean, default false, turned on in inventory/group_vars/k3s_prod/vars.yml. Not a hostname
|
||||
# test — that would encode the environment into the role (see node_baseline's defaults for the full
|
||||
# argument, which this role follows rather than restates).
|
||||
#
|
||||
# ═════════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# STATE BEFORE THIS ROLE (measured on all three prod nodes, 2026-10-17, via `dpkg-query -W`)
|
||||
# ═════════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# unattended-upgrades NOT INSTALLED
|
||||
# update-notifier-common NOT INSTALLED
|
||||
# needrestart NOT INSTALLED
|
||||
# apt-mark showhold EMPTY (no package is held on any node)
|
||||
# /var/run/reboot-required ABSENT, and nothing on the node can create it
|
||||
# /etc/apt/preferences.d/ EMPTY (no apt pin declares the backports-kernel intent)
|
||||
# So: zero automatic patching, and no mechanism by which a reboot could ever be signalled. That last
|
||||
# point is why this role and platform/kured must land together — kured watches
|
||||
# /var/run/reboot-required, and on these nodes today NOTHING WRITES IT. Deploying kured alone would
|
||||
# produce a daemon that is permanently, silently idle while looking perfectly healthy.
|
||||
#
|
||||
# ═════════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# DECISION 1 — SECURITY ONLY, NOT FULL UPGRADES
|
||||
# ═════════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# Measured `apt-cache policy` on k3s-prod-1, 2026-10-17, gives the four archives and their pins:
|
||||
# trixie o=Debian, a=stable, l=Debian pin 500
|
||||
# trixie-updates o=Debian, a=stable-updates, l=Debian pin 500
|
||||
# trixie-security o=Debian, a=stable-security, l=Debian-Security pin 500
|
||||
# trixie-backports o=Debian Backports, a=stable-backports, l=Debian Backports pin 100
|
||||
#
|
||||
# Only `Debian-Security` is allowed below. The reasoning, in order of weight:
|
||||
#
|
||||
# * Security updates are the class where the cost of NOT applying them is unbounded and the
|
||||
# change itself is minimal by Debian policy (targeted fixes against the frozen stable version,
|
||||
# no feature churn). It is the one class worth applying with nobody watching.
|
||||
# * A full `apt upgrade` on these nodes would also pull trixie point-release changes, which is a
|
||||
# larger surface than anyone has reviewed for a 3-node etcd cluster, for no urgency.
|
||||
# * `trixie-updates` is DELIBERATELY EXCLUDED, not overlooked. It carries non-security fixes that
|
||||
# are too urgent to wait for a point release — useful, but it is the archive through which an
|
||||
# unreviewed change would most plausibly arrive, and it is not what "unattended" should mean on
|
||||
# a control-plane node. Add it consciously later if the operator wants it; the knob is below.
|
||||
# * `trixie-backports` is EXCLUDED, AND THAT IS THE KERNEL ARGUMENT — see DECISION 2.
|
||||
#
|
||||
# ═════════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# DECISION 2 — THE KERNEL IS EXCLUDED TWICE OVER. THIS IS THE HIGHEST-RISK PART OF THE CHANGE.
|
||||
# ═════════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# The constraint: HP's R21 firmware on these EliteDesk 800 G5 minis publishes a corrupt EFI Memory
|
||||
# Attributes Table, and Linux < 6.13 does not validate it before reserving memory from its contents
|
||||
# — k3s-prod-2 twice booted with 2.6 GiB of 16 GB. Upstream 8fbe4c49c0cc (first in v6.13) is the
|
||||
# ONLY fix; there is no kernel cmdline workaround. So these nodes run the backports metapackage
|
||||
# `linux-image-amd64` (measured 2026-10-17: version 7.1.8-1~bpo13+1 installed and running on all
|
||||
# three, well above the floor). Full write-up: infra/PROD-PROVISIONING.md §0b, and the rationale
|
||||
# block in roles/node_baseline/defaults/main.yml:21-51.
|
||||
#
|
||||
# A NODE THAT COMES BACK FROM AN UNATTENDED UPGRADE ON A 6.12 KERNEL IS A NODE THAT MAY COME BACK
|
||||
# WITH A FRACTION OF ITS RAM. Note that 6.12 kernels are not hypothetical here — measured
|
||||
# 2026-10-17, `linux-image-6.12.107+deb13-amd64` and `linux-image-6.12.94+deb13-amd64` are BOTH
|
||||
# still installed on all three nodes, below the floor, bootable, and nothing holds them.
|
||||
#
|
||||
# Two independent mechanisms keep unattended-upgrades away from the kernel, and the role asserts
|
||||
# both rather than trusting either:
|
||||
#
|
||||
# 1. ORIGIN EXCLUSION (structural). The running kernel comes from `o=Debian Backports`, which is
|
||||
# not in Allowed-Origins. u-u only ever considers packages from allowed origins, so it cannot
|
||||
# see a backports kernel upgrade at all. This also means the backports kernel receives no
|
||||
# automatic patching — accepted, and in any case backports carries no security guarantees
|
||||
# (node_baseline's defaults state that cost plainly: "no security-tracker commitment, no DSAs").
|
||||
# 2. PACKAGE BLACKLIST (belt and braces). `linux-*` is blacklisted below, so even if the origin
|
||||
# list is later widened — the most likely future edit to this file, and the one that would
|
||||
# silently re-arm the hazard — the kernel is still untouchable. Without this, widening origins
|
||||
# to include backports would hand an unattended process the EFI-memattr-critical package.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* DO, AND MUST NOT BE READ AS DOING: it does not stop the kernel moving. The
|
||||
# operator still advances it deliberately with
|
||||
# ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel --limit <host>
|
||||
# which is the only thing that maintains the backports source, the metapackage and the on-node
|
||||
# provenance stamp. node_baseline's own comment ("`ButAutomaticUpgrades: yes` ... means `apt upgrade`
|
||||
# keeps it current once installed") still holds for an operator-run apt; the measured pin priority of
|
||||
# 100 is exactly the NotAutomatic + ButAutomaticUpgrades signature that produces that behaviour.
|
||||
# Nothing here changes it. What changes is that a NEW unattended actor is forbidden from the kernel.
|
||||
#
|
||||
# NOT BLACKLISTED, BECAUSE THEY ARE NOT APT PACKAGES AT ALL — measured 2026-10-17: `dpkg-query` finds
|
||||
# no `k3s`, no `containerd` and no `containerd.io`. k3s is a single static binary at
|
||||
# /usr/local/bin/k3s installed by roles/k3s_server, and it bundles its own containerd and kubelet.
|
||||
# So the entire Kubernetes control plane and container runtime are OUTSIDE apt's reach and cannot be
|
||||
# upgraded by this role even in principle. That is worth knowing before worrying about it.
|
||||
unattended_upgrades_enabled: false
|
||||
|
||||
# The packages this role installs. Each one is load-bearing; see the notes.
|
||||
unattended_upgrades_packages:
|
||||
# The upgrader itself.
|
||||
- unattended-upgrades
|
||||
# NOTE: `update-notifier-common` was listed here and REMOVED (2026-10-18). It is an UBUNTU
|
||||
# package and does not exist in Debian at all -- `apt-cache policy update-notifier-common` on
|
||||
# trixie returns `Candidate: (none)`, so the play failed outright with "No package matching
|
||||
# 'update-notifier-common' is available" on all three nodes. The sentinel writer is now provided
|
||||
# by this role itself; see unattended_upgrades_sentinel_* below and tasks/main.yml.
|
||||
# Restarts services still mapped to a deleted library after a security upgrade. Without it a
|
||||
# security-only policy patches bytes on disk that no running process reads — see DECISION 3.
|
||||
- needrestart
|
||||
|
||||
# --- Allowed origins -----------------------------------------------------------------------------
|
||||
# Verbatim apt origin expressions, substituted into 50unattended-upgrades. The ${distro_*} tokens
|
||||
# here are UNATTENDED-UPGRADES' OWN placeholders, expanded by it at runtime — they are not Ansible
|
||||
# or Flux substitutions, which is why they are single-quoted and why this role's files never travel
|
||||
# through Flux envsubst.
|
||||
#
|
||||
# `l=Debian-Security` plus the `-security` codename is the form Debian's own shipped config uses, and
|
||||
# both halves were confirmed against the measured archive metadata above (l=Debian-Security,
|
||||
# n=trixie-security). Matching on the LABEL rather than the suite name is what makes this survive the
|
||||
# next Debian release.
|
||||
unattended_upgrades_allowed_origins:
|
||||
- 'origin=Debian,codename=${distro_codename},label=Debian-Security'
|
||||
- 'origin=Debian,codename=${distro_codename}-security,label=Debian-Security'
|
||||
# Set this to add 'origin=Debian,codename=${distro_codename}-updates,label=Debian' — a deliberate
|
||||
# widening, kept as a knob so that turning it on is a reviewable one-line change rather than an edit
|
||||
# to a heredoc. See DECISION 1 for why it is off.
|
||||
unattended_upgrades_include_updates: false
|
||||
|
||||
# --- Package blacklist ---------------------------------------------------------------------------
|
||||
# Regular expressions matched against package names. See DECISION 2: `linux-` is the EFI-memattr
|
||||
# guard and must not be removed without re-reading PROD-PROVISIONING.md §0b.
|
||||
unattended_upgrades_blacklist:
|
||||
# Every kernel package: linux-image-*, linux-headers-*, linux-image-amd64 (the metapackage that
|
||||
# actually tracks backports), plus linux-kbuild / linux-libc-dev. Anchored at the start so it
|
||||
# cannot accidentally match an unrelated package containing the substring.
|
||||
- '^linux-'
|
||||
# Microcode changes take effect only on reboot and can alter boot behaviour on the very firmware
|
||||
# that is already misbehaving here. Operator-only, same argument as the kernel.
|
||||
- '^intel-microcode$'
|
||||
- '^amd64-microcode$'
|
||||
# The bootloader. A grub upgrade rewrites boot configuration on a machine with a documented
|
||||
# warm-reboot hang (k3s-prod-2, 2026-09-03). Never unattended.
|
||||
- '^grub-'
|
||||
- '^grub2-common$'
|
||||
|
||||
# --- Reboot policy -------------------------------------------------------------------------------
|
||||
# FALSE, AND THIS IS THE DIVISION OF LABOUR WITH kured, NOT AN OVERSIGHT.
|
||||
# unattended-upgrades can reboot a node itself (Automatic-Reboot "true"). It must not here: it has
|
||||
# no notion of a cluster, so it would not cordon, would not drain, would not evict the 25 attached
|
||||
# Longhorn volumes cleanly, would not check whether another node is already down, and — on three
|
||||
# control-plane nodes where etcd quorum is 2 of 3 — three nodes each deciding independently to
|
||||
# reboot at their own `Automatic-Reboot-Time` is precisely how you lose the cluster.
|
||||
# u-u therefore only ever leaves the sentinel behind; kured owns every reboot, one node at a time,
|
||||
# under a cluster-wide lock. See platform/kured/helmrelease.yaml.
|
||||
unattended_upgrades_automatic_reboot: false
|
||||
|
||||
# --- needrestart (DECISION 3) --------------------------------------------------------------------
|
||||
# See the role README for the full argument. Summary: 'a' (restart services automatically) is what
|
||||
# makes a security-only policy actually effective — a patched libssl on disk does nothing for a
|
||||
# process still mapping the old one — but k3s.service is EXCLUDED, because restarting the
|
||||
# Kubernetes control plane unattended at 06:00 is a worse outcome than a delayed library fix.
|
||||
# Measured 2026-10-17: the only matching unit on these nodes is `k3s.service` (no separate
|
||||
# containerd or kubelet unit exists; k3s bundles both), so this blacklist is complete today.
|
||||
unattended_upgrades_needrestart_mode: a
|
||||
unattended_upgrades_needrestart_blacklist:
|
||||
- '^k3s'
|
||||
|
||||
# --- Schedule ------------------------------------------------------------------------------------
|
||||
# apt's own systemd timers (apt-daily, apt-daily-upgrade) drive u-u; these set their windows.
|
||||
# 04:00 with a 1h randomised delay, so the upgrade lands INSIDE kured's 03:00-06:00 reboot window
|
||||
# (platform/kured/helmrelease.yaml) rather than hours away from it: if an upgrade ever does set the
|
||||
# sentinel, the reboot can be taken the same night instead of waiting ~24h. The randomised delay
|
||||
# keeps the three nodes from hitting the mirror simultaneously; it does NOT serialise anything, and
|
||||
# it does not need to — u-u never reboots, so three concurrent upgrades are harmless.
|
||||
unattended_upgrades_download_time: "03:30"
|
||||
unattended_upgrades_upgrade_time: "04:00"
|
||||
unattended_upgrades_randomised_delay: 60m
|
||||
|
||||
|
||||
# ── THE REBOOT SENTINEL, PROVIDED BY THIS ROLE ──────────────────────────────────────────────────
|
||||
# /var/run/reboot-required is the ONLY thing kured watches, and on Debian trixie **nothing creates
|
||||
# it**. Measured on all three nodes 2026-10-18:
|
||||
# * `update-notifier-common` (which owns that hook on Ubuntu) has `Candidate: (none)` — the
|
||||
# package does not exist in Debian. Installing it is what broke the first run of this play.
|
||||
# * `/usr/share/update-notifier/notify-reboot-required` is absent.
|
||||
# * NO `linux-image-*` postinst references `notify-reboot-required` — calling it is an Ubuntu
|
||||
# patch, not Debian behaviour, so merely dropping a script at that path would never be invoked.
|
||||
# * No hook under /etc/apt/apt.conf.d or /etc/kernel/ mentions reboot-required.
|
||||
# So the writer has to be ours. Without it kured is a permanent, INVISIBLE no-op: it stays 1/1
|
||||
# Running and reports kured_reboot_required=0 forever, which is the "failure mode is a green tick"
|
||||
# shape this repo keeps getting burned by.
|
||||
#
|
||||
# MECHANISM: a dpkg Post-Invoke hook runs a script that compares the RUNNING kernel against the
|
||||
# newest INSTALLED one and touches the sentinel when they differ. Deliberately kernel-only, which
|
||||
# matches the design: unattended-upgrades is security-only with `^linux-` blacklisted, so it will
|
||||
# essentially never set this — the sentinel appears when the OPERATOR deliberately runs the
|
||||
# node_baseline kernel play, and kured then automates only the mechanical reboot.
|
||||
#
|
||||
# It uses `linux-version` (from `linux-base`, a dependency of every linux-image package, so
|
||||
# guaranteed present — measured `linux-base 4.12.1` installed) rather than parsing `dpkg -l`
|
||||
# output, because kernel version ordering is not string ordering.
|
||||
#
|
||||
# /run is a tmpfs (measured), so the sentinel self-clears on boot: after a reboot the running
|
||||
# kernel IS the newest and the script writes nothing. No cleanup task is needed, and none exists.
|
||||
unattended_upgrades_sentinel_script: /usr/local/sbin/k3s-notify-reboot-required
|
||||
unattended_upgrades_sentinel_hook: /etc/apt/apt.conf.d/99-k3s-reboot-required
|
||||
@@ -1,29 +0,0 @@
|
||||
---
|
||||
# Re-arm apt's periodic timers after a config or drop-in change.
|
||||
#
|
||||
# WHY A daemon_reload IS REQUIRED AND NOT OPTIONAL. The timer windows are set by drop-ins under
|
||||
# /etc/systemd/system/apt-daily{,-upgrade}.timer.d/. systemd does not re-read unit files on its own,
|
||||
# so until it is told to, the RUNNING manager keeps the shipped schedule and the new window is
|
||||
# ignored — the files look right in git and on disk, `systemctl cat` shows them, and the upgrade
|
||||
# still fires at the old time. That is a silent config no-op, which is the failure mode this whole
|
||||
# role is written against.
|
||||
#
|
||||
# `daemon_reload` re-parses unit files and re-runs generators. It starts and stops nothing, so it
|
||||
# cannot disturb k3s, containerd or Longhorn — the same reasoning node_baseline's "Reload systemd
|
||||
# unit files" handler records.
|
||||
#
|
||||
# RESTARTING THE TIMERS IS ALSO SAFE, and is not the same as running the upgrade. `*.timer` units
|
||||
# only hold the schedule; restarting one recomputes its next elapse and does NOT trigger its
|
||||
# `*.service`. With `Persistent=true` set in the drop-in, a window missed while the node was down is
|
||||
# caught up on the next boot rather than skipped.
|
||||
#
|
||||
# Both fire only when a task actually changed, so a re-run of the role is silent.
|
||||
- name: Restart unattended-upgrades timers
|
||||
ansible.builtin.systemd_service:
|
||||
name: "{{ item }}"
|
||||
state: restarted
|
||||
enabled: true
|
||||
daemon_reload: true
|
||||
loop:
|
||||
- apt-daily.timer
|
||||
- apt-daily-upgrade.timer
|
||||
@@ -1,369 +0,0 @@
|
||||
---
|
||||
# roles/unattended_upgrades — automatic Debian security patching, kernel excluded, reboots delegated
|
||||
# to kured. Every decision and its reasoning lives in defaults/main.yml; this file is the mechanism.
|
||||
#
|
||||
# TAGS — following roles/node_baseline, every task carries one of `packages`, `config` or `verify`.
|
||||
# An untagged run does all three in that order. The split exists because `verify` is the half an
|
||||
# operator will want to re-run on its own to answer "is automatic patching actually working on this
|
||||
# box", without re-writing any configuration:
|
||||
# ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify
|
||||
#
|
||||
# IT DOES NOT UPGRADE ANYTHING, AND IT DOES NOT REBOOT. This role installs and CONFIGURES the
|
||||
# upgrader; it never runs `apt upgrade` itself and never takes a reboot. Both of those are deliberate
|
||||
# and both follow roles/node_baseline/tasks/main.yml:61-67 ("IT DOES NOT REBOOT", after k3s-prod-2
|
||||
# hung on a warm reboot on 2026-09-03 and needed a physical power button). Applying this role is
|
||||
# therefore a SAFE operation on a live cluster at any time of day: the first unattended upgrade
|
||||
# happens at the next timer firing, and any reboot it implies is kured's to schedule inside the
|
||||
# 03:00-06:00 window.
|
||||
- name: Automatic security upgrades (unattended-upgrades; kernel excluded, reboots via kured)
|
||||
when: unattended_upgrades_enabled | bool
|
||||
tags: [packages, config, verify]
|
||||
block:
|
||||
|
||||
# ── PREFLIGHT ───────────────────────────────────────────────────────────────────────────────
|
||||
- name: Assert this is a Debian-family host
|
||||
# The config file layout, the origin expressions and the package names below are all
|
||||
# Debian-specific. On anything else this role would write files that are silently ignored,
|
||||
# which is the failure mode this whole PR exists to avoid.
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- ansible_facts['os_family'] == 'Debian'
|
||||
fail_msg: >-
|
||||
REFUSING TO CONFIGURE unattended-upgrades on {{ inventory_hostname }}: os_family is
|
||||
'{{ ansible_facts["os_family"] }}', not 'Debian'. The allowed-origin expressions and
|
||||
config paths in this role are Debian-specific.
|
||||
tags: [packages, config, verify]
|
||||
|
||||
# ── PACKAGES ────────────────────────────────────────────────────────────────────────────────
|
||||
- name: Install unattended-upgrades and needrestart
|
||||
# The reboot-sentinel writer is NOT a package here. On Debian nothing ships one -- see
|
||||
# defaults/main.yml -- so this role installs its own script and dpkg hook, two tasks below.
|
||||
ansible.builtin.apt:
|
||||
name: "{{ unattended_upgrades_packages }}"
|
||||
state: present
|
||||
update_cache: true
|
||||
cache_valid_time: 3600
|
||||
tags: [packages]
|
||||
|
||||
# ── THE REBOOT SENTINEL WRITER (ours, because Debian ships none) ────────────────────────────
|
||||
# Full rationale and the four measurements behind it: defaults/main.yml. Short version: kured
|
||||
# watches /var/run/reboot-required and nothing on Debian creates it, so without these two files
|
||||
# the entire reboot half is an invisible no-op.
|
||||
- name: Install the reboot-sentinel writer
|
||||
ansible.builtin.copy:
|
||||
dest: "{{ unattended_upgrades_sentinel_script }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
content: |
|
||||
#!/bin/sh
|
||||
# MANAGED BY ANSIBLE -- infra/ansible/roles/unattended_upgrades. Local edits are reverted.
|
||||
#
|
||||
# Touch /var/run/reboot-required when the RUNNING kernel is not the newest INSTALLED one.
|
||||
# kured watches that path; see the role's defaults/main.yml for why this is ours to ship.
|
||||
#
|
||||
# `linux-version` comes from linux-base (a dependency of every linux-image package), and
|
||||
# is used rather than sorting `dpkg -l` output because kernel version ordering is not
|
||||
# string ordering. If it is somehow missing, exit 0 and write nothing: a dpkg hook must
|
||||
# never fail an apt transaction, and a missing sentinel is caught by the verify assert.
|
||||
set -eu
|
||||
command -v linux-version >/dev/null 2>&1 || exit 0
|
||||
|
||||
running="$(uname -r)"
|
||||
newest="$(linux-version list 2>/dev/null | linux-version sort --reverse 2>/dev/null | head -n1)"
|
||||
|
||||
# No newest (nothing installed we can read) -> nothing to say.
|
||||
[ -n "$newest" ] || exit 0
|
||||
|
||||
if [ "$running" != "$newest" ]; then
|
||||
touch /var/run/reboot-required 2>/dev/null || exit 0
|
||||
# kured only needs the file, but the .pkgs companion is what an operator reads to find
|
||||
# out WHY a node is waiting, so write a reason rather than an empty marker.
|
||||
printf 'linux-image-%s\n' "$newest" > /var/run/reboot-required.pkgs 2>/dev/null || true
|
||||
fi
|
||||
exit 0
|
||||
tags: [packages, config]
|
||||
|
||||
- name: Install the dpkg hook that runs the sentinel writer
|
||||
# Post-Invoke, so it runs after dpkg has finished and a newly installed kernel is already
|
||||
# visible to `linux-version list`. `|| true` so a failure here can never abort an apt
|
||||
# transaction -- an unwritten sentinel is a missed reboot, but a broken apt is a broken node.
|
||||
ansible.builtin.copy:
|
||||
dest: "{{ unattended_upgrades_sentinel_hook }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: |
|
||||
// MANAGED BY ANSIBLE -- infra/ansible/roles/unattended_upgrades. Local edits reverted.
|
||||
// Writes /var/run/reboot-required when the running kernel is not the newest installed.
|
||||
DPkg::Post-Invoke { "{{ unattended_upgrades_sentinel_script }} || true"; };
|
||||
tags: [packages, config]
|
||||
|
||||
# ── CONFIG ──────────────────────────────────────────────────────────────────────────────────
|
||||
# Written as a whole file rather than patched in place, for the reason node_baseline gives for
|
||||
# its own apt source: an idempotent full-file write with a managed-by header is reviewable and
|
||||
# re-runnable, whereas line-editing a conffile fights dpkg on every package upgrade.
|
||||
#
|
||||
# 52- RATHER THAN 50-: Debian ships /etc/apt/apt.conf.d/50unattended-upgrades as a CONFFILE of
|
||||
# the unattended-upgrades package. Overwriting it would make every future apt upgrade of that
|
||||
# package prompt about a locally-modified conffile (or, under the non-interactive defaults a
|
||||
# scripted run uses, silently keep one side and discard the other). apt reads the directory in
|
||||
# lexical order and later files win, so a separate 52- file overrides the shipped defaults
|
||||
# without ever touching them, and `apt-config dump` in the verify block below proves which value
|
||||
# actually won rather than assuming this ordering holds.
|
||||
- name: Configure unattended-upgrades (52-k3s-unattended-upgrades)
|
||||
ansible.builtin.copy:
|
||||
dest: /etc/apt/apt.conf.d/52-k3s-unattended-upgrades
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: |
|
||||
// MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades. Local edits are reverted
|
||||
// on the next run. Rationale for every value: that role's defaults/main.yml.
|
||||
//
|
||||
// Overrides the shipped 50unattended-upgrades by lexical ordering (52 > 50). The shipped
|
||||
// file is a dpkg conffile and is deliberately left untouched.
|
||||
|
||||
// SECURITY ONLY. Deliberately excludes trixie-updates (non-security fixes) and
|
||||
// trixie-backports (o=Debian Backports) -- and excluding backports is what makes it
|
||||
// structurally impossible for this upgrader to move the EFI-memattr-critical kernel,
|
||||
// which is installed from there. See defaults/main.yml DECISION 2.
|
||||
Unattended-Upgrade::Origins-Pattern {
|
||||
{% for origin in unattended_upgrades_allowed_origins %}
|
||||
"{{ origin }}";
|
||||
{% endfor %}
|
||||
{% if unattended_upgrades_include_updates | bool %}
|
||||
"origin=Debian,codename=${distro_codename}-updates,label=Debian";
|
||||
{% endif %}
|
||||
};
|
||||
|
||||
// NEVER AUTOMATICALLY UPGRADED, whatever the origin list later says. `^linux-` is the
|
||||
// EFI-memattr guard: these HP minis need kernel >= 6.13 or they can boot with a fraction
|
||||
// of their RAM, and 6.12 kernels are still installed and bootable on all three nodes.
|
||||
// Do not remove without re-reading infra/PROD-PROVISIONING.md §0b.
|
||||
Unattended-Upgrade::Package-Blacklist {
|
||||
{% for pkg in unattended_upgrades_blacklist %}
|
||||
"{{ pkg }}";
|
||||
{% endfor %}
|
||||
};
|
||||
|
||||
// REBOOTS BELONG TO kured (platform/kured), NOT TO apt. All three nodes are
|
||||
// control-plane + etcd with quorum 2 of 3, so three hosts independently deciding to
|
||||
// reboot at their own Automatic-Reboot-Time is how the cluster is lost. u-u leaves
|
||||
// /var/run/reboot-required behind; kured takes a cluster-wide lock and reboots exactly
|
||||
// one node at a time, after cordon + drain, inside 03:00-06:00, gated on Longhorn volume
|
||||
// health.
|
||||
Unattended-Upgrade::Automatic-Reboot "{{ unattended_upgrades_automatic_reboot | bool | lower }}";
|
||||
Unattended-Upgrade::Automatic-Reboot-WithUsers "false";
|
||||
|
||||
// Keep the node's disk from filling with superseded .debs, and keep a log an operator can
|
||||
// read after the fact (/var/log/unattended-upgrades/). MinAge guards against removing a
|
||||
// package still needed by a rollback.
|
||||
Unattended-Upgrade::Remove-Unused-Kernel-Packages "false";
|
||||
Unattended-Upgrade::Remove-New-Unused-Dependencies "true";
|
||||
Unattended-Upgrade::Remove-Unused-Dependencies "false";
|
||||
Unattended-Upgrade::AutoFixInterruptedDpkg "true";
|
||||
Unattended-Upgrade::MinimalSteps "true";
|
||||
Unattended-Upgrade::SyslogEnable "true";
|
||||
|
||||
// Remove-Unused-Kernel-Packages is FALSE on purpose, against the Debian default. These
|
||||
// nodes deliberately carry more than one kernel: the backports kernel they boot and the
|
||||
// distro kernels that are the documented rollback path (node_baseline writes the rollback
|
||||
// procedure onto the node itself). An unattended process pruning kernels could remove the
|
||||
// fallback an operator needs at 3am in front of a box that will not boot.
|
||||
notify: Restart unattended-upgrades timers
|
||||
tags: [config]
|
||||
|
||||
- name: Enable the apt periodic timers that actually run unattended-upgrades
|
||||
# Installing the package is NOT sufficient to make it run on a schedule: without these
|
||||
# counters apt's daily units do nothing, and the whole mechanism is inert while looking
|
||||
# installed. `1` means "every day".
|
||||
ansible.builtin.copy:
|
||||
dest: /etc/apt/apt.conf.d/20auto-upgrades
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: |
|
||||
// MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades.
|
||||
// Without these four counters the apt-daily timers fire and do nothing, so
|
||||
// unattended-upgrades would be installed, configured, and never once invoked.
|
||||
APT::Periodic::Update-Package-Lists "1";
|
||||
APT::Periodic::Download-Upgradeable-Packages "1";
|
||||
APT::Periodic::Unattended-Upgrade "1";
|
||||
APT::Periodic::AutocleanInterval "7";
|
||||
tags: [config]
|
||||
|
||||
- name: Ensure the systemd timer drop-in directories exist
|
||||
# `copy` does not create parent directories, and these two .timer.d/ dirs do not exist on a
|
||||
# stock Debian node — without this task the next task fails with "Destination directory does
|
||||
# not exist" on a first run.
|
||||
ansible.builtin.file:
|
||||
path: "/etc/systemd/system/{{ item }}.timer.d"
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
loop:
|
||||
- apt-daily
|
||||
- apt-daily-upgrade
|
||||
tags: [config]
|
||||
|
||||
- name: Pin the apt timer windows into kured's reboot window
|
||||
# Overriding the shipped timers rather than editing them: a drop-in survives package upgrades.
|
||||
# OnCalendar is cleared first (`OnCalendar=` with no value) because systemd ACCUMULATES
|
||||
# OnCalendar entries across drop-ins — omit the reset and the unit keeps its shipped schedule
|
||||
# IN ADDITION to ours, which would run the upgrade at both times and look like it worked.
|
||||
ansible.builtin.copy:
|
||||
dest: "/etc/systemd/system/{{ item.unit }}.timer.d/override.conf"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: |
|
||||
# MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades.
|
||||
# Lands the upgrade inside kured's 03:00-06:00 reboot window so a sentinel written by an
|
||||
# upgrade can be acted on the same night rather than ~24h later.
|
||||
[Timer]
|
||||
OnCalendar=
|
||||
OnCalendar={{ item.time }}
|
||||
RandomizedDelaySec={{ unattended_upgrades_randomised_delay }}
|
||||
Persistent=true
|
||||
force: true
|
||||
loop:
|
||||
- { unit: apt-daily, time: "{{ unattended_upgrades_download_time }}" }
|
||||
- { unit: apt-daily-upgrade, time: "{{ unattended_upgrades_upgrade_time }}" }
|
||||
loop_control:
|
||||
label: "{{ item.unit }} at {{ item.time }}"
|
||||
notify: Restart unattended-upgrades timers
|
||||
tags: [config]
|
||||
|
||||
- name: Configure needrestart (restart services, but never k3s)
|
||||
# See defaults/main.yml DECISION 3. Auto-restart is what makes a security-only policy real --
|
||||
# a patched library on disk does nothing for a process still mapping the deleted one -- but
|
||||
# k3s.service is excluded because restarting the control plane unattended is worse than a
|
||||
# delayed library fix. `$nrconf{restart}` is Perl config read by needrestart itself.
|
||||
ansible.builtin.copy:
|
||||
dest: /etc/needrestart/conf.d/50-k3s.conf
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: |
|
||||
# MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades.
|
||||
# Restart mode: '{{ unattended_upgrades_needrestart_mode }}' (a = automatically, l = list only, i = interactive)
|
||||
$nrconf{restart} = '{{ unattended_upgrades_needrestart_mode }}';
|
||||
# Units matching these patterns are NEVER restarted automatically. k3s bundles containerd
|
||||
# and the kubelet, so this one entry covers the entire control plane and runtime on these
|
||||
# nodes (measured 2026-10-17: k3s.service is the only matching unit). A library fix that
|
||||
# needs k3s restarted therefore waits for the reboot kured will take.
|
||||
$nrconf{blacklist_rc} = [
|
||||
{% for pat in unattended_upgrades_needrestart_blacklist %}
|
||||
qr({{ pat }}),
|
||||
{% endfor %}
|
||||
];
|
||||
tags: [config]
|
||||
|
||||
# ── VERIFY ──────────────────────────────────────────────────────────────────────────────────
|
||||
# These tasks exist because every component of this mechanism fails SILENTLY. An installed
|
||||
# package, a written config file and a green play are all compatible with nothing ever being
|
||||
# upgraded and no reboot ever being signalled.
|
||||
|
||||
- name: Ask apt-config what the effective settings actually are
|
||||
# `apt-config dump` is apt's OWN parse of the whole /etc/apt/apt.conf.d/ directory, so it
|
||||
# proves what WON after lexical ordering rather than what this role wrote. If the shipped
|
||||
# 50unattended-upgrades were ever renamed above 52-, this is the task that would catch it.
|
||||
ansible.builtin.command: apt-config dump
|
||||
register: unattended_upgrades_aptconfig
|
||||
changed_when: false
|
||||
check_mode: false
|
||||
tags: [verify]
|
||||
|
||||
- name: Assert the kernel is blacklisted and automatic reboots are off
|
||||
# The two properties that must never silently regress. Asserted against apt's own view.
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
# apt-config renders each list entry as `Unattended-Upgrade::Package-Blacklist:: "^linux-";`
|
||||
# so the anchored pattern itself is the thing to look for. Checking for the bare string
|
||||
# `linux-` would pass on almost any dump and prove nothing.
|
||||
- "'\"^linux-\"' in unattended_upgrades_aptconfig.stdout"
|
||||
- "'Unattended-Upgrade::Automatic-Reboot \"false\"' in unattended_upgrades_aptconfig.stdout"
|
||||
# The origin list must be non-empty, or u-u runs nightly and upgrades nothing — installed,
|
||||
# scheduled, green, and inert.
|
||||
- "'Debian-Security' in unattended_upgrades_aptconfig.stdout"
|
||||
fail_msg: >-
|
||||
EFFECTIVE apt CONFIG ON {{ inventory_hostname }} IS NOT WHAT THIS ROLE INTENDED.
|
||||
Either the `^linux-` kernel blacklist or `Automatic-Reboot "false"` is not winning.
|
||||
Both are safety properties, not preferences: without the blacklist an unattended process
|
||||
can move a kernel these HP minis must keep >= 6.13 (PROD-PROVISIONING.md §0b), and
|
||||
without Automatic-Reboot=false three control-plane nodes can each reboot themselves
|
||||
independently and take etcd below quorum. Inspect with `apt-config dump | grep -i
|
||||
unattended` and check for a file sorting after 52- in /etc/apt/apt.conf.d/.
|
||||
success_msg: >-
|
||||
Effective apt config confirmed: kernel blacklisted, automatic reboot disabled.
|
||||
tags: [verify]
|
||||
|
||||
# THE SENTINEL IS THE SINGLE POINT OF FAILURE FOR THE ENTIRE REBOOT HALF OF THIS DESIGN.
|
||||
# This was originally a `find` over /etc/kernel/postinst.d and /etc/apt/apt.conf.d, on the
|
||||
# theory that the writer came from a package and the role could only discover whatever that
|
||||
# package happened to ship. That theory was wrong: no such package exists on Debian (see
|
||||
# defaults/main.yml), so the writer is now OURS and lives at two known paths. Checking two
|
||||
# exact paths is strictly stronger than a glob -- a glob would have matched Debian's own
|
||||
# unattended-upgrades apt conffile and passed while no sentinel writer existed at all.
|
||||
- name: Stat the sentinel writer script
|
||||
ansible.builtin.stat:
|
||||
path: "{{ unattended_upgrades_sentinel_script }}"
|
||||
register: unattended_upgrades_sentinel_script_stat
|
||||
changed_when: false
|
||||
tags: [verify]
|
||||
|
||||
- name: Stat the sentinel dpkg hook
|
||||
ansible.builtin.stat:
|
||||
path: "{{ unattended_upgrades_sentinel_hook }}"
|
||||
register: unattended_upgrades_sentinel_hook_stat
|
||||
changed_when: false
|
||||
tags: [verify]
|
||||
|
||||
- name: Assert something on this node can write the reboot sentinel
|
||||
# If this fails, kured is a no-op: it watches /var/run/reboot-required forever, the file is
|
||||
# never created, no reboot is ever taken, and NOTHING GOES RED -- the DaemonSet stays 1/1
|
||||
# Running and `kured_reboot_required` reports a steady 0 on every node. That is the exact
|
||||
# shape of failure (a check whose failure mode is a green tick) this repo has been burned by
|
||||
# before, so it is an assert and not a debug message.
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- unattended_upgrades_sentinel_script_stat.stat.exists
|
||||
- unattended_upgrades_sentinel_script_stat.stat.executable | default(false)
|
||||
- unattended_upgrades_sentinel_hook_stat.stat.exists
|
||||
fail_msg: >-
|
||||
NOTHING ON {{ inventory_hostname }} CAN CREATE /var/run/reboot-required.
|
||||
{{ unattended_upgrades_sentinel_script }} (executable) and
|
||||
{{ unattended_upgrades_sentinel_hook }} must both exist -- this role ships both, because
|
||||
Debian provides no sentinel writer of its own (update-notifier-common is an UBUNTU
|
||||
package and is not installable here). kured watches that path and reboots only when it
|
||||
exists, so with no writer the reboot automation is permanently and INVISIBLY idle --
|
||||
kured stays 1/1 Running and reports kured_reboot_required=0 on every node forever.
|
||||
Re-run this role without `--tags verify` to install them.
|
||||
success_msg: >-
|
||||
Reboot sentinel writer present ({{ unattended_upgrades_sentinel_script }} + dpkg hook)
|
||||
-- a kernel change will signal kured on the next apt transaction.
|
||||
tags: [verify]
|
||||
|
||||
- name: Report the pending-reboot state of this node
|
||||
# Read-only. Tells the operator whether this node is currently waiting on kured for a reboot,
|
||||
# which is the question `--tags verify` is usually being run to answer.
|
||||
ansible.builtin.stat:
|
||||
path: /var/run/reboot-required
|
||||
register: unattended_upgrades_reboot_required
|
||||
changed_when: false
|
||||
tags: [verify]
|
||||
|
||||
- name: Summary
|
||||
ansible.builtin.debug:
|
||||
msg: >-
|
||||
{{ inventory_hostname }}: security-only unattended upgrades ACTIVE
|
||||
(upgrade timer {{ unattended_upgrades_upgrade_time }} +/-{{ unattended_upgrades_randomised_delay }}),
|
||||
kernel and microcode blacklisted, automatic reboot DISABLED.
|
||||
Reboot currently pending: {{ unattended_upgrades_reboot_required.stat.exists }}.
|
||||
{% if unattended_upgrades_reboot_required.stat.exists %}
|
||||
This node is waiting for kured to reboot it in the next 03:00-06:00 window; it will be
|
||||
cordoned and drained first, and no other node will be rebooted at the same time.
|
||||
{% endif %}
|
||||
tags: [verify]
|
||||
Executable
+265
@@ -0,0 +1,265 @@
|
||||
#!/usr/bin/env bash
|
||||
# =================================================================================================
|
||||
# quiesce.sh — bring the prod cluster to a safe standstill before powering nodes off, and bring it
|
||||
# back afterwards. Companion to runbooks/cluster-maintenance-shutdown.md.
|
||||
# =================================================================================================
|
||||
# WHY THIS EXISTS. On 2026-10-17 all three nodes had to be powered off to move them onto a UPS, and
|
||||
# there was NO procedure for it: `infra/cluster.sh stop` is `virsh`-based and only ever applied to
|
||||
# the decommissioned dev cluster. The whole sequence was improvised live. This script is that
|
||||
# sequence, so the next person does not improvise it.
|
||||
#
|
||||
# WHY IT IS NOT JUST `shutdown -h`. Graceful node shutdown was unconfigured on every prod node that
|
||||
# day, so systemd stopped containerd and SIGKILLed every pod where it stood. roles/k3s_server now
|
||||
# sets shutdown-grace-period (+ the logind InhibitDelayMaxSec cap that actually enforces it), which
|
||||
# makes a bare reboot safe. This script is still the right tool for a PLANNED whole-cluster
|
||||
# shutdown, for three reasons the kubelet's drain does not cover:
|
||||
# 1. The UniFi VM needs an ACPI guest shutdown, not an eviction.
|
||||
# 2. CloudNativePG wants hibernation, not pod deletion -- and hibernation is IMPERATIVE state
|
||||
# that Flux will not set or clear for you.
|
||||
# 3. Draining three nodes one at a time makes pods RESCHEDULE onto the survivors, churning
|
||||
# Longhorn replicas for no reason when the whole cluster is going down anyway. Scaling in
|
||||
# place avoids all of it.
|
||||
#
|
||||
# SAFE TO RE-RUN. Every step is idempotent and `resume` is the exact inverse of `quiesce`.
|
||||
#
|
||||
# WHAT IT DELIBERATELY DOES NOT DO: power anything off. Node poweroff is an operator action
|
||||
# (CLAUDE.md §5) and needs the become password; the runbook gives the ansible one-liner. It also
|
||||
# does not touch TrueNAS (not in the inventory, SSH refused -- web UI only) or the router.
|
||||
set -euo pipefail
|
||||
|
||||
CONTEXT="${CONTEXT:-k3s-prod}"
|
||||
KUBECTL=(kubectl --context "$CONTEXT" --request-timeout=30s)
|
||||
TIMEOUT="${TIMEOUT:-600}"
|
||||
|
||||
# The three CloudNativePG clusters, as namespace/name. Hibernation is per-Cluster.
|
||||
CNPG=(keycloak/keycloak-db nextcloud-db/nextcloud-db semaphore-db/semaphore-db)
|
||||
|
||||
log() { printf '\033[34m[quiesce]\033[0m %s\n' "$*"; }
|
||||
ok() { printf '\033[32m[ ok ]\033[0m %s\n' "$*"; }
|
||||
warn() { printf '\033[33m[warn]\033[0m %s\n' "$*"; }
|
||||
die() { printf '\033[31m[fail]\033[0m %s\n' "$*" >&2; exit 1; }
|
||||
|
||||
need() { command -v "$1" >/dev/null || die "$1 not on PATH — run this inside ./infra/devcontainer/dev.sh"; }
|
||||
|
||||
# ---------------------------------------------------------------- discovery
|
||||
# Every Deployment/StatefulSet whose pods mount a PVC — Longhorn OR the TrueNAS NFS bulk tier.
|
||||
# Computed from the LIVE cluster rather than hardcoded, so a newly onboarded service is covered
|
||||
# without anyone remembering to add it here. PROTECTED namespaces are excluded because scaling
|
||||
# them down would remove the very machinery that performs the detach (longhorn-system), keep the
|
||||
# API reachable (kube-system), or finish the hibernation and the VM shutdown (cnpg-system,
|
||||
# kubevirt).
|
||||
pvc_workloads() {
|
||||
"${KUBECTL[@]}" get pods -A -o json | python3 -c '
|
||||
import json,sys
|
||||
PROTECT={"longhorn-system","kube-system","cilium-secrets","cnpg-system","kubevirt","flux-system","cdi"}
|
||||
out={}
|
||||
for p in json.load(sys.stdin)["items"]:
|
||||
ns=p["metadata"]["namespace"]
|
||||
if ns in PROTECT: continue
|
||||
if not any(v.get("persistentVolumeClaim") for v in (p["spec"].get("volumes") or [])): continue
|
||||
ow=(p["metadata"].get("ownerReferences") or [])
|
||||
if not ow: continue
|
||||
k,n=ow[0]["kind"],ow[0]["name"]
|
||||
if k=="ReplicaSet": k,n="deployment",n.rsplit("-",1)[0]
|
||||
elif k=="StatefulSet": k="statefulset"
|
||||
else: continue # Jobs, and CNPG Clusters, are handled separately
|
||||
out[(ns,k,n)]=1
|
||||
for ns,k,n in sorted(out): print(ns,k,n)
|
||||
'
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------- the PVC-deletion guard
|
||||
# *** DO NOT REMOVE THIS CHECK. IT EXISTS BECAUSE SCALING A StatefulSet DESTROYED DATA. ***
|
||||
# A StatefulSet may carry `persistentVolumeClaimRetentionPolicy.whenScaled: Delete`, and then
|
||||
# scaling it to 0 DELETES ITS PVCs -- and with them the Longhorn volumes. MEASURED 2026-10-17: the
|
||||
# quiesce for the UPS move scaled `monitoring/loki` (whenScaled: Delete) to 0 and its
|
||||
# `storage-loki-0` PVC and volume were deleted outright. Loki stores with `type: filesystem`, so
|
||||
# that PVC was the only copy; 72h retention bounded the loss to a rolling window, but nothing about
|
||||
# the mechanism is bounded -- the same action on a Delete-policy StatefulSet holding real state
|
||||
# would be unrecoverable. The volume count going 25 -> 24 was the only visible symptom, and it was
|
||||
# first misread as a miscount.
|
||||
#
|
||||
# So: such a StatefulSet is NEVER scaled. It is left RUNNING and its volume hard-detaches at
|
||||
# poweroff, which costs a replica rebuild on restart (degraded -> healthy, automatic). A rebuild is
|
||||
# an enormously better outcome than a deletion, and that is the whole trade being made here.
|
||||
sts_deletes_pvc_on_scale() { # ns name -> 0 if scaling it would delete its PVCs
|
||||
local policy
|
||||
policy=$("${KUBECTL[@]}" -n "$1" get statefulset "$2" \
|
||||
-o jsonpath='{.spec.persistentVolumeClaimRetentionPolicy.whenScaled}' 2>/dev/null || true)
|
||||
[ "$policy" = "Delete" ]
|
||||
}
|
||||
|
||||
wait_for() { # wait_for <description> <shell-condition>
|
||||
local desc="$1" cond="$2" waited=0
|
||||
log "waiting: $desc (up to ${TIMEOUT}s)"
|
||||
until eval "$cond"; do
|
||||
(( waited += 10 )); (( waited >= TIMEOUT )) && { warn "TIMED OUT waiting: $desc"; return 1; }
|
||||
sleep 10
|
||||
done
|
||||
ok "$desc"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------- quiesce
|
||||
do_quiesce() {
|
||||
need kubectl; need python3
|
||||
|
||||
# Refuse to start if Longhorn is already unhealthy. Powering off on top of a degraded volume is
|
||||
# how one replica becomes zero; better to stop here and let it rebuild first.
|
||||
local unhealthy
|
||||
unhealthy=$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io \
|
||||
-o jsonpath='{range .items[*]}{.status.robustness}{"\n"}{end}' | grep -cvE '^(healthy|unknown)$' || true)
|
||||
[ "$unhealthy" = "0" ] || die "$unhealthy Longhorn volume(s) are not healthy — fix or wait for rebuild BEFORE powering off"
|
||||
ok "Longhorn: all volumes healthy"
|
||||
|
||||
# 1. Flux first, or it reverts every scale-down below within its reconcile interval.
|
||||
log "suspending Flux Kustomizations"
|
||||
local k
|
||||
for k in $("${KUBECTL[@]}" -n flux-system get kustomizations -o name); do
|
||||
"${KUBECTL[@]}" -n flux-system patch "${k}" --type=merge -p '{"spec":{"suspend":true}}' >/dev/null
|
||||
done
|
||||
ok "all Kustomizations suspended"
|
||||
|
||||
# 2. The VM needs an ACPI shutdown of the GUEST. KubeVirt does that on running:false; a plain
|
||||
# pod eviction would be a power cut to the VM.
|
||||
if "${KUBECTL[@]}" -n unifi get vm unifi >/dev/null 2>&1; then
|
||||
log "ACPI-stopping the UniFi VM"
|
||||
"${KUBECTL[@]}" -n unifi patch vm unifi --type=merge -p '{"spec":{"running":false}}' >/dev/null
|
||||
fi
|
||||
|
||||
# 3. Postgres: hibernation is a clean smart shutdown that KEEPS the PVCs. CNPG does one instance
|
||||
# at a time, primary last-ish, so this is not instant.
|
||||
log "hibernating CloudNativePG clusters"
|
||||
local c ns name
|
||||
for c in "${CNPG[@]}"; do
|
||||
ns="${c%%/*}"; name="${c##*/}"
|
||||
"${KUBECTL[@]}" -n "$ns" annotate cluster.postgresql.cnpg.io "$name" \
|
||||
cnpg.io/hibernation=on --overwrite >/dev/null 2>&1 || warn "could not hibernate $c"
|
||||
done
|
||||
|
||||
# 4. Everything holding a PVC, scaled in place. NFS consumers first so nothing holds a hard
|
||||
# TrueNAS mount; PiHole last because it is the LAN resolver.
|
||||
log "scaling down PVC-holding workloads"
|
||||
local line ns2 kind nm deferred=() skipped_delete_policy=()
|
||||
while read -r ns2 kind nm; do
|
||||
[ -z "${ns2:-}" ] && continue
|
||||
if [ "$kind" = "statefulset" ] && sts_deletes_pvc_on_scale "$ns2" "$nm"; then
|
||||
warn "SKIPPING $ns2/$nm — whenScaled: Delete would DELETE its PVC (see the guard above)."
|
||||
warn " Left running; its volume hard-detaches at poweroff and rebuilds on restart."
|
||||
skipped_delete_policy+=("$ns2/$nm")
|
||||
continue
|
||||
fi
|
||||
if [ "$ns2" = "pihole" ]; then deferred+=("$ns2 $kind $nm"); continue; fi
|
||||
"${KUBECTL[@]}" -n "$ns2" scale "$kind" "$nm" --replicas=0 >/dev/null 2>&1 \
|
||||
&& printf ' scaled 0 %s/%s %s\n' "$ns2" "$kind" "$nm" \
|
||||
|| warn "could not scale $ns2/$kind/$nm"
|
||||
done < <(pvc_workloads)
|
||||
for line in "${deferred[@]:-}"; do
|
||||
[ -z "${line:-}" ] && continue
|
||||
# shellcheck disable=SC2086
|
||||
set -- $line
|
||||
"${KUBECTL[@]}" -n "$1" scale "$2" "$3" --replicas=0 >/dev/null 2>&1 \
|
||||
&& printf ' scaled 0 %s/%s %s (LAN DNS — expect name resolution to stop)\n' "$1" "$2" "$3"
|
||||
done
|
||||
|
||||
# 5. The only assertion that matters. An attached volume at poweroff comes back degraded and
|
||||
# rebuilds; this is what we are buying with all of the above.
|
||||
wait_for "UniFi VMI gone" \
|
||||
'[ "$('"${KUBECTL[*]}"' -n unifi get vmi --no-headers 2>/dev/null | wc -l)" = "0" ]' || true
|
||||
wait_for "all Postgres instances stopped" \
|
||||
'[ "$('"${KUBECTL[*]}"' get pods -A --no-headers 2>/dev/null | grep -cE "(keycloak|nextcloud|semaphore)-db-[0-9]")" = "0" ]' || true
|
||||
wait_for "all Longhorn volumes detached" \
|
||||
'[ "$('"${KUBECTL[*]}"' -n longhorn-system get volumes.longhorn.io -o jsonpath="{range .items[*]}{.status.state}{\"\n\"}{end}" 2>/dev/null | grep -c attached)" = "0" ]' || \
|
||||
warn "some volumes are STILL ATTACHED — they will come back degraded and rebuild; acceptable, but note which"
|
||||
|
||||
if [ "${#skipped_delete_policy[@]}" -gt 0 ]; then
|
||||
echo
|
||||
warn "NOT scaled down (whenScaled: Delete — scaling them would delete their PVCs):"
|
||||
printf ' %s\n' "${skipped_delete_policy[@]}"
|
||||
warn "Expect these volumes to come back 'degraded' and rebuild. That is correct and automatic."
|
||||
fi
|
||||
echo
|
||||
ok "QUIESCED. Safe to power off."
|
||||
cat <<'NEXT'
|
||||
|
||||
Next, from inside the devcontainer (it holds the become password):
|
||||
cd /work/infra/ansible && ansible -i inventory/hosts.ini k3s_prod -b \
|
||||
-m shell -a 'shutdown -h +1 "maintenance"'
|
||||
|
||||
THEN, in this order and not before:
|
||||
1. TrueNAS (192.168.2.30) — web UI only; not in the ansible inventory and SSH is refused.
|
||||
2. Router / switch last.
|
||||
NEXT
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------- resume
|
||||
do_resume() {
|
||||
need kubectl
|
||||
|
||||
# THE STEP THAT IS ALWAYS FORGOTTEN. Hibernation is imperative state: Flux neither sets nor
|
||||
# clears it, so resuming Flux alone leaves all three databases down and every app that needs
|
||||
# one looking broken for reasons that point nowhere near Postgres.
|
||||
log "un-hibernating CloudNativePG clusters"
|
||||
local c ns name
|
||||
for c in "${CNPG[@]}"; do
|
||||
ns="${c%%/*}"; name="${c##*/}"
|
||||
"${KUBECTL[@]}" -n "$ns" annotate cluster.postgresql.cnpg.io "$name" \
|
||||
cnpg.io/hibernation=off --overwrite >/dev/null 2>&1 || warn "could not un-hibernate $c"
|
||||
done
|
||||
|
||||
# Flux restores every replica count AND the VM's running:true from git, so nothing here needs to
|
||||
# undo the scale-downs by hand.
|
||||
log "resuming Flux Kustomizations"
|
||||
local k
|
||||
for k in $("${KUBECTL[@]}" -n flux-system get kustomizations -o name); do
|
||||
"${KUBECTL[@]}" -n flux-system patch "${k}" --type=merge -p '{"spec":{"suspend":false}}' >/dev/null
|
||||
done
|
||||
ok "all Kustomizations resumed"
|
||||
|
||||
wait_for "Flux fully reconciled" \
|
||||
'[ "$('"${KUBECTL[*]}"' -n flux-system get kustomizations --no-headers 2>/dev/null | grep -vc True)" = "0" ]' || \
|
||||
warn "some Kustomizations are not Ready — the graph serialises behind platform-config, so give it a few minutes before worrying"
|
||||
|
||||
echo
|
||||
ok "RESUMED."
|
||||
cat <<'NEXT'
|
||||
|
||||
Known cold-start race, not a fault in this script: oauth2-proxy performs OIDC discovery at
|
||||
startup and exits 1 on the 503 it gets while Keycloak is still coming up, then sits in
|
||||
exponential CrashLoopBackOff. If a replica is stuck, delete the pod once:
|
||||
kubectl --context k3s-prod -n oauth2-proxy delete pod <name>
|
||||
NEXT
|
||||
}
|
||||
|
||||
do_status() {
|
||||
need kubectl
|
||||
printf 'nodes: %s Ready\n' "$("${KUBECTL[@]}" get nodes --no-headers | grep -c ' Ready')"
|
||||
printf 'flux: %s/%s Ready\n' \
|
||||
"$("${KUBECTL[@]}" -n flux-system get kustomizations --no-headers | grep -c True)" \
|
||||
"$("${KUBECTL[@]}" -n flux-system get kustomizations --no-headers | wc -l)"
|
||||
printf 'longhorn: %s attached, %s detached, %s not healthy\n' \
|
||||
"$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.state}{"\n"}{end}' | grep -c attached)" \
|
||||
"$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.state}{"\n"}{end}' | grep -c detached)" \
|
||||
"$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.robustness}{"\n"}{end}' | grep -cvE '^(healthy|unknown)$' || true)"
|
||||
printf 'postgres: %s instance pod(s)\n' \
|
||||
"$("${KUBECTL[@]}" get pods -A --no-headers | grep -cE '(keycloak|nextcloud|semaphore)-db-[0-9]' || true)"
|
||||
printf 'unifi VM: %s VMI\n' "$("${KUBECTL[@]}" -n unifi get vmi --no-headers 2>/dev/null | wc -l)"
|
||||
}
|
||||
|
||||
case "${1:-}" in
|
||||
quiesce) do_quiesce ;;
|
||||
resume) do_resume ;;
|
||||
status) do_status ;;
|
||||
*) cat >&2 <<USAGE
|
||||
usage: $0 {quiesce|resume|status}
|
||||
|
||||
quiesce suspend Flux, ACPI-stop the UniFi VM, hibernate Postgres, scale down every
|
||||
PVC-holding workload, and wait for all Longhorn volumes to detach.
|
||||
Does NOT power anything off — see runbooks/cluster-maintenance-shutdown.md.
|
||||
resume the exact inverse: un-hibernate Postgres, resume Flux, wait for reconcile.
|
||||
status one-screen readout of nodes / Flux / Longhorn / Postgres / VM.
|
||||
|
||||
Run it inside the pinned toolbox, which supplies kubectl and the context:
|
||||
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh status'
|
||||
USAGE
|
||||
exit 2 ;;
|
||||
esac
|
||||
@@ -1,127 +0,0 @@
|
||||
# platform/kured
|
||||
|
||||
**kured (Kubernetes Reboot Daemon)** — coordinated, one-node-at-a-time OS reboots of the three prod
|
||||
control-plane nodes.
|
||||
|
||||
- **Flux tier:** `clusters/base/kured.yaml` (its own Kustomization, `dependsOn: [platform]`,
|
||||
`wait: true`)
|
||||
- **Chart:** `kured` 6.1.0 → appVersion 1.23.0, from `https://kubereboot.github.io/charts`
|
||||
- **The other half:** `infra/ansible/roles/unattended_upgrades` — the package upgrades. **Neither
|
||||
half is useful alone.**
|
||||
- **Runbook:** `/runbooks/node-reboot-automation`
|
||||
- **Every tuned value and its reasoning:** `helmrelease.yaml`. That file is the argument; this one is
|
||||
the orientation.
|
||||
- **Design record:** `decisions/2026-10-17-automated-upgrades.md`
|
||||
|
||||
## What this component owns
|
||||
|
||||
The *mechanics* of a reboot: take a cluster-wide lock, cordon, drain, reboot, uncordon, hold the
|
||||
lock a while longer. Nothing else.
|
||||
|
||||
**It does not decide whether to reboot.** kured reboots a node if and only if
|
||||
`/var/run/reboot-required` exists there. On these nodes that file is written by a deliberate
|
||||
operator kernel upgrade (`playbooks/node-baseline.yml --tags kernel`) — unattended-upgrades is
|
||||
security-only with the kernel blacklisted, so it essentially never produces one. **The human gate is
|
||||
the sentinel, not the daemon.**
|
||||
|
||||
## The problem this fixes
|
||||
|
||||
Rebooting these three nodes was a manual, serialized, supervised procedure, and
|
||||
`roles/node_baseline` has to end its kernel run by shouting the rules at the operator: *"Reboot prod
|
||||
nodes ONE AT A TIME, verify each returns before touching the next, and NEVER node 2 first"*. That is
|
||||
a correct procedure and an error-prone one, and nothing enforced it.
|
||||
|
||||
What it has to get right, measured on prod 2026-10-17:
|
||||
|
||||
- **etcd quorum is 2 of 3** — all three nodes are control-plane+etcd. Two down is a lost cluster.
|
||||
- **Every drain degrades all 25 Longhorn volumes to 2/3**, because 3 nodes × 3 replicas ×
|
||||
`replica-soft-anti-affinity: false` leaves nowhere to place a replacement, and rebuild is
|
||||
serialized behind a 600s wait. Reboot the next node too early and volumes hit 1/3, where
|
||||
`block-if-contains-last-replica` can block a drain forever.
|
||||
- **Six PodDisruptionBudgets allow zero voluntary disruptions** — 3 Longhorn `instance-manager-*`
|
||||
(structurally 0 by design; this is how Longhorn gates the drain) and 3 CNPG `*-primary`.
|
||||
|
||||
## Placement — why its own Kustomization
|
||||
|
||||
`platform` is `wait: true` over twelve components and gates `platform-config` → every apps wave. A
|
||||
reboot daemon that failed to go healthy there would stall **every application in the cluster** on
|
||||
"dependency not ready". A component added to make maintenance safer must not be able to cause an
|
||||
outage. `wait: true` here is a health **signal** for something nothing depends on — and nothing
|
||||
should ever `dependsOn: [kured]`.
|
||||
|
||||
The `dependsOn: [platform]` edge is load-bearing, not conventional: it supplies the
|
||||
`monitoring.coreos.com` CRDs this component's `PrometheusRule` needs (F-MON1 — a pass containing an
|
||||
unknown kind aborts), plus the Prometheus that kured **queries** before every reboot.
|
||||
|
||||
## How concurrent reboots are made impossible
|
||||
|
||||
| mechanism | value | why |
|
||||
|---|---|---|
|
||||
| cluster-wide lock | one annotation on kured's own DaemonSet | a node reboots only while holding it; enforced by API-server optimistic concurrency, not timing |
|
||||
| `concurrency` | `1` | one lock holder at a time |
|
||||
| **`lockTtl`** | **`0` (disabled)** | a TTL lets reboots continue after the holder dies — on 3 etcd nodes that is the catastrophe. A hung node holds the lock **forever** and nothing cascades. |
|
||||
| `lockReleaseDelay` | `30m` | the next node waits out Longhorn's 600s replenishment even if no alert latched |
|
||||
| alert gate | `alertFilterMatchOnly: true` + anchored allow-list | refuses to proceed while a volume is degraded/faulted or a node is down |
|
||||
| window | 03:00–06:00 daily, America/New_York | matches `renovate.json`'s window; daily (not weekly) is what makes `KuredRebootPending`'s `for: 36h` meaningful |
|
||||
|
||||
## Deliberate non-choices
|
||||
|
||||
- **No `--ignore-daemonsets` / `--delete-emptydir-data` / `--force`.** Those flags **do not exist**
|
||||
in kured 1.23.0 — it hardcodes all three in the drain helper (`cmd/kured/main.go:438-449`).
|
||||
Passing them would CrashLoop the DaemonSet. All three are genuinely required here (12 DaemonSets,
|
||||
135 emptyDir pods).
|
||||
- **`rebootMethod: command`, not `signal`.** `signal` would allow `privileged: false` + `CAP_KILL`,
|
||||
but needs a host-side `kured-reboot.service` that does not exist on these nodes. A real future
|
||||
hardening step; explicitly not taken here.
|
||||
- **No tolerations.** No prod node is tainted (measured). A future tainted node would be silently
|
||||
uncovered — hence `KuredNodeCoverageGap`.
|
||||
- **`NodeEtcdRequestsSlow` / `NodeIOPressure*` are NOT in the alert gate**, despite being exactly
|
||||
what you would want to pause for. prod-1 has a documented etcd WAL I/O starvation history and
|
||||
including them risks a permanent wedge that looks like safety.
|
||||
- **No Slack/shoutrrr notifications.** Would require a SOPS Secret and a decryption patch; the three
|
||||
alerts cover the same ground through Alertmanager.
|
||||
|
||||
## Versioning
|
||||
|
||||
Chart pinned `>=6.1.0 <6.2.0`. The minor ceiling is **deliberate here**, unlike elsewhere in this
|
||||
repo where it is a main cause of pin rot: a kured minor may change drain or lock semantics, and this
|
||||
is the one component where "a human read the changelog" should be a precondition. `renovate.json`'s
|
||||
`blast-radius` rule additionally forbids auto-merging it, so the ceiling is a review prompt, not a
|
||||
wall.
|
||||
|
||||
## What breaks silently, and what catches it
|
||||
|
||||
Every failure mode of this component leaves a cluster that looks healthy — kured stays 1/1 Running
|
||||
while rebooting nothing. `alerts.yaml` ships three rules over kured's single metric, with a
|
||||
companion promtool suite:
|
||||
|
||||
| alert | catches |
|
||||
|---|---|
|
||||
| `KuredNotReporting` | the scrape pipeline is dead (chart defaults `service.create`/`metrics.create` to **false**; a NetworkPolicy can time the scrape out) — and the other two are inert while this is true |
|
||||
| `KuredNodeCoverageGap` | kured missing on *some* node, which `absent()` cannot see |
|
||||
| `KuredRebootPending` | a reboot was offered and blocked for >36h |
|
||||
|
||||
## Post-merge verification
|
||||
|
||||
```bash
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds,pod,svc,servicemonitor -o wide'
|
||||
infra/devcontainer/dev.sh run bash -c 'flux --context k3s-prod get kustomization kured'
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured logs ds/kured --tail=40'
|
||||
```
|
||||
|
||||
Expect `3/3` ready, the Kustomization `Ready=True`, and kured logging a reboot-check decision each
|
||||
period. Then confirm the scrape is real — this is the step that proves the alerts are not inert:
|
||||
|
||||
```bash
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n monitoring exec \
|
||||
sts/prometheus-kube-prometheus-stack-prometheus -c prometheus -- \
|
||||
wget -qO- "http://localhost:9090/api/v1/query?query=count(kured_reboot_required)"'
|
||||
```
|
||||
|
||||
**Expect `3`.** Anything else (and especially an empty result) means the Service, ServiceMonitor or
|
||||
NetworkPolicy is wrong and all three alerts are silently dead.
|
||||
|
||||
**Before trusting the timer with a real kernel upgrade**, do one watched drain to settle the open
|
||||
question the design record flags: whether CNPG actually switches its primary away in response to a
|
||||
drain-induced eviction. If it does not, draining `k3s-prod-1` blocks on three 0-disruption PDBs at
|
||||
once.
|
||||
@@ -1,282 +0,0 @@
|
||||
---
|
||||
# promtool unit tests for the three rules in platform/kured/alerts.yaml.
|
||||
#
|
||||
# WHY THIS FILE EXISTS. The rules it covers are the ONLY thing that can tell an operator that the
|
||||
# reboot automation has stopped working, and every failure mode they describe is silent — kured stays
|
||||
# 1/1 Running while rebooting nothing. An alert that cannot fire is therefore not a cosmetic defect
|
||||
# here, it is the removal of the only signal. This repo has shipped that bug in both directions
|
||||
# already (a permanently-firing VPNTunnelDown; a structurally inert VeleroNoRecentBackup over an
|
||||
# absent series), which is why the harness exists at all. Each rule below is pinned BOTH ways: quiet
|
||||
# on the healthy state, firing on the constructed bad one, with the `for:` boundary asserted rather
|
||||
# than assumed.
|
||||
#
|
||||
# HOW TO RUN. `platform/monitoring/tests/run.sh` — the one repo-wide rule-unit harness. It finds this
|
||||
# file by the COMPANION convention (`<manifest>.promtool-test.yaml` beside `<manifest>.yaml`), so the
|
||||
# manifest under test is exactly platform/kured/alerts.yaml and nothing else. Run just this file with
|
||||
# platform/monitoring/tests/run.sh platform/kured/alerts.promtool-test.yaml
|
||||
# The harness flattens the PrometheusRule CR's `.spec.groups` into the bare rules file promtool wants,
|
||||
# `promtool check rules` it first so a non-parsing file cannot pass vacuously, and runs it all in a
|
||||
# temp dir. It leaves Flux's `${BASE_DOMAIN}` in the runbook_url annotations verbatim, which is safe
|
||||
# because the assertions below query the `ALERTS` series and never compare annotations.
|
||||
#
|
||||
# WHY THE ASSERTIONS QUERY `ALERTS` RATHER THAN USING `exp_alerts`. promtool's alert_rule_test
|
||||
# compares annotations VERBATIM, which would mean pasting each rule's description prose in here and
|
||||
# re-pasting it on every wording change. Querying the synthetic `ALERTS` series proves what actually
|
||||
# matters — the alert reaches `pending` and then `firing` across its `for:` boundary, with the right
|
||||
# labels — and stays decoupled from annotation text. Same choice as
|
||||
# platform/cnpg/alerts.promtool-test.yaml.
|
||||
#
|
||||
# THE SERIES SHAPES ARE REAL. `kured_reboot_required{node=...}` is a GaugeVec with exactly one label,
|
||||
# `node` — read from kured's cmd/kured/main.go:92-96 (the only registered metric in the project).
|
||||
# `kube_node_info{node=...}` was measured on prod Prometheus on 2026-10-17 returning
|
||||
# `count(kube_node_info) == 3`, matching the three nodes. The extra kube_node_info labels prod
|
||||
# carries are omitted deliberately: every rule aggregates with `count()`, so they cannot affect the
|
||||
# result, and inventing plausible-but-wrong values would be worse than leaving them out.
|
||||
#
|
||||
# evaluation_interval is 5m, not the usual 1m: KuredRebootPending has `for: 36h`, and at a 1m
|
||||
# interval promtool would evaluate that test over 2000+ steps for no added coverage.
|
||||
rule_files:
|
||||
- alerts.rules.yaml
|
||||
evaluation_interval: 5m
|
||||
|
||||
tests:
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
# 1. THE HEALTHY STEADY STATE. Three nodes, kured running on all three, nothing pending.
|
||||
# ALL THREE RULES MUST BE SILENT. This is the case that a badly-written `absent()` or a
|
||||
# `count() <` comparison with an off-by-one gets wrong, and it is also the state the cluster
|
||||
# is in ~100% of the time, so a false positive here would train the operator to ignore the
|
||||
# whole group.
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
- interval: 5m
|
||||
name: healthy cluster - no reboots pending, kured on every node, all three rules quiet
|
||||
input_series:
|
||||
- series: 'kured_reboot_required{node="k3s-prod-1"}'
|
||||
values: '0x60'
|
||||
- series: 'kured_reboot_required{node="k3s-prod-2"}'
|
||||
values: '0x60'
|
||||
- series: 'kured_reboot_required{node="k3s-prod-3"}'
|
||||
values: '0x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-1"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-2"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-3"}'
|
||||
values: '1x60'
|
||||
promql_expr_test:
|
||||
# Nothing from this group may be pending or firing, at any point in the window.
|
||||
- expr: 'count(ALERTS{alertname=~"Kured.*"})'
|
||||
eval_time: 4h
|
||||
exp_samples: []
|
||||
# And the underlying expressions must each be genuinely false, not merely unevaluated:
|
||||
# absent() yields no series because the vector is populated...
|
||||
- expr: 'absent(kured_reboot_required)'
|
||||
eval_time: 4h
|
||||
exp_samples: []
|
||||
# ...coverage is exact (3 == 3), so the `<` comparison drops the sample...
|
||||
- expr: 'count(kured_reboot_required) < count(kube_node_info)'
|
||||
eval_time: 4h
|
||||
exp_samples: []
|
||||
# ...and no node reports a pending reboot.
|
||||
- expr: 'max by (node) (kured_reboot_required) == 1'
|
||||
eval_time: 4h
|
||||
exp_samples: []
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
# 2. KuredNotReporting — THE WHOLE PIPELINE IS DEAD. No kured series at all, which is what the
|
||||
# cluster looks like if the chart's `service.create`/`metrics.create` regress to their FALSE
|
||||
# defaults, if the ServiceMonitor is dropped, or if a NetworkPolicy silently times out the
|
||||
# scrape (the Longhorn precedent). The nodes still exist, so kube_node_info is present —
|
||||
# proving the rule keys on kured's absence and not on a dead cluster.
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
- interval: 5m
|
||||
name: no kured series anywhere - KuredNotReporting fires after 30m
|
||||
input_series:
|
||||
- series: 'kube_node_info{node="k3s-prod-1"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-2"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-3"}'
|
||||
values: '1x60'
|
||||
promql_expr_test:
|
||||
# The expression itself yields 1 immediately.
|
||||
- expr: 'absent(kured_reboot_required)'
|
||||
eval_time: 10m
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: 1
|
||||
# `for: 30m` is honoured — still only pending at 25m.
|
||||
- expr: 'ALERTS{alertname="KuredNotReporting"}'
|
||||
eval_time: 25m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="KuredNotReporting", alertstate="pending", severity="warning"}'
|
||||
value: 1
|
||||
# ...and firing once past it.
|
||||
- expr: 'ALERTS{alertname="KuredNotReporting", alertstate="firing"}'
|
||||
eval_time: 35m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="KuredNotReporting", alertstate="firing", severity="warning"}'
|
||||
value: 1
|
||||
# THE GUARD CLAUSE: with kured absent, the other two rules are inert — exactly the reason
|
||||
# this rule has to exist. KuredRebootPending cannot fire (no series to be 1) and
|
||||
# KuredNodeCoverageGap cannot either, because `count()` over an empty vector returns NO
|
||||
# SAMPLE rather than 0, so the `<` comparison has nothing to compare. Asserting that here
|
||||
# pins the dependency: if a future edit made coverage-gap fire in this state, the overlap
|
||||
# would be deliberate rather than accidental.
|
||||
- expr: 'ALERTS{alertname="KuredRebootPending"}'
|
||||
eval_time: 35m
|
||||
exp_samples: []
|
||||
- expr: 'ALERTS{alertname="KuredNodeCoverageGap"}'
|
||||
eval_time: 35m
|
||||
exp_samples: []
|
||||
- expr: 'count(kured_reboot_required) < count(kube_node_info)'
|
||||
eval_time: 35m
|
||||
exp_samples: []
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
# 3. KuredNodeCoverageGap — kured COVERS ONLY TWO OF THREE NODES. k3s-prod-3's series stops at
|
||||
# 1h and never returns (a node taint this DaemonSet does not tolerate, a node that never came
|
||||
# back from a reboot, a DaemonSet nodeSelector edit). This also pins the `for: 2h` choice,
|
||||
# which exists SO THAT AN ORDINARY REBOOT DOES NOT TRIP IT: the gap is real from ~1h05 but
|
||||
# the alert is only pending an hour later, and a healthy reboot is minutes.
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
- interval: 5m
|
||||
name: kured missing on one node - KuredNodeCoverageGap fires after 2h, not before
|
||||
input_series:
|
||||
- series: 'kured_reboot_required{node="k3s-prod-1"}'
|
||||
values: '0x60'
|
||||
- series: 'kured_reboot_required{node="k3s-prod-2"}'
|
||||
values: '0x60'
|
||||
# Ends after 1h (13 samples at 5m) and is stale thereafter.
|
||||
- series: 'kured_reboot_required{node="k3s-prod-3"}'
|
||||
values: '0x12'
|
||||
- series: 'kube_node_info{node="k3s-prod-1"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-2"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-3"}'
|
||||
values: '1x60'
|
||||
promql_expr_test:
|
||||
# Before the gap opens, coverage is exact and nothing is pending.
|
||||
- expr: 'ALERTS{alertname="KuredNodeCoverageGap"}'
|
||||
eval_time: 30m
|
||||
exp_samples: []
|
||||
# The shortfall is real well before the alert fires — 2 < 3.
|
||||
- expr: 'count(kured_reboot_required)'
|
||||
eval_time: 2h
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: 2
|
||||
# A SHORT OUTAGE DOES NOT ALERT: still only pending at 2h, an hour after the gap opened.
|
||||
- expr: 'ALERTS{alertname="KuredNodeCoverageGap"}'
|
||||
eval_time: 2h
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="KuredNodeCoverageGap", alertstate="pending", severity="warning"}'
|
||||
value: 1
|
||||
# Firing once the gap has persisted past `for: 2h`.
|
||||
- expr: 'ALERTS{alertname="KuredNodeCoverageGap", alertstate="firing"}'
|
||||
eval_time: 3h15m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="KuredNodeCoverageGap", alertstate="firing", severity="warning"}'
|
||||
value: 1
|
||||
# KuredNotReporting must stay QUIET: two nodes are still publishing, so the vector is not
|
||||
# absent. This is the distinction between "kured is gone" and "kured is incomplete", and
|
||||
# getting it wrong in either direction would collapse two different diagnoses into one.
|
||||
- expr: 'ALERTS{alertname="KuredNotReporting"}'
|
||||
eval_time: 3h15m
|
||||
exp_samples: []
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
# 4. KuredRebootPending — A REBOOT IS BLOCKED. k3s-prod-1 reports a pending reboot continuously:
|
||||
# the sentinel was written (an operator kernel upgrade) and the reboot has not happened across
|
||||
# more than one 03:00-06:00 window. Asserts the 36h boundary in both directions, which is the
|
||||
# assertion that makes the DAILY-window decision load-bearing rather than cosmetic: on a
|
||||
# weekly window this threshold could not distinguish "blocked" from "waiting for Sunday".
|
||||
# Series run to 38h20m (456 samples at 5m) so the post-boundary evaluation has real data.
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
- interval: 5m
|
||||
name: reboot pending on one node for over 36h - KuredRebootPending fires, per-node label kept
|
||||
input_series:
|
||||
- series: 'kured_reboot_required{node="k3s-prod-1"}'
|
||||
values: '1x456'
|
||||
- series: 'kured_reboot_required{node="k3s-prod-2"}'
|
||||
values: '0x456'
|
||||
- series: 'kured_reboot_required{node="k3s-prod-3"}'
|
||||
values: '0x456'
|
||||
- series: 'kube_node_info{node="k3s-prod-1"}'
|
||||
values: '1x456'
|
||||
- series: 'kube_node_info{node="k3s-prod-2"}'
|
||||
values: '1x456'
|
||||
- series: 'kube_node_info{node="k3s-prod-3"}'
|
||||
values: '1x456'
|
||||
promql_expr_test:
|
||||
# The expression selects ONLY the pending node — the operator needs to know which box.
|
||||
- expr: 'max by (node) (kured_reboot_required) == 1'
|
||||
eval_time: 1h
|
||||
exp_samples:
|
||||
- labels: '{node="k3s-prod-1"}'
|
||||
value: 1
|
||||
# A NORMAL WAIT IS NOT AN ALERT. At 24h the node is still inside its legitimate
|
||||
# once-per-day window wait, so the alert may be pending but must NOT be firing.
|
||||
- expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}'
|
||||
eval_time: 24h
|
||||
exp_samples: []
|
||||
# Still not firing just short of the boundary.
|
||||
- expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}'
|
||||
eval_time: 35h30m
|
||||
exp_samples: []
|
||||
# Firing past it, carrying the node label through to the notification.
|
||||
- expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}'
|
||||
eval_time: 36h30m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="KuredRebootPending", alertstate="firing", node="k3s-prod-1", severity="warning"}'
|
||||
value: 1
|
||||
# The two healthy nodes must NOT be alerted on. A rule written with `kured_reboot_required`
|
||||
# unaggregated, or compared with `>= 0`, would raise all three and bury the real one.
|
||||
- expr: 'count(ALERTS{alertname="KuredRebootPending", alertstate="firing"})'
|
||||
eval_time: 36h30m
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: 1
|
||||
# Liveness rules stay quiet: kured is present and covering all three nodes. A pending reboot
|
||||
# is NOT a wiring fault and must not be reported as one.
|
||||
- expr: 'ALERTS{alertname="KuredNotReporting"}'
|
||||
eval_time: 36h30m
|
||||
exp_samples: []
|
||||
- expr: 'ALERTS{alertname="KuredNodeCoverageGap"}'
|
||||
eval_time: 36h30m
|
||||
exp_samples: []
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
# 5. THE RULE MUST STOP. A reboot that SUCCEEDS clears the sentinel, and the alert must clear
|
||||
# with it — the other half of the failure space this harness exists for (a permanently-firing
|
||||
# rule is how VPNTunnelDown taught this repo the lesson). k3s-prod-1 is pending for ~40h,
|
||||
# long enough to fire, then the reboot happens and the gauge returns to 0.
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────────
|
||||
- interval: 5m
|
||||
name: reboot completes after firing - KuredRebootPending resolves
|
||||
input_series:
|
||||
# 1 for 480 samples (40h), then 0 for the remainder.
|
||||
- series: 'kured_reboot_required{node="k3s-prod-1"}'
|
||||
values: '1x480 0x120'
|
||||
- series: 'kured_reboot_required{node="k3s-prod-2"}'
|
||||
values: '0x600'
|
||||
- series: 'kured_reboot_required{node="k3s-prod-3"}'
|
||||
values: '0x600'
|
||||
- series: 'kube_node_info{node="k3s-prod-1"}'
|
||||
values: '1x600'
|
||||
- series: 'kube_node_info{node="k3s-prod-2"}'
|
||||
values: '1x600'
|
||||
- series: 'kube_node_info{node="k3s-prod-3"}'
|
||||
values: '1x600'
|
||||
promql_expr_test:
|
||||
# Firing while the reboot is still outstanding.
|
||||
- expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}'
|
||||
eval_time: 39h
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="KuredRebootPending", alertstate="firing", node="k3s-prod-1", severity="warning"}'
|
||||
value: 1
|
||||
# Cleared once the sentinel is gone. If this assertion ever fails, the rule has become
|
||||
# self-sustaining and the operator will start ignoring it.
|
||||
- expr: 'ALERTS{alertname=~"Kured.*"}'
|
||||
eval_time: 45h
|
||||
exp_samples: []
|
||||
@@ -1,124 +0,0 @@
|
||||
---
|
||||
# Alert rules for the kured reboot automation.
|
||||
#
|
||||
# WHY THESE THREE, AND WHY THEY ARE NOT OPTIONAL. Reboot automation's characteristic failure is
|
||||
# SILENCE, not noise. Every way this component can break leaves a cluster that looks perfectly
|
||||
# healthy: a sentinel nothing writes, a Service the chart did not create, a NetworkPolicy that drops
|
||||
# the scrape, a taint that excludes a node, a drain wedged on a PodDisruptionBudget, a cluster lock
|
||||
# held forever by a node that hung on warm reboot. In none of those cases does anything turn red on
|
||||
# its own. kured itself stays 1/1 Running throughout.
|
||||
#
|
||||
# kured EXPOSES EXACTLY ONE METRIC. Read from cmd/kured/main.go:92-96 and :113 on 2026-10-17 — the
|
||||
# only `prometheus.MustRegister` in the tree is a GaugeVec `Subsystem: "kured"`, `Name:
|
||||
# "reboot_required"`, labelled `{node}`. So the full vocabulary available is
|
||||
# `kured_reboot_required{node="..."}`, 1 when the sentinel exists and 0 when it does not. Each rule
|
||||
# below is built from that one series plus `kube_node_info` (kube-state-metrics), and nothing else.
|
||||
# There is deliberately no rule over an invented metric name: this repo has shipped a
|
||||
# structurally-inert alert before (VeleroNoRecentBackup over an absent series, which is why
|
||||
# platform/monitoring/tests/run.sh exists), and an alert that cannot fire is worse than no alert
|
||||
# because it also removes the suspicion that would have caught it.
|
||||
#
|
||||
# THESE RULES LIVE HERE, NOT IN platform/monitoring/alerts.yaml, for two reasons. (1) The unit-test
|
||||
# harness treats a `<manifest>.promtool-test.yaml` COMPANION as covering just its sibling, whereas a
|
||||
# `tests/*.test.yaml` SUITE covers every PrometheusRule in its directory — so shipping them beside
|
||||
# the component keeps the test obligation local and legible (the platform/cnpg precedent). (2)
|
||||
# platform/monitoring/alerts.yaml is concurrently edited by other open branches; adding 3 rules to
|
||||
# its tail would create a merge conflict for no benefit.
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: kured
|
||||
namespace: kured
|
||||
labels:
|
||||
app.kubernetes.io/name: kured
|
||||
spec:
|
||||
groups:
|
||||
- name: platform-health.node-reboots
|
||||
rules:
|
||||
# ── 1. IS THE MECHANISM ALIVE AT ALL? ────────────────────────────────────────────────
|
||||
- alert: KuredNotReporting
|
||||
# `absent()` yields a series only when the ENTIRE vector is empty, so this fires when no
|
||||
# node at all is publishing — kured gone, or its Service/ServiceMonitor/NetworkPolicy
|
||||
# regressed so nothing is scraped. It stays quiet while a single node is mid-reboot
|
||||
# (the other two still report), which is what makes it specific to a wiring failure
|
||||
# rather than to a reboot in progress.
|
||||
#
|
||||
# THIS IS THE RULE THAT GUARDS THE OTHER TWO. Both of them read `kured_reboot_required`,
|
||||
# so if that series stops existing they go quiet and look like good news. This one turns
|
||||
# the absence itself into the signal. The specific regression it is aimed at: the kured
|
||||
# chart defaults `service.create` and `metrics.create` to FALSE, and Longhorn has already
|
||||
# demonstrated on this cluster (see the companion note in
|
||||
# conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh) that a
|
||||
# NetworkPolicy can let targets be DISCOVERED while every scrape times out and the TSDB
|
||||
# stays empty.
|
||||
expr: absent(kured_reboot_required)
|
||||
# 30m absorbs a HelmRelease upgrade rolling the DaemonSet and a Prometheus restart,
|
||||
# without tolerating a real wiring break for more than one quiet night's worth of time.
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: kured is publishing no reboot_required series on any node
|
||||
description: >-
|
||||
No node is reporting kured_reboot_required, so the reboot automation is either not
|
||||
running or not being scraped. While this is true, OS reboots are not happening and
|
||||
the two alerts that would say so (KuredRebootPending, KuredNodeCoverageGap) are
|
||||
themselves inert. Nodes will silently accumulate pending kernel upgrades.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-reboot-automation"
|
||||
|
||||
# ── 2. IS IT ALIVE ON *EVERY* NODE? ──────────────────────────────────────────────────
|
||||
- alert: KuredNodeCoverageGap
|
||||
# A per-node agent that silently covers only some nodes is the subtler half of the
|
||||
# liveness question, and `absent()` cannot see it. kured publishes one series per node it
|
||||
# runs on; kube-state-metrics publishes one `kube_node_info` per node that exists
|
||||
# (measured 2026-10-17: 3, matching the 3 nodes). A shortfall means at least one node has
|
||||
# no reboot automation — most likely a new or re-provisioned node carrying a taint this
|
||||
# DaemonSet does not tolerate (it declares none, because no prod node is tainted today).
|
||||
# That node would then never reboot, with nothing else to indicate it.
|
||||
expr: count(kured_reboot_required) < count(kube_node_info)
|
||||
# 2h, chosen to sit comfortably ABOVE the duration of a healthy reboot. A node being
|
||||
# rebooted stops publishing for a few minutes and legitimately creates a shortfall; a
|
||||
# node that is still missing after two hours is either genuinely uncovered or has hung on
|
||||
# reboot, and both of those want a human.
|
||||
for: 2h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: kured is not running on every node
|
||||
description: >-
|
||||
Fewer nodes are reporting kured_reboot_required than exist in the cluster, so at
|
||||
least one node has no reboot automation and will never be rebooted by it. Most
|
||||
likely a node carries a taint this DaemonSet does not tolerate, or has hung during
|
||||
a reboot.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-reboot-automation"
|
||||
|
||||
# ── 3. IS A REBOOT STUCK? ────────────────────────────────────────────────────────────
|
||||
- alert: KuredRebootPending
|
||||
# A node has wanted a reboot for longer than a full maintenance window plus margin, so
|
||||
# the reboot was OFFERED and did not happen. The causes are all real and all invisible
|
||||
# otherwise: the Prometheus alert gate is blocking on a degraded Longhorn volume that is
|
||||
# not recovering; the drain hit `drainTimeout` against a 0-disruption PDB; another node
|
||||
# holds the cluster lock and never released it (`lockTtl: 0` means a lock held by a hung
|
||||
# node is held FOREVER — that is deliberate, see helmrelease.yaml, and this is the alert
|
||||
# that makes the resulting stall visible).
|
||||
expr: max by (node) (kured_reboot_required) == 1
|
||||
# 36h, and the window being DAILY is what makes this number meaningful — see the window
|
||||
# block in helmrelease.yaml. With a 03:00-06:00 window every day, a pending reboot waits
|
||||
# at most ~24h legitimately, so 36h means at least one full window elapsed without the
|
||||
# reboot completing. On a weekly window this threshold would have to exceed 7 days, which
|
||||
# is longer than this cluster's 7d Prometheus retention and is reset by any Prometheus
|
||||
# restart past the 1h for-outage-tolerance — i.e. unreliable in the one direction that
|
||||
# matters. Note this alert can still be LATE for that reason, but it cannot be spurious.
|
||||
for: 36h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: Node {{ $labels.node }} has needed a reboot for over 36h
|
||||
description: >-
|
||||
kured reports a pending reboot on {{ $labels.node }} that has survived at least one
|
||||
full 03:00-06:00 maintenance window. The reboot is being blocked, not merely
|
||||
waiting: likely a degraded Longhorn volume holding the Prometheus alert gate, a
|
||||
drain timing out against a PodDisruptionBudget, or a stale cluster lock held by a
|
||||
node that did not come back. The kernel or security update that needed the reboot is
|
||||
installed but NOT running.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-reboot-automation"
|
||||
@@ -1,289 +0,0 @@
|
||||
---
|
||||
# kured (Kubernetes Reboot Daemon) — coordinated, one-node-at-a-time reboots of the three prod
|
||||
# control-plane nodes.
|
||||
#
|
||||
# ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# READ THIS FIRST: kured DOES NOT DECIDE *WHETHER* TO REBOOT. IT ONLY DECIDES *WHEN* AND *HOW*.
|
||||
# ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# kured reboots a node if and only if the file `/var/run/reboot-required` exists on it. On these
|
||||
# nodes exactly one thing creates that file: a deliberate operator run of
|
||||
# `playbooks/node-baseline.yml --tags kernel` (which advances the backports kernel). The
|
||||
# unattended-upgrades role this PR also adds installs SECURITY updates only and BLACKLISTS
|
||||
# `linux-image-*`/`linux-headers-*`, so it can never produce a reboot-required sentinel on its own —
|
||||
# see infra/ansible/roles/unattended_upgrades/defaults/main.yml for that argument in full.
|
||||
#
|
||||
# That is the whole answer to this repo's two standing objections to reboot automation:
|
||||
#
|
||||
# * infra/ansible/roles/node_baseline/tasks/main.yml:61-67 — "IT DOES NOT REBOOT." k3s-prod-2 hung
|
||||
# on a warm reboot on 2026-09-03 (powered, off the LAN ~38 minutes, physical power button
|
||||
# required), so rebooting was made "a SERIALIZED, SUPERVISED operator step ... and a role cannot
|
||||
# enforce that." STILL TRUE, and nothing here changes it: node_baseline continues not to reboot.
|
||||
# * docs/plans/cluster-upgrade-plan.md:291-299 — system-upgrade-controller was rejected because it
|
||||
# "automates away the human gate between nodes", and the operator's hard requirement is an
|
||||
# explicit PASS/STOP verification BETWEEN every node.
|
||||
#
|
||||
# THE HUMAN GATE IS PRESERVED, AND IT IS THE SENTINEL. The operator decides whether a kernel moves
|
||||
# and on which hosts (`--limit k3s-prod-1`), which is what writes the sentinel on those hosts and
|
||||
# only those hosts. kured then performs the cordon/drain/reboot/uncordon that the operator would
|
||||
# otherwise have typed by hand — the error-prone mechanical half, including the ordering rule
|
||||
# node_baseline's own message has to shout ("NEVER node 2 first"). Run the kernel play one host at a
|
||||
# time and the operator keeps per-node PASS/STOP control exactly as before; run it on all three and
|
||||
# kured still does them strictly one at a time, just in an order it chooses. Both are safe for
|
||||
# quorum; only the first preserves the ordering preference. See runbooks/node-reboot-automation.md.
|
||||
#
|
||||
# cluster-upgrade-plan.md:329-330 and :545-548 name the condition under which its own rejection
|
||||
# reverses — "SUC becomes viable once the cluster is on fast disks ... where drain-driven rebuild is
|
||||
# cheap" — and prod is that cluster. This component takes that clause up, but NOT its drain
|
||||
# assumption: see the Longhorn section below, where drain-driven rebuild is measurably NOT free even
|
||||
# on SSD, and is gated on explicitly rather than waved through.
|
||||
#
|
||||
# WHAT IS STILL UNMITIGATED, STATED PLAINLY: kured cannot press a power button. If a node hangs on
|
||||
# warm reboot the way k3s-prod-2 did in September, kured will not recover it. The blast radius is
|
||||
# bounded by design rather than by luck — `concurrency: 1` plus `lockTtl: 0` means the hung node
|
||||
# keeps the cluster lock forever, so NO second node is ever rebooted behind it; etcd keeps 2/3 quorum
|
||||
# and the cluster stays up, degraded, until a human attends. The KuredRebootPending alert in
|
||||
# alerts.yaml is what tells them. That is strictly better than the previous automation story (none)
|
||||
# but it is NOT the same as "safe to ignore at 3am on a Sunday".
|
||||
apiVersion: helm.toolkit.fluxcd.io/v2
|
||||
kind: HelmRelease
|
||||
metadata:
|
||||
name: kured
|
||||
namespace: kured
|
||||
spec:
|
||||
interval: 30m
|
||||
chart:
|
||||
spec:
|
||||
chart: kured
|
||||
# Chart 6.1.0 -> appVersion 1.23.0 (published 2026-06-30; newest of the 80 entries in
|
||||
# https://kubereboot.github.io/charts/index.yaml, read 2026-10-17).
|
||||
#
|
||||
# THE `<6.2.0` CEILING IS A DELIBERATE CHOICE HERE, NOT THE REFLEX IT IS ELSEWHERE. This PR's
|
||||
# own audit found that 9 of 12 HelmReleases in this repo pin a minor-locking range, and that
|
||||
# this is a MAIN CAUSE of pin rot: Renovate can then only ever offer patches inside a line it
|
||||
# has been told not to leave (kube-prometheus-stack is stuck 4 chart majors back that way).
|
||||
# For a component whose job is to reboot control-plane nodes, that ceiling is worth paying on
|
||||
# purpose: a kured MINOR may change drain or lock semantics, and this is the one component
|
||||
# where "a human looked at the changelog" should be a precondition, not a nice-to-have.
|
||||
# Renovate will raise the minor as a non-automerged PR (see the kured packageRule in
|
||||
# renovate.json), so the ceiling is a review prompt, not a wall.
|
||||
version: ">=6.1.0 <6.2.0"
|
||||
sourceRef:
|
||||
kind: HelmRepository
|
||||
name: kured
|
||||
namespace: kured
|
||||
install:
|
||||
remediation:
|
||||
retries: 3
|
||||
upgrade:
|
||||
remediation:
|
||||
retries: 3
|
||||
values:
|
||||
# ── IDENTITY / PLACEMENT ───────────────────────────────────────────────────────────────────
|
||||
# No tolerations. Measured 2026-10-17: all three prod nodes carry `spec.taints` EMPTY (k3s does
|
||||
# not taint its servers), so a DaemonSet reaches every node untainted. If a future node IS
|
||||
# tainted, kured silently stops covering it and that node then never reboots — invisible,
|
||||
# because the other two keep reporting. KuredNodeCoverageGap in alerts.yaml is the detector: it
|
||||
# compares kured's per-node series count against the node count rather than merely asking
|
||||
# whether kured exists.
|
||||
priorityClassName: system-node-critical
|
||||
# The chart's defaults are kept: `hostPID: true` and `containerSecurityContext.privileged: true`
|
||||
# with `readOnlyRootFilesystem: true`. Both are load-bearing and neither is reducible:
|
||||
# `rebootMethod: command` runs the reboot by nsenter'ing /proc/1/ns/mnt, which needs the host PID
|
||||
# namespace to see PID 1 and the privilege to enter its mount namespace. The documented
|
||||
# lower-privilege alternative (privileged:false + CAP_KILL) applies ONLY to
|
||||
# `rebootMethod: signal`, which signals a host-side kured-reboot.service that does not exist on
|
||||
# these nodes. Choosing `signal` would mean shipping and maintaining that unit via Ansible for a
|
||||
# capability reduction on a DaemonSet that is already cluster-admin-adjacent by function; it is a
|
||||
# reasonable future hardening step and is explicitly NOT taken here.
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
# No CPU limit on purpose (the repo's node-pressure work treats CPU limits on tiny system
|
||||
# agents as a throttling hazard). Memory is capped: kured holds no state beyond one
|
||||
# annotation and a gauge, and 128Mi is ~4x its idle RSS upstream reports.
|
||||
memory: 128Mi
|
||||
|
||||
configuration:
|
||||
# ── THE QUORUM CONSTRAINT: TWO NODES MUST NEVER BE DOWN AT ONCE ──────────────────────────
|
||||
# All three prod nodes are control-plane + etcd (measured 2026-10-17: every node reports
|
||||
# `control-plane,etcd`). etcd quorum is 2 of 3, so one node down is degraded-but-up and two
|
||||
# nodes down is a LOST CLUSTER that does not self-heal. Everything in this block exists to
|
||||
# make the second case structurally unreachable rather than merely unlikely.
|
||||
#
|
||||
# concurrency: 1 — kured's cluster-wide lock is a single annotation
|
||||
# (`weave.works/kured-node-lock`) on kured's OWN DaemonSet object, and a node reboots only
|
||||
# while it holds that lock. One lock + concurrency 1 = one reboot, enforced by the API server's
|
||||
# optimistic concurrency rather than by timing. This is the chart default; it is written out
|
||||
# explicitly because it is the single most important value in this file.
|
||||
concurrency: 1
|
||||
# lockTtl: 0 (DISABLED) — THIS IS A SAFETY DECISION, NOT A DEFAULT WE FORGOT TO TUNE.
|
||||
# A TTL makes the lock EXPIRE, which exists for the case "the lock holder died and we want
|
||||
# reboots to continue". On a 3-node etcd cluster that is precisely the catastrophe: if node A
|
||||
# has hung on a warm reboot (which has happened here — see the header) and its lock expires,
|
||||
# node B becomes eligible and takes the cluster from 2/3 to 1/3. We would rather the reboot
|
||||
# cycle STOP FOREVER and page a human than have it step over a corpse. The cost is that a
|
||||
# genuinely stale lock needs manual clearing; runbooks/node-reboot-automation.md has the
|
||||
# procedure, and KuredRebootPending is the alert that surfaces it.
|
||||
lockTtl: 0
|
||||
# lockReleaseDelay: 30m — hold the lock for 30 minutes AFTER this node is back and uncordoned,
|
||||
# before any other node may take it. See the Longhorn section: this is the STRUCTURAL half of
|
||||
# the inter-node gate, deliberately not dependent on an alert's `for:` timer having latched.
|
||||
lockReleaseDelay: 30m
|
||||
|
||||
# ── THE MAINTENANCE WINDOW: 03:00-06:00 EVERY DAY, America/New_York ──────────────────────
|
||||
# The timezone and the 03:00-06:00 band match renovate.json's existing
|
||||
# `"schedule": ["after 3am and before 6am"]` / `"timezone": "America/New_York"`, so every
|
||||
# unattended change in this repo lands in the same quiet 3-hour window.
|
||||
#
|
||||
# DAILY, NOT WEEKLY, AND THE REASON IS THE ALERT — not impatience. A weekly (`- sun`) window
|
||||
# was the first draft and it breaks the only alert that can tell you this mechanism has
|
||||
# stalled. With a Sunday-only window, a kernel upgraded on a Monday legitimately waits six
|
||||
# days, so "a reboot has been pending too long" cannot be distinguished from "it is Tuesday"
|
||||
# until the threshold exceeds a full 7-day cycle. That forces `for: 8d`+, which is longer than
|
||||
# this cluster's Prometheus retention (MONITORING_PROMETHEUS_RETENTION is 7d) and is reset by
|
||||
# any Prometheus restart beyond the 1h default for-outage-tolerance — i.e. the alert would be
|
||||
# unreliable in exactly the direction that matters. A daily window caps the legitimate wait at
|
||||
# under 24h, which makes KuredRebootPending's `for: 36h` both robust and meaningful: 36 hours
|
||||
# with a sentinel still set means at least one full window passed and SOMETHING BLOCKED IT.
|
||||
#
|
||||
# Daily does NOT weaken the quorum guarantee — that is `concurrency: 1` plus the lock, not the
|
||||
# calendar. It does mean a 3-node kernel rollout can finish in one to three nights instead of
|
||||
# three weeks, with each node still gated on the previous one's Longhorn rebuilds (below).
|
||||
rebootDays:
|
||||
- su
|
||||
- mo
|
||||
- tu
|
||||
- we
|
||||
- th
|
||||
- fr
|
||||
- sa
|
||||
startTime: "03:00"
|
||||
endTime: "06:00"
|
||||
timeZone: America/New_York
|
||||
# How often to re-evaluate (sentinel + window + alerts + lock). 1h is the chart default; at a
|
||||
# 3h window that is three evaluation opportunities per node per night.
|
||||
period: 1h
|
||||
|
||||
# ── THE SENTINEL ─────────────────────────────────────────────────────────────────────────
|
||||
# Written ONLY by a deliberate operator kernel upgrade (see the header). `update-notifier-common`
|
||||
# — the package that owns /etc/kernel/postinst.d and actually creates this file on Debian — was
|
||||
# measured ABSENT on all three nodes on 2026-10-17, which is why the unattended_upgrades role
|
||||
# installs it. Without that package nothing ever writes this path, kured correctly never
|
||||
# reboots, every probe looks healthy, and the entire mechanism is a silent no-op. That failure
|
||||
# mode is the reason KuredNotReporting exists.
|
||||
#
|
||||
# THE PATH kured LOGS IS NOT THIS PATH, and that trips people debugging it. The chart
|
||||
# defaults `useRebootSentinelHostPath: true` (kept), which bind-mounts the DIRNAME of this
|
||||
# value onto /sentinel read-only and renders the flag as
|
||||
# `--reboot-sentinel=/sentinel/reboot-required` (daemonset.yaml:134). So the container watches
|
||||
# /sentinel/reboot-required and the host file is /var/run/reboot-required. Same file, two
|
||||
# names. Set this value as the HOST path, as below.
|
||||
rebootSentinel: /var/run/reboot-required
|
||||
|
||||
# ── THE LONGHORN GATE: WHY POD READINESS IS NOT A SUFFICIENT INTER-NODE SIGNAL ────────────
|
||||
# Measured on prod 2026-10-17, and this is the finding that shaped this file:
|
||||
# * 25 Longhorn volumes, ALL `numberOfReplicas: 3`, all healthy, with exactly 25 replicas on
|
||||
# each of the 3 nodes. So there is no single-replica volume to lose today.
|
||||
# * BUT `replica-soft-anti-affinity: false` (replicas may not share a node) and
|
||||
# `disable-scheduling-on-cordoned-node: true`. With 3 nodes and 3 replicas, cordoning ONE
|
||||
# node leaves nowhere to place a replacement — so every drain takes ALL 25 volumes to
|
||||
# degraded 2/3 for the whole reboot.
|
||||
# * Recovery is slow and serialized: `replica-replenishment-wait-interval: 600` (10 minutes
|
||||
# before a rebuild even starts) and `concurrent-replica-rebuild-per-node-limit: 1`.
|
||||
# Therefore: if the next node is rebooted before rebuilds finish, those volumes go to 1/3, and
|
||||
# the drain after that genuinely CAN hit Longhorn's `node-drain-policy:
|
||||
# block-if-contains-last-replica` and block forever. The inter-node gate must be VOLUME
|
||||
# ROBUSTNESS, not "are the pods Ready again".
|
||||
#
|
||||
# It is gated twice, on purpose, because each half covers the other's blind spot:
|
||||
# 1. lockReleaseDelay: 30m (above) — unconditional. Covers the 600s replenishment wait with
|
||||
# margin and does not depend on any alert having fired.
|
||||
# 2. The Prometheus alert gate below — semantic. Covers a rebuild that takes LONGER than 30
|
||||
# minutes (25 serialized rebuilds on one node can), which a fixed delay cannot.
|
||||
prometheusUrl: http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090
|
||||
# Only consider alerts that are FIRING, not merely pending. A `pending` alert is a 10-minute
|
||||
# guess; blocking a weekly reboot on one would make the gate fire on noise.
|
||||
alertFiringOnly: true
|
||||
# ── alertFilterMatchOnly: true IS LOAD-BEARING. DO NOT DROP IT. ──────────────────────────
|
||||
# kured's two alert modes are near-opposites and the flag name does not advertise it:
|
||||
# * false (the chart default): alertFilterRegexp is an IGNORE list — kured blocks on ANY
|
||||
# firing alert except those matching.
|
||||
# * true: alertFilterRegexp is an ALLOW list — kured blocks ONLY on alerts that match.
|
||||
# We need `true`, because this cluster always has something firing. Measured via the Prometheus
|
||||
# API on 2026-10-17, these five were firing with no reboot pending and no end in sight:
|
||||
# CNPGPrimariesCoLocated, KubeDeploymentReplicasMismatch, KubeDeploymentRolloutStuck,
|
||||
# KubePodNotReady, OAuth2ProxyGateUnobserved (the last three all trace to the pm-agent pod,
|
||||
# ImagePullBackOff on a placeholder digest for 12 days). Under the default `false`, kured would
|
||||
# have been blocked by those on day one and would never have rebooted anything — a silent
|
||||
# no-op indistinguishable, from outside, from "no reboots were needed".
|
||||
alertFilterMatchOnly: true
|
||||
# The ALLOW list: the conditions under which rebooting ANOTHER node risks data or quorum.
|
||||
# Anchored and explicit — a prefix match here would silently re-admit the chronic five above.
|
||||
# LonghornVolumeDegraded — the inter-node gate (see above). `longhorn_volume_robustness
|
||||
# {state="degraded"} == 1`, for: 10m, in platform/monitoring/alerts.yaml.
|
||||
# LonghornVolumeFaulted — data already at risk; never reboot into this.
|
||||
# KubeNodeNotReady — a node is already down. Rebooting a second one loses quorum. This
|
||||
# is the single most important entry for the etcd constraint.
|
||||
# NodeExporterDown — a node we cannot see the health of. Treated as "assume the worst".
|
||||
# DELIBERATELY NOT INCLUDED: NodeEtcdRequestsSlow and the NodeIOPressure* family. They are
|
||||
# exactly the conditions you would want to pause for, but prod-1 has a documented history of
|
||||
# etcd WAL I/O starvation, so including them risks a permanent wedge that looks like safety.
|
||||
# Revisit once those are quiet for a sustained period; KuredRebootPending makes a wedge visible
|
||||
# either way.
|
||||
alertFilterRegexp: ^(LonghornVolumeDegraded|LonghornVolumeFaulted|KubeNodeNotReady|NodeExporterDown)$
|
||||
|
||||
# ── THE DRAIN ────────────────────────────────────────────────────────────────────────────
|
||||
# NOTE ON FLAGS THAT DO NOT EXIST: there is no `--ignore-daemonsets`, no
|
||||
# `--delete-emptydir-data` and no `--force` in kured 1.23.0. Read from
|
||||
# cmd/kured/main.go:438-449 on 2026-10-17, kured HARDCODES
|
||||
# `IgnoreAllDaemonSets: true`, `DeleteEmptyDirData: true` and `Force: true` in the drain
|
||||
# helper. Passing them as extraArgs would CrashLoop the DaemonSet on an unknown flag. This
|
||||
# matters because all three are genuinely REQUIRED here and it is easy to assume they must be
|
||||
# configured: 12 DaemonSets and 135 emptyDir-carrying pods were measured on 2026-10-17, and a
|
||||
# drain without those three settings refuses to start.
|
||||
#
|
||||
# drainTimeout: 15m — the chart/kured default is 0, meaning INFINITE, and infinite is the wrong
|
||||
# answer here. Measured 2026-10-17, six PodDisruptionBudgets currently allow ZERO voluntary
|
||||
# disruptions, and a plain drain loops on them indefinitely:
|
||||
# * 3x longhorn-system/instance-manager-* — one per node, `minAvailable: 1` against a
|
||||
# selector pinned to a single node's single pod, so disruptionsAllowed is STRUCTURALLY 0
|
||||
# forever. This is the mechanism by which Longhorn's node-drain-policy blocks: the
|
||||
# longhorn-manager deletes the node's instance-manager PDB only once it judges the drain
|
||||
# safe. The drain is SUPPOSED to wait on this one.
|
||||
# * 3x CNPG `<cluster>-primary` (keycloak-db, nextcloud-db, semaphore-db) — `minAvailable: 1`
|
||||
# against a selector matching exactly one pod (the primary), so again structurally 0 until
|
||||
# the CNPG operator switches the primary away. All three primaries were on k3s-prod-1.
|
||||
# With a timeout, a drain that is genuinely stuck ABORTS, the lock is released, and kured
|
||||
# retries next period instead of holding the cluster lock forever on a wedged node. 15m is
|
||||
# chosen to be longer than a CNPG switchover and longer than the `unifi` VM's eviction (below)
|
||||
# usually needs, while still being well inside the 3h window.
|
||||
drainTimeout: 15m
|
||||
# Taint a node PreferNoSchedule while its reboot is pending, so the scheduler stops placing new
|
||||
# pods on a node that is about to go away. With concurrency 1 this is a small win (there is no
|
||||
# second rebooting node to shed pods onto) but it is free and it shortens the drain.
|
||||
preferNoScheduleTaint: weave.works/kured-node-reboot
|
||||
# Record reboot state on the Node objects (`weave.works/kured-reboot-in-progress`,
|
||||
# `weave.works/kured-most-recent-reboot-needed`). Off by default. Turned on because the only
|
||||
# other observability kured offers is ONE gauge, and during an incident `kubectl get node -o
|
||||
# yaml` is the thing an operator can reach without Prometheus.
|
||||
annotateNodes: true
|
||||
logFormat: json
|
||||
|
||||
# ── METRICS: BOTH OF THESE MUST BE true OR THE ALERTS IN alerts.yaml CANNOT FIRE ───────────
|
||||
# The chart ships `service.create: false` AND `metrics.create: false`. Leaving either at its
|
||||
# default means no Service, or a ServiceMonitor with nothing to select, so `kured_reboot_required`
|
||||
# is never scraped — and a PrometheusRule over a series that no target produces is INERT: quiet
|
||||
# forever, indistinguishable from healthy. This repo has shipped that exact bug twice
|
||||
# (VeleroNoRecentBackup over an absent series; the inert gotk_* rules in
|
||||
# runbooks/image-automation.md), which is why platform/monitoring/tests/run.sh exists at all.
|
||||
# KuredNotReporting (alerts.yaml) is the standing detector for this pair regressing.
|
||||
# Prometheus selects ServiceMonitors cluster-wide: measured 2026-10-17, `serviceMonitorSelector`
|
||||
# and `serviceMonitorNamespaceSelector` are both `{}`, so no label plumbing is needed here.
|
||||
service:
|
||||
create: true
|
||||
port: 8080
|
||||
metrics:
|
||||
create: true
|
||||
interval: 60s
|
||||
@@ -1,12 +0,0 @@
|
||||
---
|
||||
# HelmRepository for the official kured chart (github.com/kubereboot/charts). Renovate maintains the
|
||||
# pinned chart version in helmrelease.yaml via its `flux` manager, whose managerFilePatterns already
|
||||
# cover `platform/**` (renovate.json). Same Flux source pattern as platform/headlamp.
|
||||
apiVersion: source.toolkit.fluxcd.io/v1
|
||||
kind: HelmRepository
|
||||
metadata:
|
||||
name: kured
|
||||
namespace: kured
|
||||
spec:
|
||||
interval: 1h
|
||||
url: https://kubereboot.github.io/charts
|
||||
@@ -1,36 +0,0 @@
|
||||
---
|
||||
# platform/kured — the Kubernetes Reboot Daemon.
|
||||
#
|
||||
# Rendered by its OWN Flux Kustomization (clusters/base/kured.yaml, dependsOn platform), NOT by a
|
||||
# line in platform/kustomization.yaml. The reasoning is the same one the Cilium split-out wrote down
|
||||
# and intel-gpu-plugin reused: inside the shared `wait: true` platform tier, this component's health
|
||||
# would be indistinguishable from eleven others, the only emergency valve would be suspending the
|
||||
# whole tier, and that tier's 5m timeout is wrong for a 3-node DaemonSet rollout. It matters more
|
||||
# than usual here, because `platform` gates platform-config which gates every apps wave — so a wedged
|
||||
# reboot daemon placed there could stall every application in the cluster. A component whose entire
|
||||
# purpose is to make maintenance safer must not be able to cause an outage.
|
||||
#
|
||||
# `namespace: kured` applies to every resource below, so none of them repeats it. Keep it: the
|
||||
# CiliumNetworkPolicy in networkpolicy.yaml is a CR whose namespace kustomize would otherwise leave
|
||||
# to chance.
|
||||
#
|
||||
# NO SECRET OF ANY KIND. kured authenticates to the API server with its own ServiceAccount token and
|
||||
# talks to nothing off-cluster — no registry credential, no webhook URL, no notification token. So
|
||||
# neither clusters/dev nor clusters/prod adds a SOPS decryption patch for this tier (same as
|
||||
# homepage/runbooks/intel-gpu-plugin). If Slack or shoutrrr notifications are ever wanted, the chart
|
||||
# takes them via `configuration.notifyUrl`, and THAT would need a SOPS-encrypted Secret plus a
|
||||
# decryption patch — do not inline a webhook URL here.
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
namespace: kured
|
||||
resources:
|
||||
- namespace.yaml
|
||||
- helmrepository.yaml
|
||||
- helmrelease.yaml # chart installs the DaemonSet / RBAC / SA / Service / ServiceMonitor
|
||||
# default-deny baseline + the Prometheus scrape ingress + the apiserver/Prometheus egress.
|
||||
# The scrape ingress is NOT optional decoration: without it the ServiceMonitor's targets are
|
||||
# discovered and every scrape times out, leaving the TSDB empty and all three rules in alerts.yaml
|
||||
# permanently quiet. See the header in that file for the measured Longhorn precedent.
|
||||
- networkpolicy.yaml
|
||||
# Three rules over kured's single metric, with a companion promtool unit test beside them.
|
||||
- alerts.yaml
|
||||
@@ -1,34 +0,0 @@
|
||||
---
|
||||
# kured — namespace for the Kubernetes Reboot Daemon DaemonSet.
|
||||
#
|
||||
# PLATFORM INFRA, NOT AN APPLICATION NAMESPACE, and here that distinction is the difference between
|
||||
# a working component and an admission rejection. kured runs `hostPID: true` with a `privileged: true`
|
||||
# container (it `nsenter`s /proc/1/ns/mnt to run `/bin/systemctl reboot` on the host). Both are
|
||||
# forbidden by platform/kyverno/policies/restrict-host-namespaces.yaml — but that policy, like every
|
||||
# other policy in this repo, is scoped by
|
||||
# `namespaceSelector: matchLabels: homelab.keithjackson.dev/app-namespace: "true"`
|
||||
# and its own header states the carve-out: "platform components that do [need these] live in platform
|
||||
# namespaces that do NOT carry the app-namespace label, so they are out of scope."
|
||||
# Verified by reading the policies on this branch, not assumed:
|
||||
# * restrict-host-namespaces.yaml rule `disallow-host-namespaces` (hostPID)
|
||||
# * restrict-host-namespaces.yaml rule `disallow-privileged-containers` (privileged)
|
||||
#
|
||||
# We set the label EXPLICITLY to "false" rather than leaving it unset, following
|
||||
# platform/intel-gpu-plugin/namespace.yaml and platform/vpn-egress: an unset label and a deliberate
|
||||
# "false" are identical to the admission engine but not to a reviewer, and this component's
|
||||
# correctness depends entirely on the answer.
|
||||
#
|
||||
# NO PodSecurity labels. There is no cluster-wide PSA baseline in this repo — `pod-security.*`
|
||||
# appears only where a namespace opts IN (conformance/incluster, platform/kyverno/admission,
|
||||
# platform/pm-agent), all at `enforce: restricted`. Adding them here would reject this DaemonSet for
|
||||
# the same reasons Kyverno would. Leaving them off is the convention for a privileged platform
|
||||
# namespace (platform/vpn-ingress/namespace.yaml sets none either).
|
||||
#
|
||||
# NO `gateway-access` label. kured serves no HTTP to users and attaches no HTTPRoute; its only
|
||||
# listener is the :8080 metrics endpoint, reached in-cluster by Prometheus. See networkpolicy.yaml.
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: kured
|
||||
labels:
|
||||
homelab.keithjackson.dev/app-namespace: "false"
|
||||
@@ -1,125 +0,0 @@
|
||||
---
|
||||
# Default-deny baseline (R-NET-01): deny all ingress + egress, allow only DNS. `kured` is a PLATFORM
|
||||
# namespace, so Kyverno's generator — scoped to `app-namespace=true` — does not produce this; we ship
|
||||
# it explicitly, as crafty/headlamp/homepage/runbooks all do.
|
||||
#
|
||||
# THESE POLICIES DO APPLY TO kured, UNLIKE platform/vpn-ingress'. That component's identical-looking
|
||||
# baseline is honest-but-inert because its DaemonSet runs `hostNetwork: true` and therefore has no
|
||||
# Cilium endpoint for any policy to select. kured's chart defaults to `hostNetwork: false` and we
|
||||
# keep that default (it needs the host PID namespace to reboot, not the host network stack), so it
|
||||
# has a normal pod identity and every rule here is enforced.
|
||||
#
|
||||
# Cilium enforces the UNION of these policies (allow if ANY policy allows), so the CiliumNetworkPolicy
|
||||
# below adds the kube-apiserver path on top of this deny-all floor.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: default-deny
|
||||
namespace: kured
|
||||
labels:
|
||||
homelab.keithjackson.dev/baseline: default-deny
|
||||
spec:
|
||||
podSelector: {}
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress: []
|
||||
egress:
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
---
|
||||
# ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# PROMETHEUS MUST REACH :8080, OR EVERY RULE IN alerts.yaml IS INERT.
|
||||
# ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# This is not a theoretical hazard in this repo, it is a measured one. From the Longhorn entry in
|
||||
# conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh: with the ServiceMonitor
|
||||
# applied and no NetworkPolicy allowance, all three targets were DISCOVERED and every scrape failed
|
||||
# with "context deadline exceeded" after 10.0s — "a drop, not a refusal" — and the TSDB stayed
|
||||
# empty. Target discovery succeeding is therefore NOT evidence the scrape works, and the alerts
|
||||
# would have been quiet rather than broken. KuredNotReporting is the standing detector for exactly
|
||||
# this regression, and it can only work if this policy is correct.
|
||||
#
|
||||
# THE WHOLE `monitoring` NAMESPACE, NOT THE PROMETHEUS POD LABELS — following the same choice
|
||||
# longhorn/networkpolicy-metrics.yaml made and for the same reason: the prometheus-operator owns
|
||||
# those pod labels and rewrites them on upgrade (measured 2026-10-17, the pod carries
|
||||
# `app.kubernetes.io/name: prometheus` plus five operator-managed labels including a
|
||||
# controller-revision-hash), so selecting on them makes the scrape silently fail on a chart bump.
|
||||
# The namespace holds only the monitoring stack.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: kured-allow-prometheus-scrape
|
||||
namespace: kured
|
||||
labels:
|
||||
app.kubernetes.io/name: kured
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
# Set by the chart's `kured.matchLabels` helper (_helpers.tpl:70-71) with release name `kured`.
|
||||
app.kubernetes.io/name: kured
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: monitoring
|
||||
ports:
|
||||
- protocol: TCP
|
||||
# The container port the chart names `metrics` (daemonset.yaml:211-213), from
|
||||
# `configuration.metricsPort`, which we leave at the chart default of 8080 and mirror in
|
||||
# the HelmRelease's `service.port`.
|
||||
port: 8080
|
||||
---
|
||||
# Egress allows. kured talks to exactly two things and nothing else; both are required and the
|
||||
# component is useless without either.
|
||||
#
|
||||
# 1. THE KUBE-APISERVER — the lock annotation on its own DaemonSet (`weave.works/kured-node-lock`),
|
||||
# cordon/uncordon (`nodes` patch), the node annotations, listing pods, and creating
|
||||
# `pods/eviction` objects during the drain. Reached via the `kube-apiserver` Cilium entity
|
||||
# rather than an IP: on this cluster the API is fronted by a kube-vip VIP that moves between
|
||||
# nodes on every reboot, so any CIDR-based rule here would break during the exact operation
|
||||
# this component exists to perform. Same construct as platform/headlamp/networkpolicy.yaml.
|
||||
#
|
||||
# 2. PROMETHEUS, on 9090 — the alert gate. The HelmRelease sets
|
||||
# `prometheusUrl: http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090`
|
||||
# and `alertFilterMatchOnly: true`, so before taking the cluster lock kured asks Prometheus
|
||||
# whether any of LonghornVolumeDegraded / LonghornVolumeFaulted / KubeNodeNotReady /
|
||||
# NodeExporterDown is firing. IF THIS EGRESS IS BLOCKED, kured cannot evaluate the gate.
|
||||
# It treats the query error as "do not proceed" and skips the reboot — which is the correct
|
||||
# fail-safe direction, but it fails SILENTLY and indefinitely: reboots simply stop happening
|
||||
# and nothing goes red except, eventually, KuredRebootPending. Do not remove this rule
|
||||
# thinking it is only for metrics; metrics ingress is the policy above, this is the gate.
|
||||
#
|
||||
# NOTHING ELSE. No egress to the internet (kured pulls no content at runtime), no DNS beyond the
|
||||
# baseline above, no Longhorn or etcd access — kured reads Longhorn's health THROUGH Prometheus
|
||||
# alerts, never from Longhorn's own API, which is why that allow is absent here.
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: kured-allow-egress
|
||||
namespace: kured
|
||||
labels:
|
||||
app.kubernetes.io/name: kured
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: kured
|
||||
egress:
|
||||
- toEntities:
|
||||
- kube-apiserver
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: monitoring
|
||||
app.kubernetes.io/name: prometheus
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: "9090"
|
||||
protocol: TCP
|
||||
@@ -63,75 +63,6 @@ spec:
|
||||
# gets oauth2-proxy's HTML sign-in page (HTTP 403) instead of an immediate redirect (docs/15 UX).
|
||||
skip-provider-button: "true"
|
||||
scope: "openid email profile"
|
||||
# ── Cold-start gate: do not start the proxy until Keycloak's discovery doc answers ──────────
|
||||
# MEASURED on prod 2026-10-17, during the UPS move: oauth2-proxy performs OIDC discovery ONCE at
|
||||
# startup and EXITS 1 if it fails. On a whole-cluster cold boot Keycloak is not serving yet, so
|
||||
# the Gateway answers 503 and the proxy dies:
|
||||
#
|
||||
# [provider.go:55] Performing OIDC Discovery...
|
||||
# [main.go:73] ERROR: Failed to initialise OAuth2 Proxy: ... failed to discover OIDC
|
||||
# configuration: unexpected status "503": no healthy upstream
|
||||
#
|
||||
# It then sits in exponential CrashLoopBackOff long after Keycloak is healthy -- one replica was
|
||||
# still down minutes later and only a manual pod delete cleared it. There is no retry flag; the
|
||||
# process simply does not survive a cold start. A readinessProbe cannot help (the container is
|
||||
# already dead), and CrashLoopBackOff backoff is not tunable per-pod.
|
||||
#
|
||||
# `extraInitContainers` is the chart's OWN documented hook for this -- its commented example in
|
||||
# values.yaml is literally `- name: wait-for-idp` -- and the chart already uses an alpine init
|
||||
# container for its built-in waitForRedis gate, so this adds no new base image to the platform.
|
||||
# A pod in Init does not accrue CrashLoopBackOff backoff and retries promptly, which is the whole
|
||||
# point.
|
||||
#
|
||||
# BOUNDED, AND FAILS OPEN. After ~5 minutes it gives up and starts the proxy anyway: the worst
|
||||
# case is then exactly today's behaviour, never something new. An unbounded wait would trade a
|
||||
# CrashLoop for a pod wedged in Init forever, which is not an improvement.
|
||||
#
|
||||
# SHELL VARIABLES ARE WRITTEN BARE ($i, $url, $(( ))), WITH NO BRACED FORM ANYWHERE. This file is
|
||||
# rendered through Flux envsubst (platform declares postBuild.substituteFrom cluster-vars), which
|
||||
# CONSUMES braced expansions -- so a braced shell variable renders to the empty string and
|
||||
# silently breaks the loop. BASE_DOMAIN below is braced ON PURPOSE: it is a real cluster-vars
|
||||
# knob and must be substituted. Same hazard, and the same convention, as the gitea and nextcloud
|
||||
# OIDC bootstrap hooks. (ci: flux-postbuild-substitution enforces it.)
|
||||
extraInitContainers:
|
||||
- name: wait-for-idp
|
||||
image: docker.io/library/alpine:3.22
|
||||
imagePullPolicy: IfNotPresent
|
||||
command: ["/bin/sh", "-c"]
|
||||
args:
|
||||
- |
|
||||
set -u
|
||||
url=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
|
||||
i=0
|
||||
while [ "$i" -lt 60 ]; do
|
||||
# busybox wget validates against alpine's ca-certificates-bundle, so a real TLS
|
||||
# failure is a non-zero exit here rather than a false pass.
|
||||
if wget -q -O /dev/null --timeout=5 "$url" 2>/dev/null; then
|
||||
echo "[wait-for-idp] Keycloak discovery reachable; starting oauth2-proxy"
|
||||
exit 0
|
||||
fi
|
||||
i=$(( i + 1 ))
|
||||
sleep 5
|
||||
done
|
||||
echo "[wait-for-idp] discovery still unreachable after ~5m; starting anyway so this" >&2
|
||||
echo "[wait-for-idp] gate can never be worse than the un-gated behaviour it replaces" >&2
|
||||
exit 0
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
runAsNonRoot: true
|
||||
runAsUser: 65534
|
||||
runAsGroup: 65534
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 16Mi
|
||||
limits:
|
||||
memory: 32Mi
|
||||
|
||||
# Metrics for the ServiceMonitor (../oauth2-proxy servicemonitor in platform-config).
|
||||
metrics:
|
||||
enabled: true
|
||||
|
||||
+5
-31
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
|
||||
"_comment": "Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). AS OF 2026-10-17 RENOVATE OWNS BOTH HALVES: Helm chart/dependency bumps AND per-service container image tags. It opens PRs for everything; allowed low-risk classes auto-merge on green CI. The ADR-0009 split that gave image tags to Flux image-automation is formally closed — see .gitea/workflows/renovate.yaml for the three measured reasons (it pushes commits rather than opening PRs, so it bypasses the CLAUDE.md §1 gate; `main` is protected with user_can_push:false and the flux-system GitRepository has no secretRef, so it cannot write anyway; and enabling it would need an in-cluster git WRITE credential for the repo that defines the cluster). image-automation has been commented out at platform/kustomization.yaml:52 since it was authored and the cluster holds zero ImagePolicy/ImageRepository/ImageUpdateAutomation objects, so nothing is being torn down. CONSEQUENCE: the `docker` and `kubernetes` managers, previously disabled ONLY to avoid fighting that controller, are now ENABLED — without them every plain `image: repo/name:tag` line in this repo was maintained by nothing at all, which is how the oauth2-proxy auth sidecars reached 7 minors behind the oauth2-proxy running in the same cluster. RUNNER: .gitea/workflows/renovate.yaml (scheduled daily); platform/endpoint are set there, not here.",
|
||||
"_comment": "Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). Renovate owns HELM CHART + DEPENDENCY bumps (opens PRs; allowed classes auto-merge on green CI). Flux image-automation owns per-service container IMAGE TAGS (the $imagepolicy setters). The two do NOT overlap: this config DISABLES image-tag updates so Renovate never fights Flux over a running version. Self-hosted against the bootstrap Gitea; platform: gitea set at runtime by the operator's renovate runner.",
|
||||
"extends": [
|
||||
"config:recommended",
|
||||
":dependencyDashboard",
|
||||
@@ -15,24 +15,13 @@
|
||||
"labels": ["dependencies", "automated-update"],
|
||||
"ignoreUnstable": true,
|
||||
"respectLatest": true,
|
||||
"enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "regex", "github-actions"],
|
||||
"enabledManagers": ["flux", "kustomize", "helmv3", "regex", "github-actions"],
|
||||
"flux": {
|
||||
"managerFilePatterns": ["/(^|/)platform/.+\\.ya?ml$/", "/(^|/)apps/.+\\.ya?ml$/"]
|
||||
},
|
||||
"kustomize": {
|
||||
"managerFilePatterns": ["/(^|/)kustomization\\.ya?ml$/"]
|
||||
},
|
||||
"_kubernetes_comment": "ADDED 2026-10-17. The `kubernetes` manager reads plain `image:` fields out of Kubernetes manifests, and it is NOT auto-detected — it only looks at files matched here, which is why this block is as broad as it is. Scope covers every place this repo writes a container image: platform components, the rendered app waves, the ServiceDefinition catalog that generates them, the runbooks web server and the pm-agent job template. `catalog/` matters as much as `apps/`: apps/wave-*/*.yaml are GENERATED by catalog/render.py, so a bump applied only to the rendered output is reverted by the next render and would also fail ci/render-golden.sh. Renovate will raise BOTH, and both must land together.",
|
||||
"kubernetes": {
|
||||
"managerFilePatterns": [
|
||||
"/(^|/)platform/.+\\.ya?ml$/",
|
||||
"/(^|/)platform-config/.+\\.ya?ml$/",
|
||||
"/(^|/)apps/.+\\.ya?ml$/",
|
||||
"/(^|/)catalog/.+\\.ya?ml$/",
|
||||
"/(^|/)runbooks/serving/.+\\.ya?ml$/",
|
||||
"/(^|/)infra/agent/.+\\.ya?ml$/"
|
||||
]
|
||||
},
|
||||
"customManagers": [
|
||||
{
|
||||
"customType": "regex",
|
||||
@@ -71,29 +60,14 @@
|
||||
"matchFileNames": ["ci/versions.env", "infra/devcontainer/versions.env"],
|
||||
"automerge": false,
|
||||
"addLabels": ["ci-tooling"]
|
||||
},
|
||||
{
|
||||
"description": "ADDED 2026-10-17. NEVER AUTO-MERGE THE BLAST-RADIUS COMPONENTS, not even a patch. The default rule above auto-merges minor/patch/digest on a green gate, which is right for an app image: a bad Sonarr bump inconveniences one pod. It is wrong for these five, because the gate is a STATIC check — it renders manifests and lints them, it does not boot a node, drain a volume or form an etcd quorum (vars.CLUSTER_AVAILABLE is unset, so L2/L3 conformance does not run in CI at all). So for anything whose failure mode is the CLUSTER rather than a service, a green gate is not evidence and a human must read the changelog. kured is here because it reboots control-plane nodes; longhorn because a storage-engine bump migrates live volume data; cilium because it is the CNI and a bad bump is a total partition (and its version is DUAL-PINNED in infra/ansible/roles/cilium/defaults/main.yml, which Renovate does not see, so the two would silently desync); cloudnative-pg because it drives Postgres switchovers; kube-prometheus-stack because ci/versions.env asserts an exact promtool/Prometheus lockstep and a chart bump that moves Prometheus fails the gate until that file moves with it.",
|
||||
"matchPackageNames": [
|
||||
"kured",
|
||||
"longhorn",
|
||||
"cilium",
|
||||
"cloudnative-pg",
|
||||
"kube-prometheus-stack"
|
||||
],
|
||||
"automerge": false,
|
||||
"addLabels": ["needs-review", "blast-radius"]
|
||||
}
|
||||
],
|
||||
"_unmanageable_comment": "WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A FLOATING tag has no version to compare, so Renovate skips it silently — it does not pin it and does not warn. As of 2026-10-17 that is: plexinc/pms-docker:latest, lscr.io/linuxserver/jackett:latest, pihole/pihole (NO TAG AT ALL), quay.io/keycloak/keycloak (no tag), nextcloud:31-apache and docker:27-dind. Those must be pinned BY HAND to a concrete version before any automation can maintain them, and until then they drift on every pull with nothing recording what is running. Separately, the HelmRelease minor-locking ranges (9 of 12, e.g. `>=88.0.0 <89.0.0` on kube-prometheus-stack) cap Renovate at patches inside a line it has been told not to leave; raising those ceilings is deliberate per-component work and is NOT part of enabling the bot. Both lists are in decisions/2026-10-17-automated-upgrades.md.",
|
||||
"ignorePaths": [
|
||||
"**/*.example",
|
||||
"**/*.sops.yaml",
|
||||
"conformance/**",
|
||||
"platform/image-automation/per-service/**",
|
||||
"platform/kyverno/tests/**"
|
||||
"platform/image-automation/per-service/**"
|
||||
],
|
||||
"_ignorePaths_comment": "platform/kyverno/tests/** ADDED 2026-10-17: those Kyverno policy fixtures pin `alpine:latest` and a `conformance-runner:latest`, which are test inputs whose whole point is to be a fixed shape, not software to keep current. `conformance/**` was already ignored for the same reason; this closes the matching hole that enabling the `kubernetes` manager would otherwise have opened.",
|
||||
"_docker_comment": "The `docker` manager (Dockerfiles, docker-compose) and the `kubernetes` manager above were BOTH previously `enabled: false`, solely so Renovate would not fight Flux image-automation over a running tag. That controller was never enabled and never will be here (see _comment), so the only thing those two flags were achieving was leaving every plain `image:` tag in the repo unmanaged. `docker` is left out of enabledManagers because this repo contains no Dockerfile that ships to the cluster; `kubernetes` is now ON and is the one that matters.",
|
||||
"docker": { "enabled": true }
|
||||
"docker": { "enabled": false },
|
||||
"kubernetes": { "enabled": false }
|
||||
}
|
||||
@@ -0,0 +1,162 @@
|
||||
# cluster-maintenance-shutdown — powering the whole cluster off, and back on
|
||||
|
||||
**Operator procedure, not an alert page.** Nothing links here from an alert.
|
||||
**Script:** `infra/maintenance/quiesce.sh` (`quiesce` | `resume` | `status`)
|
||||
**Related:** `/runbooks/update-rollback` is for rolling node work where the cluster stays up. This
|
||||
page is for taking **all three nodes down at once**.
|
||||
|
||||
First written after the 2026-10-17 UPS move, which was improvised live because no procedure
|
||||
existed — `infra/cluster.sh stop` is `virsh`-based and only ever applied to the decommissioned dev
|
||||
cluster, so it does nothing for these three baremetal nodes.
|
||||
|
||||
## Why you cannot just run `shutdown -h`
|
||||
|
||||
You mostly can now, and could not before.
|
||||
|
||||
Until 2026-10-17 **kubelet graceful node shutdown was unconfigured on every prod node**, so systemd
|
||||
stopped k3s, containerd went with it, and every pod was SIGKILLed where it stood — 9 Postgres
|
||||
instances mid-checkpoint, the UniFi VM, and 25 attached Longhorn volumes.
|
||||
`roles/k3s_server` now sets `shutdown-grace-period` (and, critically, the
|
||||
`InhibitDelayMaxSec` logind cap that actually enforces it — without that systemd stops waiting
|
||||
after 5s and the drain is silently truncated). A plain reboot of one node is therefore safe.
|
||||
|
||||
**A planned whole-cluster shutdown still wants `quiesce`**, for three things the kubelet's drain
|
||||
does not do:
|
||||
|
||||
1. **The UniFi VM needs an ACPI guest shutdown**, not an eviction. KubeVirt does that on
|
||||
`running: false`; an eviction is a power cut to the guest.
|
||||
2. **CloudNativePG wants hibernation**, not pod deletion — and hibernation is **imperative** state
|
||||
that Flux will neither set nor clear.
|
||||
3. **Draining three nodes one at a time makes pods reschedule onto the survivors**, churning
|
||||
Longhorn replicas for no reason when everything is going down anyway. Scaling in place avoids it.
|
||||
|
||||
## The hazard that bit us — read this before scaling anything by hand
|
||||
|
||||
**A StatefulSet can carry `persistentVolumeClaimRetentionPolicy.whenScaled: Delete`, and scaling it
|
||||
to 0 then DELETES its PVCs and their Longhorn volumes.**
|
||||
|
||||
On 2026-10-17 the hand-rolled quiesce scaled `monitoring/loki` (which has that policy) to 0, and
|
||||
`storage-loki-0` was deleted outright. Loki stores with `type: filesystem`, so that PVC was the only
|
||||
copy. Its 72h retention bounded the loss to a rolling window — nothing about the *mechanism* is
|
||||
bounded. The only visible symptom was the Longhorn volume count going 25 → 24, which was first
|
||||
misread as a miscount.
|
||||
|
||||
`quiesce.sh` now refuses to scale any such StatefulSet, leaves it running, and tells you which
|
||||
volumes to expect `degraded` on restart. **If you quiesce by hand, check the policy first:**
|
||||
|
||||
```bash
|
||||
kubectl --context k3s-prod get statefulset -A \
|
||||
-o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,WHEN_SCALED:.spec.persistentVolumeClaimRetentionPolicy.whenScaled'
|
||||
```
|
||||
|
||||
A hard-detached volume comes back `degraded` and rebuilds itself automatically. That is a vastly
|
||||
better outcome than a deleted one — take the rebuild.
|
||||
|
||||
## Shutting down
|
||||
|
||||
Run everything through the pinned toolbox; it supplies `kubectl` and the context.
|
||||
|
||||
```bash
|
||||
# 0. Look before you leap. Refuses to proceed if any Longhorn volume is unhealthy —
|
||||
# powering off on top of a degraded volume is how one replica becomes zero.
|
||||
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh status'
|
||||
|
||||
# 1. Quiesce: suspend Flux, ACPI-stop the VM, hibernate Postgres, scale down every
|
||||
# PVC-holding workload (NFS consumers first, PiHole last), wait for detach.
|
||||
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh quiesce'
|
||||
```
|
||||
|
||||
**Expect DNS to stop** when PiHole scales down — it is the LAN resolver. `kubectl` is unaffected
|
||||
because the kubeconfig points at the kube-vip IP `192.168.2.9`, not a name.
|
||||
|
||||
```bash
|
||||
# 2. Power the nodes off. Inside the devcontainer, which holds the become password.
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'cd /work/infra/ansible && ansible -i inventory/hosts.ini k3s_prod -b \
|
||||
-m shell -a "shutdown -h +1 \"maintenance\""'
|
||||
```
|
||||
|
||||
`+1` gives Ansible time to disconnect cleanly. The group resolves to all three nodes; confirm with
|
||||
`--list-hosts` if you want to see them first.
|
||||
|
||||
```bash
|
||||
# 3. Confirm they are actually down before touching anything else.
|
||||
for h in 192.168.2.10 192.168.2.11 192.168.2.12; do ping -c1 -W2 $h >/dev/null && echo "$h UP" || echo "$h down"; done
|
||||
```
|
||||
|
||||
**Then, in this order and not before:**
|
||||
|
||||
| # | Machine | How |
|
||||
|---|---|---|
|
||||
| 4 | **TrueNAS** `192.168.2.30` | **Web UI only.** Not in the ansible inventory, and SSH is **refused** (service disabled). `https://192.168.2.30` → power icon → Shut Down. Safe once the nodes are down, because every NFS consumer (`plex-media`, `*-media`, `qbittorrent-downloads`, `nextcloud-legacy-archive`) is already scaled to zero and nothing holds a hard mount. |
|
||||
| 5 | **Router / switch** | Last. |
|
||||
|
||||
TrueNAS must go **after** the k3s nodes: a hard NFS mount on a live node wedges kubelet in
|
||||
D-state (`/runbooks/nfs-export-latency`, and the load-average symptom in the 2026-10-?? note).
|
||||
|
||||
## Bringing it back
|
||||
|
||||
**Power on in the reverse order: router/switch → TrueNAS → all three nodes together.** etcd needs
|
||||
2 of 3 for quorum, so start the nodes as a group rather than one at a time.
|
||||
|
||||
```bash
|
||||
# 1. Nodes Ready, etcd quorum, storage reachable.
|
||||
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
||||
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get --raw=/healthz/etcd'
|
||||
|
||||
# 2. Resume: un-hibernate Postgres, then resume Flux.
|
||||
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh resume'
|
||||
```
|
||||
|
||||
**Un-hibernating Postgres is the step everyone forgets.** The annotation is imperative and survives
|
||||
a Flux resume, so resuming Flux alone leaves all three databases down and Keycloak, Nextcloud and
|
||||
Semaphore looking broken for reasons that point nowhere near Postgres. `resume` does it first, for
|
||||
exactly that reason.
|
||||
|
||||
Flux restores every scaled-down replica count **and** the VM's `running: true` from git, so nothing
|
||||
needs undoing by hand.
|
||||
|
||||
### What "still broken" looks like when it is actually fine
|
||||
|
||||
* **`4/19` Kustomizations Ready for several minutes.** The dependency graph serialises behind
|
||||
`platform-config`; 15 of them report `dependency ... is not ready` until it finishes. Check
|
||||
`platform-config` itself is `Ready=True` and its `lastAppliedRevision` matches the GitRepository
|
||||
artifact — if it is, wait.
|
||||
* **One or more Longhorn volumes `degraded`.** Any volume that was still attached at poweroff
|
||||
rebuilds its lagging replica automatically. On 2026-10-17 it was Prometheus's 50Gi TSDB; it went
|
||||
healthy on its own. Worry only if it is still degraded after ~30 minutes.
|
||||
* **`keycloak-operator` crashlooping, then recovering.** It is the CRD reconciler, not in the
|
||||
request path; Keycloak's StatefulSet stays 2/2 and auth never goes down. See
|
||||
`/runbooks/keycloak-*` and the known OOM-at-450Mi behaviour.
|
||||
|
||||
### What is actually broken and needs you
|
||||
|
||||
* **An `oauth2-proxy` replica stuck in CrashLoopBackOff.** It performs OIDC discovery at startup and
|
||||
**exits 1** on the `503 no healthy upstream` it gets while Keycloak is still coming up, then sits
|
||||
in exponential backoff long after Keycloak is fine. It has no startup dependency on Keycloak.
|
||||
Delete the pod once and it comes up clean:
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n oauth2-proxy delete pod <name>'
|
||||
```
|
||||
|
||||
* **Orphaned `Unknown`/`Error` Job pods** from CronJobs that were mid-run at poweroff. They never
|
||||
recover; delete them.
|
||||
|
||||
## Verifying you are actually back
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh status'
|
||||
# nodes 3 Ready | flux 19/19 | longhorn 0 not healthy | postgres 9 pods | unifi 1 VMI
|
||||
|
||||
# DNS, through PiHole's VIP
|
||||
./infra/devcontainer/dev.sh run bash -c 'nslookup home.keithjackson.dev 192.168.2.23'
|
||||
|
||||
# The Gateway actually serving (401 on an auth-gated host is CORRECT)
|
||||
for h in home.keithjackson.dev gitea.home.keithjackson.dev plex.home.keithjackson.dev; do
|
||||
curl -s -o /dev/null -w "$h %{http_code}\n" -k --resolve $h:443:192.168.2.20 https://$h/
|
||||
done
|
||||
```
|
||||
|
||||
Expect `200`, `200`, `401`.
|
||||
@@ -52,6 +52,13 @@ configMapGenerator:
|
||||
- backup-failure=backup-failure.md
|
||||
- backup-missed=backup-missed.md
|
||||
- backup-partial=backup-partial.md
|
||||
# Operator procedure, not an alert page — nothing links here from an alert. Served because
|
||||
# the whole-cluster shutdown had NO written procedure until 2026-10-17 (infra/cluster.sh is
|
||||
# virsh-based and only ever applied to the decommissioned dev cluster), so the UPS move was
|
||||
# improvised live, and the one hazard it exposed -- a StatefulSet with
|
||||
# persistentVolumeClaimRetentionPolicy whenScaled: Delete loses its PVC when scaled to 0 --
|
||||
# destroyed monitoring/loki's volume before anyone noticed.
|
||||
- cluster-maintenance-shutdown=cluster-maintenance-shutdown.md
|
||||
- cdi-import-stuck=cdi-import-stuck.md
|
||||
- cert-expiry=cert-expiry.md
|
||||
- cert-issuance=cert-issuance.md
|
||||
@@ -85,9 +92,6 @@ configMapGenerator:
|
||||
- node-io=node-io.md
|
||||
- node-memory=node-memory.md
|
||||
- node-memory-swap=node-memory-swap.md
|
||||
# Covers all three platform-health.node-reboots rules (KuredRebootPending,
|
||||
# KuredNodeCoverageGap, KuredNotReporting) from platform/kured/alerts.yaml.
|
||||
- node-reboot-automation=node-reboot-automation.md
|
||||
# Linked from OAuth2ProxyGateUnobserved, and the diagnostic procedure for any gated app
|
||||
# that "is up but does not work" — the gate answers an unauthenticated XHR with a 302,
|
||||
# not a 401, so the failure arrives as an HTTP 200 carrying the wrong body.
|
||||
|
||||
@@ -1,254 +0,0 @@
|
||||
# Node reboot automation (kured)
|
||||
|
||||
**Alerts:** `KuredRebootPending`, `KuredNodeCoverageGap`, `KuredNotReporting`
|
||||
**File:** `platform/kured/alerts.yaml`
|
||||
**Expressions:** `max by (node) (kured_reboot_required) == 1` / `count(kured_reboot_required) < count(kube_node_info)` / `absent(kured_reboot_required)`
|
||||
|
||||
## What this means here
|
||||
|
||||
These three alerts cover the OS reboot half of the automated upgrade process. The design is two
|
||||
pieces that only work together:
|
||||
|
||||
- **`roles/unattended_upgrades`** (Ansible, on the host) installs Debian **security** updates
|
||||
nightly. It never upgrades the kernel, never restarts k3s, and never reboots. When something it
|
||||
installed needs a reboot, it leaves `/var/run/reboot-required` behind.
|
||||
- **kured** (`platform/kured`, a DaemonSet) watches that file. When it exists, kured takes a
|
||||
**cluster-wide lock**, cordons the node, drains it, reboots it, and uncordons it — then holds the
|
||||
lock for a further 30 minutes before any other node may go.
|
||||
|
||||
**kured does not decide whether to reboot.** On these nodes the sentinel is written almost
|
||||
exclusively by a deliberate operator kernel upgrade
|
||||
(`playbooks/node-baseline.yml --tags kernel`), because the kernel is blacklisted from unattended
|
||||
upgrades. So a pending reboot normally means *a human already asked for this*.
|
||||
|
||||
Which alert you have tells you which half is wrong:
|
||||
|
||||
| alert | meaning |
|
||||
|---|---|
|
||||
| `KuredRebootPending` | The mechanism is alive and a reboot was offered, but it did not happen for >36h. **Something is blocking it.** |
|
||||
| `KuredNodeCoverageGap` | kured is not running on every node. The uncovered node will never be rebooted, silently. |
|
||||
| `KuredNotReporting` | kured is publishing nothing at all. Reboots are not happening, **and the other two alerts are inert.** Treat this as the most serious of the three. |
|
||||
|
||||
**All three nodes are control-plane + etcd, so quorum is 2 of 3.** One node down is degraded but
|
||||
up; two is a lost cluster. Everything below is written to protect that. `concurrency: 1` plus
|
||||
`lockTtl: 0` means a node that hangs on reboot keeps the lock **forever** and no second node is ever
|
||||
rebooted behind it. That is deliberate: the cycle stops and pages a human rather than stepping over
|
||||
a corpse.
|
||||
|
||||
**A reboot is not seamless.** Most workloads here are single-replica, so each reboot cycle takes a
|
||||
brief outage of whatever runs on that node — including `coredns` (1 replica), which means a
|
||||
cluster-wide DNS gap while it reschedules. See "Expected outage per cycle" below.
|
||||
|
||||
## Running the commands in this runbook
|
||||
|
||||
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
||||
container. Every command below runs as:
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
||||
|
||||
Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
|
||||
picks a current-context per invocation, and that default has already changed once. Do not run two
|
||||
`dev.sh` invocations at once; they share one container and interfere.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `uncordon`, `rollout`) and `flux reconcile` are
|
||||
operator actions, not agent actions. Where this runbook gives a mutating command it is for an
|
||||
operator, and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
Start here, in order. The first question is always *which half is broken*.
|
||||
|
||||
# 1. Is kured running at all, and on how many nodes?
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds,pod -o wide'
|
||||
|
||||
# 2. Which node is asking for a reboot, and who holds the lock?
|
||||
# The lock is an annotation on kured's OWN DaemonSet. An empty/absent value means no holder.
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds kured \
|
||||
-o jsonpath="{.metadata.annotations}"; echo'
|
||||
|
||||
# 3. Is a node cordoned, and is a reboot in progress?
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes \
|
||||
-o custom-columns=NAME:.metadata.name,SCHEDULABLE:.spec.unschedulable,TAINTS:.spec.taints'
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes \
|
||||
-o jsonpath="{range .items[*]}{.metadata.name}{\" \"}{.metadata.annotations.weave\.works/kured-reboot-in-progress}{\"\n\"}{end}"'
|
||||
|
||||
# 4. What is kured itself saying? It logs its decision every period (1h).
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured logs ds/kured --tail=80'
|
||||
|
||||
# 5. Is the gate blocking it? These are the ONLY alerts kured blocks on.
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n longhorn-system \
|
||||
get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness' \
|
||||
| grep -v healthy
|
||||
|
||||
# 6. Does the node actually want a reboot? (read-only; SSH as k3s, no sudo needed)
|
||||
ssh -o BatchMode=yes 192.168.2.10 'ls -la /var/run/reboot-required; cat /var/run/reboot-required.pkgs 2>/dev/null'
|
||||
|
||||
## Likely causes, most likely first
|
||||
|
||||
### 1. The Longhorn gate is holding it (the common `KuredRebootPending` cause)
|
||||
|
||||
kured queries Prometheus before taking the lock and refuses to proceed while
|
||||
`LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `KubeNodeNotReady` or `NodeExporterDown` is
|
||||
firing. This is working as designed and it is the **most likely** cause.
|
||||
|
||||
Why it lingers: there are 3 nodes, every volume has 3 replicas, and
|
||||
`replica-soft-anti-affinity: false` means replicas may not share a node. So cordoning one node
|
||||
leaves nowhere to put a replacement and **every volume goes degraded 2/3** for the duration.
|
||||
Recovery is slow on purpose: `replica-replenishment-wait-interval` is 600s before a rebuild even
|
||||
starts, and `concurrent-replica-rebuild-per-node-limit` is 1, so ~25 volumes rebuild one at a time.
|
||||
|
||||
Check progress, then wait:
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n longhorn-system \
|
||||
get volumes.longhorn.io -o json' | python3 -c 'import json,sys,collections; \
|
||||
v=json.load(sys.stdin)["items"]; print(collections.Counter(x["status"]["robustness"] for x in v))'
|
||||
|
||||
If volumes are rebuilding, **do nothing**. If a volume is stuck `degraded` or `faulted` with no
|
||||
rebuild progressing, that is a storage problem to fix first — see `/runbooks/longhorn-degraded`.
|
||||
Rebooting another node while a volume is at 1/3 is how you lose data.
|
||||
|
||||
### 2. The drain timed out against a PodDisruptionBudget
|
||||
|
||||
Six PDBs currently allow **zero** voluntary disruptions, and two kinds matter:
|
||||
|
||||
- **`longhorn-system/instance-manager-*`** (one per node). `minAvailable: 1` against a selector
|
||||
pinned to a single node's single pod, so `disruptionsAllowed` is structurally **always 0**.
|
||||
Longhorn's `longhorn-manager` *deletes* the PDB when its `node-drain-policy`
|
||||
(`block-if-contains-last-replica`) judges the drain safe. The drain is **supposed** to wait here.
|
||||
- **CNPG `<cluster>-primary`** (`keycloak-db`, `nextcloud-db`, `semaphore-db`). Also structurally 0
|
||||
until the CNPG operator switches the primary to another instance. If all three primaries are on
|
||||
the node being drained, three switchovers must happen.
|
||||
|
||||
kured's `drainTimeout` is 15m: a genuinely stuck drain **aborts**, releases the lock, and is retried
|
||||
next period rather than wedging forever. Repeated timeouts show in kured's logs.
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pdb -A -o wide' | awk '$NF==0 || NR==1'
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
|
||||
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PRIMARY:.status.currentPrimary'
|
||||
|
||||
If CNPG will not switch over, an **operator** can move the primary deliberately before the next
|
||||
window rather than fighting the drain.
|
||||
|
||||
### 3. A node hung on reboot and still holds the lock
|
||||
|
||||
`k3s-prod-2` has done this before (2026-09-03: powered, off the LAN ~38 minutes, physical power
|
||||
button required). `lockTtl: 0` means the lock does **not** expire, so the cycle stops here —
|
||||
correctly, since the cluster is at 2/3 and must not go to 1/3.
|
||||
|
||||
Symptoms: one node `NotReady` or absent, `KuredNodeCoverageGap` firing alongside
|
||||
`KuredRebootPending`, and the lock annotation naming the missing node.
|
||||
|
||||
**Operator recovery:** attend to the node physically. Once it is back and `Ready`, kured clears its
|
||||
own lock on the next period. If the node is gone for good and the lock must be cleared by hand,
|
||||
that is an operator action:
|
||||
|
||||
# OPERATOR ONLY. Only when you have confirmed the named node is NOT mid-reboot.
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured annotate ds kured \
|
||||
weave.works/kured-node-lock- --overwrite'
|
||||
|
||||
If the node was pruned mid-reboot and is left **cordoned** with no daemon to uncordon it:
|
||||
|
||||
# OPERATOR ONLY.
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod uncordon <node>'
|
||||
|
||||
### 4. Nothing can write the sentinel (so nothing ever reboots)
|
||||
|
||||
If `KuredRebootPending` never fires and reboots never happen even after a kernel upgrade, suspect
|
||||
the sentinel writer. On Debian, `/var/run/reboot-required` is created by a hook shipped in
|
||||
**`update-notifier-common`** — and it was absent on all three nodes before this automation landed.
|
||||
Without it kured is permanently, invisibly idle.
|
||||
|
||||
ssh -o BatchMode=yes 192.168.2.10 'dpkg -l update-notifier-common needrestart unattended-upgrades | tail -4'
|
||||
ssh -o BatchMode=yes 192.168.2.10 'ls /etc/kernel/postinst.d/ /etc/apt/apt.conf.d/ | sort'
|
||||
|
||||
Fix by re-running the role, which asserts this explicitly:
|
||||
|
||||
cd infra/ansible && ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify
|
||||
|
||||
### 5. kured is not being scraped (`KuredNotReporting`)
|
||||
|
||||
The chart defaults `service.create` and `metrics.create` to **false**, and the `kured` namespace
|
||||
ships a default-deny NetworkPolicy. If either enablement regressed, or the Prometheus ingress allow
|
||||
was dropped, targets are *discovered* and every scrape times out — the TSDB stays empty and all
|
||||
three rules go quiet. Longhorn hit exactly this failure on this cluster.
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get svc,servicemonitor,networkpolicy,ciliumnetworkpolicy'
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n monitoring exec \
|
||||
sts/prometheus-kube-prometheus-stack-prometheus -c prometheus -- \
|
||||
wget -qO- "http://localhost:9090/api/v1/targets?state=active" ' | grep -o '"job":"kured[^,]*' | head
|
||||
|
||||
### 6. A node is uncovered by the DaemonSet (`KuredNodeCoverageGap`)
|
||||
|
||||
kured declares **no tolerations**, because no prod node is tainted. A newly provisioned or
|
||||
re-provisioned node carrying a taint gets no kured pod and will never be rebooted.
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes \
|
||||
-o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints'
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds kured \
|
||||
-o custom-columns=DESIRED:.status.desiredNumberScheduled,READY:.status.numberReady'
|
||||
|
||||
The fix is a PR adding the needed toleration to `platform/kured/helmrelease.yaml` — not a `kubectl
|
||||
patch`, which Flux reverts.
|
||||
|
||||
## Expected outage per reboot cycle
|
||||
|
||||
Stated plainly, because this process is **not** seamless. Every workload at 1 replica on the
|
||||
draining node is down until it reschedules and becomes Ready elsewhere. Measured 2026-10-17: **45
|
||||
Deployments and 3 StatefulSets run a single replica.** The ones worth knowing about:
|
||||
|
||||
- **`kube-system/coredns` — 1 replica.** A cluster-wide in-cluster DNS gap while it moves. This is
|
||||
the single most impactful item and it is **not** fixed by this PR; raising coredns to 2–3 replicas
|
||||
is recommended follow-up work.
|
||||
- **`monitoring/prometheus-…-0`, `monitoring/loki-0`, `monitoring/alertmanager-…-0`** — all
|
||||
single-replica StatefulSets on Longhorn RWO volumes, so they additionally wait for volume
|
||||
detach/reattach. **You lose monitoring and alerting for part of the cycle**, including these very
|
||||
alerts.
|
||||
- **All three CNPG Postgres primaries** blip during switchover, which means Keycloak, Nextcloud and
|
||||
Semaphore briefly lose their database.
|
||||
- **`gitea`, `nextcloud`, `plex`, `jackett`, `qbittorrent`, `radarr`, `sonarr`, `seerr`, `crafty`,
|
||||
`homepage`, `headlamp`, `semaphore`** — each down for its reschedule.
|
||||
- **`unifi` (a KubeVirt VM)** has `evictionStrategy: LiveMigrate` but
|
||||
`migrationMethod: BlockMigration`, so evicting it copies the whole root disk over the network.
|
||||
Expect the drain of its node to be slow.
|
||||
- **`kube-vip`** fails the control-plane VIP over on every node reboot.
|
||||
|
||||
Multi-replica and therefore fine: `cilium-operator`, `oauth2-proxy`, `kyverno-admission-controller`,
|
||||
the Longhorn CSI sidecars, `keycloak`, `pihole`, `runbooks`.
|
||||
|
||||
## When to escalate or stop
|
||||
|
||||
**STOP and do not intervene** if:
|
||||
|
||||
- Longhorn volumes are `degraded` **and rebuilding**. This is the gate doing its job. Wait.
|
||||
- A single node is mid-reboot and under 15 minutes have passed.
|
||||
|
||||
**STOP and get a human to the hardware** if:
|
||||
|
||||
- Any node is `NotReady` while another reboot is pending. **Never clear the kured lock in this
|
||||
state** — that is the one action that can take etcd from 2/3 to 1/3 and lose the cluster.
|
||||
- Two nodes are down. The cluster has no quorum; this runbook cannot help. Recover hardware first.
|
||||
|
||||
**Escalate to a deliberate maintenance decision** if:
|
||||
|
||||
- `KuredRebootPending` has been firing for over a week. The reboot is not going to happen on its
|
||||
own; find the blocker above, or take the reboot manually during a watched window.
|
||||
- A volume is `faulted`. Fix storage before any further reboot.
|
||||
|
||||
**Turn the automation off** if it is misbehaving and you need the cluster stable. The honest way is
|
||||
a PR, but for an emergency an operator may suspend the tier — this stops future reboots and leaves
|
||||
running nodes untouched:
|
||||
|
||||
# OPERATOR ONLY. Stops kured being reconciled; does NOT uncordon a node mid-cycle.
|
||||
infra/devcontainer/dev.sh run bash -c 'flux --context k3s-prod suspend kustomization kured'
|
||||
|
||||
Check afterwards whether a node was left cordoned (First commands, step 3) and uncordon it if so.
|
||||
Resume with `flux resume kustomization kured`. Nothing in the graph depends on `kured`, so
|
||||
suspending it stalls nothing else.
|
||||
|
||||
---
|
||||
|
||||
Source of truth: `runbooks/node-reboot-automation.md`. Index: `/runbooks/`
|
||||
|
||||
Records: `decisions/2026-10-17-automated-upgrades.md` (design + what was not verified),
|
||||
`platform/kured/helmrelease.yaml` (every tuned value and why),
|
||||
`infra/ansible/roles/unattended_upgrades/README.md` (the package half, and the kernel exclusion).
|
||||
Reference in new issue
Block a user