Two halves of one repeatable upgrade process. Neither works alone. (a) OS upgrades and reboots platform/kured/ — Kubernetes Reboot Daemon, chart 6.1.0/app 1.23.0, in its own Flux Kustomization (dependsOn platform, wait:true). NOT in platform/kustomization.yaml: that tier gates platform-config and every apps wave, so a wedged reboot daemon there could stall every application. A component added to make maintenance safer must not be able to cause an outage. infra/ansible/roles/unattended_upgrades/ — new role (node_baseline is touched by other open branches, and this is a different lifecycle), enabled for prod in group_vars. The human gate is the SENTINEL, which is how this honours node_baseline's "IT DOES NOT REBOOT" (k3s-prod-2 hung on a warm reboot 2026-09-03) and cluster-upgrade-plan.md's rejection of automating away the gate between nodes: u-u is security-only with the kernel blacklisted, so it essentially never writes /var/run/reboot-required. That file appears when an operator deliberately runs node-baseline.yml --tags kernel. kured then does only the mechanical half, including the ordering rule node_baseline has to shout. Quorum: concurrent reboots are structurally impossible, not merely unlikely. concurrency:1 plus kured's single lock annotation (API-server optimistic concurrency), and lockTtl:0 — an expiring lock is exactly the 3-node etcd catastrophe, so a hung node holds it forever and nothing cascades. Longhorn: measured 25 volumes x 3 replicas, hard anti-affinity, 600s replenishment wait, serialized rebuild — so every drain degrades ALL volumes to 2/3 and the inter-node gate must be volume robustness, not pod readiness. Doubled: lockReleaseDelay 30m (unconditional) plus kured's Prometheus alert gate (semantic). alertFilterMatchOnly:true is load-bearing — the flag inverts the regexp's meaning, and under the chart default kured would have been blocked on day one by the five alerts measured firing with no end in sight, and never rebooted anything. Kernel floor (kernel >= 6.13, EFI memattr): excluded TWICE. The running kernel comes from o=Debian Backports, which is not an allowed origin, so u-u cannot see a kernel upgrade; and ^linux- is blacklisted anyway so widening origins later cannot re-arm it. Asserted against apt-config dump, not against the file just written. 6.12 kernels are still installed and bootable on all three nodes, and nothing was held. Closes a silent no-op: update-notifier-common was absent on all three nodes, so NOTHING could create the sentinel. kured alone would have idled forever at 1/1 Running. platform/kured/alerts.yaml — three rules over kured's only metric, with a companion promtool suite pinning each both ways, plus runbooks/node-reboot-automation.md. Window is DAILY 03:00-06:00 because a weekly one forces for:8d, longer than this cluster's 7d Prometheus retention. (b) Image/dependency automation Renovate was aspirational: renovate.json present and complete, with no workflow, no in-cluster workload, and its docker+kubernetes managers disabled — so every plain image: tag was maintained by nothing while seven files claimed otherwise. .gitea/workflows/renovate.yaml — scheduled daily, dry_run default true for a safe first run, and a preflight that FAILS (not skips) without RENOVATE_TOKEN, because a skipped Gitea job reports success and that is how this stayed inert. Renovate now owns both halves; Flux image-automation is formally closed: it pushes commits rather than opening PRs (bypassing CLAUDE.md §1), and it cannot work here anyway — main is protected user_can_push:false and the flux-system GitRepository has no secretRef. Its controllers are installed and have idled 43 days with zero CRs. conformance/: +34/-0, one new SCRAPE_OWNERS entry claiming kured_. Weakens nothing; tightens the check for this component (requires service.create+metrics.create, both chart-default false, plus a companion NetworkPolicy requirement). Required because assertion (3) fails an alert over an unclaimed metric and KNOWN_UNWIRED_METRICS is shrink-only. Sensitive paths touched: clusters/**, conformance/**, .gitea/** — needs a non-author approving review; /approve-override deliberately NOT posted. NOT verified: no OS upgrade run, no node rebooted, no lock observed, the Ansible role never executed, and whether CNPG switches its primary on a drain-induced eviction (the most likely way the first real reboot fails). Full list in the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
310 lines
18 KiB
YAML
310 lines
18 KiB
YAML
---
|
|
# roles/unattended_upgrades — automatic Debian security patching, kernel excluded, reboots delegated
|
|
# to kured. Every decision and its reasoning lives in defaults/main.yml; this file is the mechanism.
|
|
#
|
|
# TAGS — following roles/node_baseline, every task carries one of `packages`, `config` or `verify`.
|
|
# An untagged run does all three in that order. The split exists because `verify` is the half an
|
|
# operator will want to re-run on its own to answer "is automatic patching actually working on this
|
|
# box", without re-writing any configuration:
|
|
# ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify
|
|
#
|
|
# IT DOES NOT UPGRADE ANYTHING, AND IT DOES NOT REBOOT. This role installs and CONFIGURES the
|
|
# upgrader; it never runs `apt upgrade` itself and never takes a reboot. Both of those are deliberate
|
|
# and both follow roles/node_baseline/tasks/main.yml:61-67 ("IT DOES NOT REBOOT", after k3s-prod-2
|
|
# hung on a warm reboot on 2026-09-03 and needed a physical power button). Applying this role is
|
|
# therefore a SAFE operation on a live cluster at any time of day: the first unattended upgrade
|
|
# happens at the next timer firing, and any reboot it implies is kured's to schedule inside the
|
|
# 03:00-06:00 window.
|
|
- name: Automatic security upgrades (unattended-upgrades; kernel excluded, reboots via kured)
|
|
when: unattended_upgrades_enabled | bool
|
|
tags: [packages, config, verify]
|
|
block:
|
|
|
|
# ── PREFLIGHT ───────────────────────────────────────────────────────────────────────────────
|
|
- name: Assert this is a Debian-family host
|
|
# The config file layout, the origin expressions and the package names below are all
|
|
# Debian-specific. On anything else this role would write files that are silently ignored,
|
|
# which is the failure mode this whole PR exists to avoid.
|
|
ansible.builtin.assert:
|
|
that:
|
|
- ansible_facts['os_family'] == 'Debian'
|
|
fail_msg: >-
|
|
REFUSING TO CONFIGURE unattended-upgrades on {{ inventory_hostname }}: os_family is
|
|
'{{ ansible_facts["os_family"] }}', not 'Debian'. The allowed-origin expressions and
|
|
config paths in this role are Debian-specific.
|
|
tags: [packages, config, verify]
|
|
|
|
# ── PACKAGES ────────────────────────────────────────────────────────────────────────────────
|
|
- name: Install unattended-upgrades, the reboot-sentinel writer and needrestart
|
|
# update-notifier-common is NOT optional padding: it owns the hook that creates
|
|
# /var/run/reboot-required, which is the only thing kured watches. See defaults/main.yml.
|
|
ansible.builtin.apt:
|
|
name: "{{ unattended_upgrades_packages }}"
|
|
state: present
|
|
update_cache: true
|
|
cache_valid_time: 3600
|
|
tags: [packages]
|
|
|
|
# ── CONFIG ──────────────────────────────────────────────────────────────────────────────────
|
|
# Written as a whole file rather than patched in place, for the reason node_baseline gives for
|
|
# its own apt source: an idempotent full-file write with a managed-by header is reviewable and
|
|
# re-runnable, whereas line-editing a conffile fights dpkg on every package upgrade.
|
|
#
|
|
# 52- RATHER THAN 50-: Debian ships /etc/apt/apt.conf.d/50unattended-upgrades as a CONFFILE of
|
|
# the unattended-upgrades package. Overwriting it would make every future apt upgrade of that
|
|
# package prompt about a locally-modified conffile (or, under the non-interactive defaults a
|
|
# scripted run uses, silently keep one side and discard the other). apt reads the directory in
|
|
# lexical order and later files win, so a separate 52- file overrides the shipped defaults
|
|
# without ever touching them, and `apt-config dump` in the verify block below proves which value
|
|
# actually won rather than assuming this ordering holds.
|
|
- name: Configure unattended-upgrades (52-k3s-unattended-upgrades)
|
|
ansible.builtin.copy:
|
|
dest: /etc/apt/apt.conf.d/52-k3s-unattended-upgrades
|
|
owner: root
|
|
group: root
|
|
mode: "0644"
|
|
content: |
|
|
// MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades. Local edits are reverted
|
|
// on the next run. Rationale for every value: that role's defaults/main.yml.
|
|
//
|
|
// Overrides the shipped 50unattended-upgrades by lexical ordering (52 > 50). The shipped
|
|
// file is a dpkg conffile and is deliberately left untouched.
|
|
|
|
// SECURITY ONLY. Deliberately excludes trixie-updates (non-security fixes) and
|
|
// trixie-backports (o=Debian Backports) -- and excluding backports is what makes it
|
|
// structurally impossible for this upgrader to move the EFI-memattr-critical kernel,
|
|
// which is installed from there. See defaults/main.yml DECISION 2.
|
|
Unattended-Upgrade::Origins-Pattern {
|
|
{% for origin in unattended_upgrades_allowed_origins %}
|
|
"{{ origin }}";
|
|
{% endfor %}
|
|
{% if unattended_upgrades_include_updates | bool %}
|
|
"origin=Debian,codename=${distro_codename}-updates,label=Debian";
|
|
{% endif %}
|
|
};
|
|
|
|
// NEVER AUTOMATICALLY UPGRADED, whatever the origin list later says. `^linux-` is the
|
|
// EFI-memattr guard: these HP minis need kernel >= 6.13 or they can boot with a fraction
|
|
// of their RAM, and 6.12 kernels are still installed and bootable on all three nodes.
|
|
// Do not remove without re-reading infra/PROD-PROVISIONING.md §0b.
|
|
Unattended-Upgrade::Package-Blacklist {
|
|
{% for pkg in unattended_upgrades_blacklist %}
|
|
"{{ pkg }}";
|
|
{% endfor %}
|
|
};
|
|
|
|
// REBOOTS BELONG TO kured (platform/kured), NOT TO apt. All three nodes are
|
|
// control-plane + etcd with quorum 2 of 3, so three hosts independently deciding to
|
|
// reboot at their own Automatic-Reboot-Time is how the cluster is lost. u-u leaves
|
|
// /var/run/reboot-required behind; kured takes a cluster-wide lock and reboots exactly
|
|
// one node at a time, after cordon + drain, inside 03:00-06:00, gated on Longhorn volume
|
|
// health.
|
|
Unattended-Upgrade::Automatic-Reboot "{{ unattended_upgrades_automatic_reboot | bool | lower }}";
|
|
Unattended-Upgrade::Automatic-Reboot-WithUsers "false";
|
|
|
|
// Keep the node's disk from filling with superseded .debs, and keep a log an operator can
|
|
// read after the fact (/var/log/unattended-upgrades/). MinAge guards against removing a
|
|
// package still needed by a rollback.
|
|
Unattended-Upgrade::Remove-Unused-Kernel-Packages "false";
|
|
Unattended-Upgrade::Remove-New-Unused-Dependencies "true";
|
|
Unattended-Upgrade::Remove-Unused-Dependencies "false";
|
|
Unattended-Upgrade::AutoFixInterruptedDpkg "true";
|
|
Unattended-Upgrade::MinimalSteps "true";
|
|
Unattended-Upgrade::SyslogEnable "true";
|
|
|
|
// Remove-Unused-Kernel-Packages is FALSE on purpose, against the Debian default. These
|
|
// nodes deliberately carry more than one kernel: the backports kernel they boot and the
|
|
// distro kernels that are the documented rollback path (node_baseline writes the rollback
|
|
// procedure onto the node itself). An unattended process pruning kernels could remove the
|
|
// fallback an operator needs at 3am in front of a box that will not boot.
|
|
notify: Restart unattended-upgrades timers
|
|
tags: [config]
|
|
|
|
- name: Enable the apt periodic timers that actually run unattended-upgrades
|
|
# Installing the package is NOT sufficient to make it run on a schedule: without these
|
|
# counters apt's daily units do nothing, and the whole mechanism is inert while looking
|
|
# installed. `1` means "every day".
|
|
ansible.builtin.copy:
|
|
dest: /etc/apt/apt.conf.d/20auto-upgrades
|
|
owner: root
|
|
group: root
|
|
mode: "0644"
|
|
content: |
|
|
// MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades.
|
|
// Without these four counters the apt-daily timers fire and do nothing, so
|
|
// unattended-upgrades would be installed, configured, and never once invoked.
|
|
APT::Periodic::Update-Package-Lists "1";
|
|
APT::Periodic::Download-Upgradeable-Packages "1";
|
|
APT::Periodic::Unattended-Upgrade "1";
|
|
APT::Periodic::AutocleanInterval "7";
|
|
tags: [config]
|
|
|
|
- name: Ensure the systemd timer drop-in directories exist
|
|
# `copy` does not create parent directories, and these two .timer.d/ dirs do not exist on a
|
|
# stock Debian node — without this task the next task fails with "Destination directory does
|
|
# not exist" on a first run.
|
|
ansible.builtin.file:
|
|
path: "/etc/systemd/system/{{ item }}.timer.d"
|
|
state: directory
|
|
owner: root
|
|
group: root
|
|
mode: "0755"
|
|
loop:
|
|
- apt-daily
|
|
- apt-daily-upgrade
|
|
tags: [config]
|
|
|
|
- name: Pin the apt timer windows into kured's reboot window
|
|
# Overriding the shipped timers rather than editing them: a drop-in survives package upgrades.
|
|
# OnCalendar is cleared first (`OnCalendar=` with no value) because systemd ACCUMULATES
|
|
# OnCalendar entries across drop-ins — omit the reset and the unit keeps its shipped schedule
|
|
# IN ADDITION to ours, which would run the upgrade at both times and look like it worked.
|
|
ansible.builtin.copy:
|
|
dest: "/etc/systemd/system/{{ item.unit }}.timer.d/override.conf"
|
|
owner: root
|
|
group: root
|
|
mode: "0644"
|
|
content: |
|
|
# MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades.
|
|
# Lands the upgrade inside kured's 03:00-06:00 reboot window so a sentinel written by an
|
|
# upgrade can be acted on the same night rather than ~24h later.
|
|
[Timer]
|
|
OnCalendar=
|
|
OnCalendar={{ item.time }}
|
|
RandomizedDelaySec={{ unattended_upgrades_randomised_delay }}
|
|
Persistent=true
|
|
force: true
|
|
loop:
|
|
- { unit: apt-daily, time: "{{ unattended_upgrades_download_time }}" }
|
|
- { unit: apt-daily-upgrade, time: "{{ unattended_upgrades_upgrade_time }}" }
|
|
loop_control:
|
|
label: "{{ item.unit }} at {{ item.time }}"
|
|
notify: Restart unattended-upgrades timers
|
|
tags: [config]
|
|
|
|
- name: Configure needrestart (restart services, but never k3s)
|
|
# See defaults/main.yml DECISION 3. Auto-restart is what makes a security-only policy real --
|
|
# a patched library on disk does nothing for a process still mapping the deleted one -- but
|
|
# k3s.service is excluded because restarting the control plane unattended is worse than a
|
|
# delayed library fix. `$nrconf{restart}` is Perl config read by needrestart itself.
|
|
ansible.builtin.copy:
|
|
dest: /etc/needrestart/conf.d/50-k3s.conf
|
|
owner: root
|
|
group: root
|
|
mode: "0644"
|
|
content: |
|
|
# MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades.
|
|
# Restart mode: '{{ unattended_upgrades_needrestart_mode }}' (a = automatically, l = list only, i = interactive)
|
|
$nrconf{restart} = '{{ unattended_upgrades_needrestart_mode }}';
|
|
# Units matching these patterns are NEVER restarted automatically. k3s bundles containerd
|
|
# and the kubelet, so this one entry covers the entire control plane and runtime on these
|
|
# nodes (measured 2026-10-17: k3s.service is the only matching unit). A library fix that
|
|
# needs k3s restarted therefore waits for the reboot kured will take.
|
|
$nrconf{blacklist_rc} = [
|
|
{% for pat in unattended_upgrades_needrestart_blacklist %}
|
|
qr({{ pat }}),
|
|
{% endfor %}
|
|
];
|
|
tags: [config]
|
|
|
|
# ── VERIFY ──────────────────────────────────────────────────────────────────────────────────
|
|
# These tasks exist because every component of this mechanism fails SILENTLY. An installed
|
|
# package, a written config file and a green play are all compatible with nothing ever being
|
|
# upgraded and no reboot ever being signalled.
|
|
|
|
- name: Ask apt-config what the effective settings actually are
|
|
# `apt-config dump` is apt's OWN parse of the whole /etc/apt/apt.conf.d/ directory, so it
|
|
# proves what WON after lexical ordering rather than what this role wrote. If the shipped
|
|
# 50unattended-upgrades were ever renamed above 52-, this is the task that would catch it.
|
|
ansible.builtin.command: apt-config dump
|
|
register: unattended_upgrades_aptconfig
|
|
changed_when: false
|
|
check_mode: false
|
|
tags: [verify]
|
|
|
|
- name: Assert the kernel is blacklisted and automatic reboots are off
|
|
# The two properties that must never silently regress. Asserted against apt's own view.
|
|
ansible.builtin.assert:
|
|
that:
|
|
# apt-config renders each list entry as `Unattended-Upgrade::Package-Blacklist:: "^linux-";`
|
|
# so the anchored pattern itself is the thing to look for. Checking for the bare string
|
|
# `linux-` would pass on almost any dump and prove nothing.
|
|
- "'\"^linux-\"' in unattended_upgrades_aptconfig.stdout"
|
|
- "'Unattended-Upgrade::Automatic-Reboot \"false\"' in unattended_upgrades_aptconfig.stdout"
|
|
# The origin list must be non-empty, or u-u runs nightly and upgrades nothing — installed,
|
|
# scheduled, green, and inert.
|
|
- "'Debian-Security' in unattended_upgrades_aptconfig.stdout"
|
|
fail_msg: >-
|
|
EFFECTIVE apt CONFIG ON {{ inventory_hostname }} IS NOT WHAT THIS ROLE INTENDED.
|
|
Either the `^linux-` kernel blacklist or `Automatic-Reboot "false"` is not winning.
|
|
Both are safety properties, not preferences: without the blacklist an unattended process
|
|
can move a kernel these HP minis must keep >= 6.13 (PROD-PROVISIONING.md §0b), and
|
|
without Automatic-Reboot=false three control-plane nodes can each reboot themselves
|
|
independently and take etcd below quorum. Inspect with `apt-config dump | grep -i
|
|
unattended` and check for a file sorting after 52- in /etc/apt/apt.conf.d/.
|
|
success_msg: >-
|
|
Effective apt config confirmed: kernel blacklisted, automatic reboot disabled.
|
|
tags: [verify]
|
|
|
|
- name: Find the hooks that can create /var/run/reboot-required
|
|
# THE SENTINEL IS THE SINGLE POINT OF FAILURE FOR THE ENTIRE REBOOT HALF OF THIS DESIGN, and
|
|
# it is the one part that cannot be asserted from the repo — it depends on what
|
|
# update-notifier-common actually ships on this Debian release. Rather than hard-code a path
|
|
# that may not exist, discover the candidates and assert that at least one is present.
|
|
ansible.builtin.find:
|
|
paths:
|
|
- /etc/kernel/postinst.d
|
|
- /etc/apt/apt.conf.d
|
|
patterns:
|
|
- "*update-notifier*"
|
|
- "*unattended-upgrades*"
|
|
- "99update-notifier"
|
|
file_type: any
|
|
register: unattended_upgrades_sentinel_hooks
|
|
changed_when: false
|
|
tags: [verify]
|
|
|
|
- name: Assert something on this node can write the reboot sentinel
|
|
# If this fails, kured is a no-op: it watches /var/run/reboot-required forever, the file is
|
|
# never created, no reboot is ever taken, and NOTHING GOES RED -- the DaemonSet stays 1/1
|
|
# Running and `kured_reboot_required` reports a steady 0 on every node. That is the exact
|
|
# shape of failure (a check whose failure mode is a green tick) this repo has been burned by
|
|
# before, so it is an assert and not a debug message.
|
|
ansible.builtin.assert:
|
|
that:
|
|
- unattended_upgrades_sentinel_hooks.matched | int > 0
|
|
fail_msg: >-
|
|
NOTHING ON {{ inventory_hostname }} CAN CREATE /var/run/reboot-required. No
|
|
update-notifier / unattended-upgrades hook was found under /etc/kernel/postinst.d or
|
|
/etc/apt/apt.conf.d. kured watches that path and reboots only when it exists, so with no
|
|
writer the reboot automation is permanently and INVISIBLY idle -- kured stays 1/1 Running
|
|
and reports kured_reboot_required=0 on every node forever. Confirm
|
|
update-notifier-common is installed (`dpkg -l update-notifier-common`) and inspect
|
|
`dpkg -L update-notifier-common | grep -E 'postinst.d|apt.conf.d'`.
|
|
success_msg: >-
|
|
Reboot sentinel writer present ({{ unattended_upgrades_sentinel_hooks.matched }} hook(s)
|
|
found) -- a kernel or flagged package upgrade will signal kured.
|
|
tags: [verify]
|
|
|
|
- name: Report the pending-reboot state of this node
|
|
# Read-only. Tells the operator whether this node is currently waiting on kured for a reboot,
|
|
# which is the question `--tags verify` is usually being run to answer.
|
|
ansible.builtin.stat:
|
|
path: /var/run/reboot-required
|
|
register: unattended_upgrades_reboot_required
|
|
changed_when: false
|
|
tags: [verify]
|
|
|
|
- name: Summary
|
|
ansible.builtin.debug:
|
|
msg: >-
|
|
{{ inventory_hostname }}: security-only unattended upgrades ACTIVE
|
|
(upgrade timer {{ unattended_upgrades_upgrade_time }} +/-{{ unattended_upgrades_randomised_delay }}),
|
|
kernel and microcode blacklisted, automatic reboot DISABLED.
|
|
Reboot currently pending: {{ unattended_upgrades_reboot_required.stat.exists }}.
|
|
{% if unattended_upgrades_reboot_required.stat.exists %}
|
|
This node is waiting for kured to reboot it in the next 03:00-06:00 window; it will be
|
|
cordoned and drained first, and no other node will be rebooted at the same time.
|
|
{% endif %}
|
|
tags: [verify]
|