jakman85andClaude Opus 5
a2440a3125
Two halves of one repeatable upgrade process. Neither works alone. (a) OS upgrades and reboots platform/kured/ — Kubernetes Reboot Daemon, chart 6.1.0/app 1.23.0, in its own Flux Kustomization (dependsOn platform, wait:true). NOT in platform/kustomization.yaml: that tier gates platform-config and every apps wave, so a wedged reboot daemon there could stall every application. A component added to make maintenance safer must not be able to cause an outage. infra/ansible/roles/unattended_upgrades/ — new role (node_baseline is touched by other open branches, and this is a different lifecycle), enabled for prod in group_vars. The human gate is the SENTINEL, which is how this honours node_baseline's "IT DOES NOT REBOOT" (k3s-prod-2 hung on a warm reboot 2026-09-03) and cluster-upgrade-plan.md's rejection of automating away the gate between nodes: u-u is security-only with the kernel blacklisted, so it essentially never writes /var/run/reboot-required. That file appears when an operator deliberately runs node-baseline.yml --tags kernel. kured then does only the mechanical half, including the ordering rule node_baseline has to shout. Quorum: concurrent reboots are structurally impossible, not merely unlikely. concurrency:1 plus kured's single lock annotation (API-server optimistic concurrency), and lockTtl:0 — an expiring lock is exactly the 3-node etcd catastrophe, so a hung node holds it forever and nothing cascades. Longhorn: measured 25 volumes x 3 replicas, hard anti-affinity, 600s replenishment wait, serialized rebuild — so every drain degrades ALL volumes to 2/3 and the inter-node gate must be volume robustness, not pod readiness. Doubled: lockReleaseDelay 30m (unconditional) plus kured's Prometheus alert gate (semantic). alertFilterMatchOnly:true is load-bearing — the flag inverts the regexp's meaning, and under the chart default kured would have been blocked on day one by the five alerts measured firing with no end in sight, and never rebooted anything. Kernel floor (kernel >= 6.13, EFI memattr): excluded TWICE. The running kernel comes from o=Debian Backports, which is not an allowed origin, so u-u cannot see a kernel upgrade; and ^linux- is blacklisted anyway so widening origins later cannot re-arm it. Asserted against apt-config dump, not against the file just written. 6.12 kernels are still installed and bootable on all three nodes, and nothing was held. Closes a silent no-op: update-notifier-common was absent on all three nodes, so NOTHING could create the sentinel. kured alone would have idled forever at 1/1 Running. platform/kured/alerts.yaml — three rules over kured's only metric, with a companion promtool suite pinning each both ways, plus runbooks/node-reboot-automation.md. Window is DAILY 03:00-06:00 because a weekly one forces for:8d, longer than this cluster's 7d Prometheus retention. (b) Image/dependency automation Renovate was aspirational: renovate.json present and complete, with no workflow, no in-cluster workload, and its docker+kubernetes managers disabled — so every plain image: tag was maintained by nothing while seven files claimed otherwise. .gitea/workflows/renovate.yaml — scheduled daily, dry_run default true for a safe first run, and a preflight that FAILS (not skips) without RENOVATE_TOKEN, because a skipped Gitea job reports success and that is how this stayed inert. Renovate now owns both halves; Flux image-automation is formally closed: it pushes commits rather than opening PRs (bypassing CLAUDE.md §1), and it cannot work here anyway — main is protected user_can_push:false and the flux-system GitRepository has no secretRef. Its controllers are installed and have idled 43 days with zero CRs. conformance/: +34/-0, one new SCRAPE_OWNERS entry claiming kured_. Weakens nothing; tightens the check for this component (requires service.create+metrics.create, both chart-default false, plus a companion NetworkPolicy requirement). Required because assertion (3) fails an alert over an unclaimed metric and KNOWN_UNWIRED_METRICS is shrink-only. Sensitive paths touched: clusters/**, conformance/**, .gitea/** — needs a non-author approving review; /approve-override deliberately NOT posted. NOT verified: no OS upgrade run, no node rebooted, no lock observed, the Ansible role never executed, and whether CNPG switches its primary on a drain-induced eviction (the most likely way the first real reboot fails). Full list in the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
37 lines
2.3 KiB
YAML
37 lines
2.3 KiB
YAML
---
|
|
# Automatic Debian SECURITY patching for the k3s nodes — the package half of the automated upgrade
|
|
# process. The reboot half is kured (platform/kured), reconciled by Flux; the two are designed
|
|
# together and neither is useful alone. Full rationale for every decision:
|
|
# roles/unattended_upgrades/defaults/main.yml and that role's README.md.
|
|
#
|
|
# TARGET (R-ENV-02): `k3s_hosts` defaults to dev's `k3s_servers`; prod is
|
|
# ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod
|
|
# Same pattern node-baseline.yml and node-health.yml established — `hosts:` is templated before
|
|
# group_vars exist, so the target is an extra var, never a group_vars value.
|
|
#
|
|
# The role is a NO-OP unless `unattended_upgrades_enabled` is true, which is set for prod only in
|
|
# inventory/group_vars/k3s_prod/vars.yml. Running this against dev changes nothing.
|
|
#
|
|
# DELIBERATELY NOT IN site.yml. site.yml is the bring-up chain (node-baseline -> k3s ->
|
|
# etcd-io-priority -> kube-vip -> cilium -> verify -> flux-bootstrap) and is "not run against prod
|
|
# casually". Automatic patching is not a prerequisite for anything in that chain and nothing in the
|
|
# chain depends on it, so adding it there would only widen what a cluster rebuild touches. It is
|
|
# applied on its own, once, and re-applied when its configuration changes.
|
|
#
|
|
# SAFE TO RUN ON A LIVE CLUSTER, AT ANY TIME. This playbook installs and configures the upgrader. It
|
|
# never runs `apt upgrade`, never restarts k3s, and never reboots — the role asserts the last of
|
|
# those against apt's own effective config rather than trusting the file it just wrote. The first
|
|
# unattended upgrade happens at the next timer firing (04:00 +/-60m), and any reboot that upgrade
|
|
# implies is kured's to schedule, one node at a time, inside 03:00-06:00.
|
|
#
|
|
# TAGS. `--tags verify` re-runs only the read-only assertions and the per-node summary, which is the
|
|
# way to answer "is automatic patching actually working on these boxes, and is anything waiting on a
|
|
# reboot" without rewriting any configuration:
|
|
# ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify
|
|
- name: Automatic security upgrades on the k3s nodes (kernel excluded; reboots delegated to kured)
|
|
hosts: "{{ k3s_hosts | default('k3s_servers') }}"
|
|
become: true
|
|
gather_facts: true
|
|
roles:
|
|
- unattended_upgrades
|