jakman85andClaude Opus 5
a2440a3125
Two halves of one repeatable upgrade process. Neither works alone. (a) OS upgrades and reboots platform/kured/ — Kubernetes Reboot Daemon, chart 6.1.0/app 1.23.0, in its own Flux Kustomization (dependsOn platform, wait:true). NOT in platform/kustomization.yaml: that tier gates platform-config and every apps wave, so a wedged reboot daemon there could stall every application. A component added to make maintenance safer must not be able to cause an outage. infra/ansible/roles/unattended_upgrades/ — new role (node_baseline is touched by other open branches, and this is a different lifecycle), enabled for prod in group_vars. The human gate is the SENTINEL, which is how this honours node_baseline's "IT DOES NOT REBOOT" (k3s-prod-2 hung on a warm reboot 2026-09-03) and cluster-upgrade-plan.md's rejection of automating away the gate between nodes: u-u is security-only with the kernel blacklisted, so it essentially never writes /var/run/reboot-required. That file appears when an operator deliberately runs node-baseline.yml --tags kernel. kured then does only the mechanical half, including the ordering rule node_baseline has to shout. Quorum: concurrent reboots are structurally impossible, not merely unlikely. concurrency:1 plus kured's single lock annotation (API-server optimistic concurrency), and lockTtl:0 — an expiring lock is exactly the 3-node etcd catastrophe, so a hung node holds it forever and nothing cascades. Longhorn: measured 25 volumes x 3 replicas, hard anti-affinity, 600s replenishment wait, serialized rebuild — so every drain degrades ALL volumes to 2/3 and the inter-node gate must be volume robustness, not pod readiness. Doubled: lockReleaseDelay 30m (unconditional) plus kured's Prometheus alert gate (semantic). alertFilterMatchOnly:true is load-bearing — the flag inverts the regexp's meaning, and under the chart default kured would have been blocked on day one by the five alerts measured firing with no end in sight, and never rebooted anything. Kernel floor (kernel >= 6.13, EFI memattr): excluded TWICE. The running kernel comes from o=Debian Backports, which is not an allowed origin, so u-u cannot see a kernel upgrade; and ^linux- is blacklisted anyway so widening origins later cannot re-arm it. Asserted against apt-config dump, not against the file just written. 6.12 kernels are still installed and bootable on all three nodes, and nothing was held. Closes a silent no-op: update-notifier-common was absent on all three nodes, so NOTHING could create the sentinel. kured alone would have idled forever at 1/1 Running. platform/kured/alerts.yaml — three rules over kured's only metric, with a companion promtool suite pinning each both ways, plus runbooks/node-reboot-automation.md. Window is DAILY 03:00-06:00 because a weekly one forces for:8d, longer than this cluster's 7d Prometheus retention. (b) Image/dependency automation Renovate was aspirational: renovate.json present and complete, with no workflow, no in-cluster workload, and its docker+kubernetes managers disabled — so every plain image: tag was maintained by nothing while seven files claimed otherwise. .gitea/workflows/renovate.yaml — scheduled daily, dry_run default true for a safe first run, and a preflight that FAILS (not skips) without RENOVATE_TOKEN, because a skipped Gitea job reports success and that is how this stayed inert. Renovate now owns both halves; Flux image-automation is formally closed: it pushes commits rather than opening PRs (bypassing CLAUDE.md §1), and it cannot work here anyway — main is protected user_can_push:false and the flux-system GitRepository has no secretRef. Its controllers are installed and have idled 43 days with zero CRs. conformance/: +34/-0, one new SCRAPE_OWNERS entry claiming kured_. Weakens nothing; tightens the check for this component (requires service.create+metrics.create, both chart-default false, plus a companion NetworkPolicy requirement). Required because assertion (3) fails an alert over an unclaimed metric and KNOWN_UNWIRED_METRICS is shrink-only. Sensitive paths touched: clusters/**, conformance/**, .gitea/** — needs a non-author approving review; /approve-override deliberately NOT posted. NOT verified: no OS upgrade run, no node rebooted, no lock observed, the Ansible role never executed, and whether CNPG switches its primary on a drain-induced eviction (the most likely way the first real reboot fails). Full list in the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
255 lines
14 KiB
Markdown
255 lines
14 KiB
Markdown
# Node reboot automation (kured)
|
||
|
||
**Alerts:** `KuredRebootPending`, `KuredNodeCoverageGap`, `KuredNotReporting`
|
||
**File:** `platform/kured/alerts.yaml`
|
||
**Expressions:** `max by (node) (kured_reboot_required) == 1` / `count(kured_reboot_required) < count(kube_node_info)` / `absent(kured_reboot_required)`
|
||
|
||
## What this means here
|
||
|
||
These three alerts cover the OS reboot half of the automated upgrade process. The design is two
|
||
pieces that only work together:
|
||
|
||
- **`roles/unattended_upgrades`** (Ansible, on the host) installs Debian **security** updates
|
||
nightly. It never upgrades the kernel, never restarts k3s, and never reboots. When something it
|
||
installed needs a reboot, it leaves `/var/run/reboot-required` behind.
|
||
- **kured** (`platform/kured`, a DaemonSet) watches that file. When it exists, kured takes a
|
||
**cluster-wide lock**, cordons the node, drains it, reboots it, and uncordons it — then holds the
|
||
lock for a further 30 minutes before any other node may go.
|
||
|
||
**kured does not decide whether to reboot.** On these nodes the sentinel is written almost
|
||
exclusively by a deliberate operator kernel upgrade
|
||
(`playbooks/node-baseline.yml --tags kernel`), because the kernel is blacklisted from unattended
|
||
upgrades. So a pending reboot normally means *a human already asked for this*.
|
||
|
||
Which alert you have tells you which half is wrong:
|
||
|
||
| alert | meaning |
|
||
|---|---|
|
||
| `KuredRebootPending` | The mechanism is alive and a reboot was offered, but it did not happen for >36h. **Something is blocking it.** |
|
||
| `KuredNodeCoverageGap` | kured is not running on every node. The uncovered node will never be rebooted, silently. |
|
||
| `KuredNotReporting` | kured is publishing nothing at all. Reboots are not happening, **and the other two alerts are inert.** Treat this as the most serious of the three. |
|
||
|
||
**All three nodes are control-plane + etcd, so quorum is 2 of 3.** One node down is degraded but
|
||
up; two is a lost cluster. Everything below is written to protect that. `concurrency: 1` plus
|
||
`lockTtl: 0` means a node that hangs on reboot keeps the lock **forever** and no second node is ever
|
||
rebooted behind it. That is deliberate: the cycle stops and pages a human rather than stepping over
|
||
a corpse.
|
||
|
||
**A reboot is not seamless.** Most workloads here are single-replica, so each reboot cycle takes a
|
||
brief outage of whatever runs on that node — including `coredns` (1 replica), which means a
|
||
cluster-wide DNS gap while it reschedules. See "Expected outage per cycle" below.
|
||
|
||
## Running the commands in this runbook
|
||
|
||
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
||
container. Every command below runs as:
|
||
|
||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
||
|
||
Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
|
||
picks a current-context per invocation, and that default has already changed once. Do not run two
|
||
`dev.sh` invocations at once; they share one container and interfere.
|
||
|
||
Mutating kubectl (`apply`, `patch`, `annotate`, `uncordon`, `rollout`) and `flux reconcile` are
|
||
operator actions, not agent actions. Where this runbook gives a mutating command it is for an
|
||
operator, and it says so.
|
||
|
||
## First commands
|
||
|
||
Start here, in order. The first question is always *which half is broken*.
|
||
|
||
# 1. Is kured running at all, and on how many nodes?
|
||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds,pod -o wide'
|
||
|
||
# 2. Which node is asking for a reboot, and who holds the lock?
|
||
# The lock is an annotation on kured's OWN DaemonSet. An empty/absent value means no holder.
|
||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds kured \
|
||
-o jsonpath="{.metadata.annotations}"; echo'
|
||
|
||
# 3. Is a node cordoned, and is a reboot in progress?
|
||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes \
|
||
-o custom-columns=NAME:.metadata.name,SCHEDULABLE:.spec.unschedulable,TAINTS:.spec.taints'
|
||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes \
|
||
-o jsonpath="{range .items[*]}{.metadata.name}{\" \"}{.metadata.annotations.weave\.works/kured-reboot-in-progress}{\"\n\"}{end}"'
|
||
|
||
# 4. What is kured itself saying? It logs its decision every period (1h).
|
||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured logs ds/kured --tail=80'
|
||
|
||
# 5. Is the gate blocking it? These are the ONLY alerts kured blocks on.
|
||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n longhorn-system \
|
||
get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness' \
|
||
| grep -v healthy
|
||
|
||
# 6. Does the node actually want a reboot? (read-only; SSH as k3s, no sudo needed)
|
||
ssh -o BatchMode=yes 192.168.2.10 'ls -la /var/run/reboot-required; cat /var/run/reboot-required.pkgs 2>/dev/null'
|
||
|
||
## Likely causes, most likely first
|
||
|
||
### 1. The Longhorn gate is holding it (the common `KuredRebootPending` cause)
|
||
|
||
kured queries Prometheus before taking the lock and refuses to proceed while
|
||
`LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `KubeNodeNotReady` or `NodeExporterDown` is
|
||
firing. This is working as designed and it is the **most likely** cause.
|
||
|
||
Why it lingers: there are 3 nodes, every volume has 3 replicas, and
|
||
`replica-soft-anti-affinity: false` means replicas may not share a node. So cordoning one node
|
||
leaves nowhere to put a replacement and **every volume goes degraded 2/3** for the duration.
|
||
Recovery is slow on purpose: `replica-replenishment-wait-interval` is 600s before a rebuild even
|
||
starts, and `concurrent-replica-rebuild-per-node-limit` is 1, so ~25 volumes rebuild one at a time.
|
||
|
||
Check progress, then wait:
|
||
|
||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n longhorn-system \
|
||
get volumes.longhorn.io -o json' | python3 -c 'import json,sys,collections; \
|
||
v=json.load(sys.stdin)["items"]; print(collections.Counter(x["status"]["robustness"] for x in v))'
|
||
|
||
If volumes are rebuilding, **do nothing**. If a volume is stuck `degraded` or `faulted` with no
|
||
rebuild progressing, that is a storage problem to fix first — see `/runbooks/longhorn-degraded`.
|
||
Rebooting another node while a volume is at 1/3 is how you lose data.
|
||
|
||
### 2. The drain timed out against a PodDisruptionBudget
|
||
|
||
Six PDBs currently allow **zero** voluntary disruptions, and two kinds matter:
|
||
|
||
- **`longhorn-system/instance-manager-*`** (one per node). `minAvailable: 1` against a selector
|
||
pinned to a single node's single pod, so `disruptionsAllowed` is structurally **always 0**.
|
||
Longhorn's `longhorn-manager` *deletes* the PDB when its `node-drain-policy`
|
||
(`block-if-contains-last-replica`) judges the drain safe. The drain is **supposed** to wait here.
|
||
- **CNPG `<cluster>-primary`** (`keycloak-db`, `nextcloud-db`, `semaphore-db`). Also structurally 0
|
||
until the CNPG operator switches the primary to another instance. If all three primaries are on
|
||
the node being drained, three switchovers must happen.
|
||
|
||
kured's `drainTimeout` is 15m: a genuinely stuck drain **aborts**, releases the lock, and is retried
|
||
next period rather than wedging forever. Repeated timeouts show in kured's logs.
|
||
|
||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pdb -A -o wide' | awk '$NF==0 || NR==1'
|
||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
|
||
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PRIMARY:.status.currentPrimary'
|
||
|
||
If CNPG will not switch over, an **operator** can move the primary deliberately before the next
|
||
window rather than fighting the drain.
|
||
|
||
### 3. A node hung on reboot and still holds the lock
|
||
|
||
`k3s-prod-2` has done this before (2026-09-03: powered, off the LAN ~38 minutes, physical power
|
||
button required). `lockTtl: 0` means the lock does **not** expire, so the cycle stops here —
|
||
correctly, since the cluster is at 2/3 and must not go to 1/3.
|
||
|
||
Symptoms: one node `NotReady` or absent, `KuredNodeCoverageGap` firing alongside
|
||
`KuredRebootPending`, and the lock annotation naming the missing node.
|
||
|
||
**Operator recovery:** attend to the node physically. Once it is back and `Ready`, kured clears its
|
||
own lock on the next period. If the node is gone for good and the lock must be cleared by hand,
|
||
that is an operator action:
|
||
|
||
# OPERATOR ONLY. Only when you have confirmed the named node is NOT mid-reboot.
|
||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured annotate ds kured \
|
||
weave.works/kured-node-lock- --overwrite'
|
||
|
||
If the node was pruned mid-reboot and is left **cordoned** with no daemon to uncordon it:
|
||
|
||
# OPERATOR ONLY.
|
||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod uncordon <node>'
|
||
|
||
### 4. Nothing can write the sentinel (so nothing ever reboots)
|
||
|
||
If `KuredRebootPending` never fires and reboots never happen even after a kernel upgrade, suspect
|
||
the sentinel writer. On Debian, `/var/run/reboot-required` is created by a hook shipped in
|
||
**`update-notifier-common`** — and it was absent on all three nodes before this automation landed.
|
||
Without it kured is permanently, invisibly idle.
|
||
|
||
ssh -o BatchMode=yes 192.168.2.10 'dpkg -l update-notifier-common needrestart unattended-upgrades | tail -4'
|
||
ssh -o BatchMode=yes 192.168.2.10 'ls /etc/kernel/postinst.d/ /etc/apt/apt.conf.d/ | sort'
|
||
|
||
Fix by re-running the role, which asserts this explicitly:
|
||
|
||
cd infra/ansible && ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify
|
||
|
||
### 5. kured is not being scraped (`KuredNotReporting`)
|
||
|
||
The chart defaults `service.create` and `metrics.create` to **false**, and the `kured` namespace
|
||
ships a default-deny NetworkPolicy. If either enablement regressed, or the Prometheus ingress allow
|
||
was dropped, targets are *discovered* and every scrape times out — the TSDB stays empty and all
|
||
three rules go quiet. Longhorn hit exactly this failure on this cluster.
|
||
|
||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get svc,servicemonitor,networkpolicy,ciliumnetworkpolicy'
|
||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n monitoring exec \
|
||
sts/prometheus-kube-prometheus-stack-prometheus -c prometheus -- \
|
||
wget -qO- "http://localhost:9090/api/v1/targets?state=active" ' | grep -o '"job":"kured[^,]*' | head
|
||
|
||
### 6. A node is uncovered by the DaemonSet (`KuredNodeCoverageGap`)
|
||
|
||
kured declares **no tolerations**, because no prod node is tainted. A newly provisioned or
|
||
re-provisioned node carrying a taint gets no kured pod and will never be rebooted.
|
||
|
||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes \
|
||
-o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints'
|
||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds kured \
|
||
-o custom-columns=DESIRED:.status.desiredNumberScheduled,READY:.status.numberReady'
|
||
|
||
The fix is a PR adding the needed toleration to `platform/kured/helmrelease.yaml` — not a `kubectl
|
||
patch`, which Flux reverts.
|
||
|
||
## Expected outage per reboot cycle
|
||
|
||
Stated plainly, because this process is **not** seamless. Every workload at 1 replica on the
|
||
draining node is down until it reschedules and becomes Ready elsewhere. Measured 2026-10-17: **45
|
||
Deployments and 3 StatefulSets run a single replica.** The ones worth knowing about:
|
||
|
||
- **`kube-system/coredns` — 1 replica.** A cluster-wide in-cluster DNS gap while it moves. This is
|
||
the single most impactful item and it is **not** fixed by this PR; raising coredns to 2–3 replicas
|
||
is recommended follow-up work.
|
||
- **`monitoring/prometheus-…-0`, `monitoring/loki-0`, `monitoring/alertmanager-…-0`** — all
|
||
single-replica StatefulSets on Longhorn RWO volumes, so they additionally wait for volume
|
||
detach/reattach. **You lose monitoring and alerting for part of the cycle**, including these very
|
||
alerts.
|
||
- **All three CNPG Postgres primaries** blip during switchover, which means Keycloak, Nextcloud and
|
||
Semaphore briefly lose their database.
|
||
- **`gitea`, `nextcloud`, `plex`, `jackett`, `qbittorrent`, `radarr`, `sonarr`, `seerr`, `crafty`,
|
||
`homepage`, `headlamp`, `semaphore`** — each down for its reschedule.
|
||
- **`unifi` (a KubeVirt VM)** has `evictionStrategy: LiveMigrate` but
|
||
`migrationMethod: BlockMigration`, so evicting it copies the whole root disk over the network.
|
||
Expect the drain of its node to be slow.
|
||
- **`kube-vip`** fails the control-plane VIP over on every node reboot.
|
||
|
||
Multi-replica and therefore fine: `cilium-operator`, `oauth2-proxy`, `kyverno-admission-controller`,
|
||
the Longhorn CSI sidecars, `keycloak`, `pihole`, `runbooks`.
|
||
|
||
## When to escalate or stop
|
||
|
||
**STOP and do not intervene** if:
|
||
|
||
- Longhorn volumes are `degraded` **and rebuilding**. This is the gate doing its job. Wait.
|
||
- A single node is mid-reboot and under 15 minutes have passed.
|
||
|
||
**STOP and get a human to the hardware** if:
|
||
|
||
- Any node is `NotReady` while another reboot is pending. **Never clear the kured lock in this
|
||
state** — that is the one action that can take etcd from 2/3 to 1/3 and lose the cluster.
|
||
- Two nodes are down. The cluster has no quorum; this runbook cannot help. Recover hardware first.
|
||
|
||
**Escalate to a deliberate maintenance decision** if:
|
||
|
||
- `KuredRebootPending` has been firing for over a week. The reboot is not going to happen on its
|
||
own; find the blocker above, or take the reboot manually during a watched window.
|
||
- A volume is `faulted`. Fix storage before any further reboot.
|
||
|
||
**Turn the automation off** if it is misbehaving and you need the cluster stable. The honest way is
|
||
a PR, but for an emergency an operator may suspend the tier — this stops future reboots and leaves
|
||
running nodes untouched:
|
||
|
||
# OPERATOR ONLY. Stops kured being reconciled; does NOT uncordon a node mid-cycle.
|
||
infra/devcontainer/dev.sh run bash -c 'flux --context k3s-prod suspend kustomization kured'
|
||
|
||
Check afterwards whether a node was left cordoned (First commands, step 3) and uncordon it if so.
|
||
Resume with `flux resume kustomization kured`. Nothing in the graph depends on `kured`, so
|
||
suspending it stalls nothing else.
|
||
|
||
---
|
||
|
||
Source of truth: `runbooks/node-reboot-automation.md`. Index: `/runbooks/`
|
||
|
||
Records: `decisions/2026-10-17-automated-upgrades.md` (design + what was not verified),
|
||
`platform/kured/helmrelease.yaml` (every tuned value and why),
|
||
`infra/ansible/roles/unattended_upgrades/README.md` (the package half, and the kernel exclusion).
|