Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
fda5c31a94 | ||
|
|
c340bbfc4b | ||
|
|
a9a529750d | ||
|
|
ef8ef5ac29 |
No files matched your search
@@ -73,7 +73,7 @@ spec:
|
||||
- name: FTLCONF_dns_rateLimit_count
|
||||
value: '0'
|
||||
- name: FTLCONF_misc_dnsmasq_lines
|
||||
value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
|
||||
value: local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
|
||||
- name: FTLCONF_webserver_api_password
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
|
||||
+32
-1
@@ -143,8 +143,39 @@ env:
|
||||
# resolved to the Gateway VIP and a connect to :32400 there returned http=000. That is a silent
|
||||
# failure: the Plex web UI on plex.${BASE_DOMAIN} keeps working through the Gateway on 443 while
|
||||
# every client that takes the ADVERTISED connection cannot reach the server at all.
|
||||
#
|
||||
# ── local-ttl=60 — THE SPLIT-HORIZON ANSWER MUST BE CACHEABLE ────────────────────────────────
|
||||
# MEASURED 2026-10-18, diagnosing "DNS flapping between the cluster ingress and the public IP":
|
||||
#
|
||||
# dig home.keithjackson.dev @192.168.2.23 (pihole) -> 192.168.2.20 TTL=0
|
||||
# dig home.keithjackson.dev @1.1.1.1 (public) -> wan.home.keithjackson.dev TTL=300
|
||||
# dig home.keithjackson.dev @192.168.2.1 (router) -> wan.... TTL=153 -> 148 -> 143
|
||||
#
|
||||
# dnsmasq answers `address=` entries with **TTL 0** unless told otherwise, and that is the whole
|
||||
# bug. The SOHO router forwards LAN queries to PiHole and CACHES the result, so:
|
||||
# * the CORRECT answer is UNCACHEABLE (TTL 0) -- the router must re-query PiHole on every
|
||||
# single lookup, which is maximum exposure to any momentary gap; while
|
||||
# * the WRONG answer is CACHEABLE FOR 5 MINUTES -- the instant the router falls back to its
|
||||
# upstream (standard forwarder behaviour when the forward target does not answer) it latches
|
||||
# the public WAN address and serves it to EVERY LAN CLIENT for the public record's 300s TTL.
|
||||
# The decrementing TTL on the router's reply above is the proof it caches rather than forwarding
|
||||
# live. So ONE sub-second PiHole gap -- a pod restart, an L2 announcement lease moving, an ARP
|
||||
# flap, a kured node reboot -- becomes FIVE MINUTES of the whole house resolving to the WAN IP.
|
||||
# That asymmetry, not the gap, is why this is visible as "flapping" rather than a blip.
|
||||
#
|
||||
# 60s is a deliberate floor, not a tuning knob: long enough to ride out a pod restart and a lease
|
||||
# re-election (both seconds), short enough that a genuine VIP change still propagates inside a
|
||||
# minute. These VIPs are pinned by name (`lbipam.cilium.io/ips`) and effectively static, so the
|
||||
# propagation cost is theoretical while the flap it prevents is measured. If flaps persist across
|
||||
# LONGER outages, raising this toward the public record's 300 would cover them -- at the cost of
|
||||
# the same delay on a real change. It does NOT help when PiHole is down longer than the TTL: the
|
||||
# cache expires, the router falls back, and the public answer latches again. That is a real
|
||||
# outage, and the fix for it is PiHole availability, not DNS caching.
|
||||
#
|
||||
# Applies to every `address=` reply below (dnsmasq: local-ttl governs replies made from
|
||||
# configuration, /etc/hosts and DHCP leases), so one directive covers all five names.
|
||||
- name: FTLCONF_misc_dnsmasq_lines
|
||||
value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
|
||||
value: "local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
|
||||
# THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
|
||||
# apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
|
||||
# carries the full rationale and the measurement that found it.
|
||||
|
||||
@@ -185,6 +185,55 @@ Measured against the live cluster and nodes, read-only, on 2026-10-17.
|
||||
- **`renovate.json` parses** as valid JSON after my edits (`json.load`).
|
||||
- **kured is not installed** and the string `kured` appears nowhere in the repo — greenfield.
|
||||
|
||||
### CORRECTION 2026-10-18 — the first real run FAILED, and the sentinel design was wrong
|
||||
|
||||
`ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod` failed on all three
|
||||
nodes at the package task:
|
||||
|
||||
```
|
||||
fatal: [k3s-prod-1]: FAILED! => "No package matching 'update-notifier-common' is available"
|
||||
```
|
||||
|
||||
**`update-notifier-common` is an UBUNTU package and does not exist in Debian.** Measured on trixie:
|
||||
`apt-cache policy update-notifier-common` → `Candidate: (none)`. The original entry asserted this
|
||||
package "owns the hook that creates /var/run/reboot-required" — true on Ubuntu, false here, and the
|
||||
role could never have worked. This was the exact risk the first version of this entry flagged ("the
|
||||
`find` patterns are a guess at which package ships what") landing in a worse place than predicted:
|
||||
not a wrong glob, a nonexistent package.
|
||||
|
||||
Three further measurements settled what to do instead:
|
||||
|
||||
* `/usr/share/update-notifier/notify-reboot-required` is **absent**.
|
||||
* **No `linux-image-*` postinst references `notify-reboot-required`** — invoking it is an Ubuntu
|
||||
patch, so merely placing a script at that path would never be called.
|
||||
* No hook under `/etc/apt/apt.conf.d` or `/etc/kernel/` mentions `reboot-required`.
|
||||
|
||||
So Debian trixie has **no stock mechanism at all**, and the role now ships the writer itself:
|
||||
`/usr/local/sbin/k3s-notify-reboot-required` (compares the running kernel against the newest
|
||||
installed via `linux-version`, from `linux-base`, which every `linux-image` depends on — measured
|
||||
`linux-base 4.12.1` present) plus a `DPkg::Post-Invoke` hook at
|
||||
`/etc/apt/apt.conf.d/99-k3s-reboot-required`. `/run` is a tmpfs (measured), so the sentinel
|
||||
self-clears on boot and needs no cleanup task.
|
||||
|
||||
The verify assert changed shape too: it was a `find` glob over `/etc/kernel/postinst.d` and
|
||||
`/etc/apt/apt.conf.d` for `*update-notifier*` / `*unattended-upgrades*`. That glob was **weaker
|
||||
than it looked** — it would have matched Debian's own `unattended-upgrades` apt conffile and
|
||||
reported "sentinel writer present" while no writer existed. It now stats the two exact paths this
|
||||
role owns and requires the script to be executable.
|
||||
|
||||
**Newly verified (this correction):** the sentinel script was executed on prod-1 as the
|
||||
unprivileged `k3s` user under `sh -x`. It resolved `running=7.1.8+deb13-amd64`,
|
||||
`newest=7.1.8+deb13-amd64`, took the equal branch, exited 0 and wrote nothing — correct, since the
|
||||
running kernel *is* the newest. `sh -n` and `bash -n` clean.
|
||||
|
||||
**Still NOT verified after this correction:** the *writing* half. Because running == newest on all
|
||||
three nodes, the `touch` path has never executed, so I have not seen the sentinel actually created,
|
||||
nor seen kured react to it. The honest test is the operator's next kernel upgrade. Also unverified:
|
||||
whether a `DPkg::Post-Invoke` hook fires in the `unattended-upgrades` code path specifically (it
|
||||
runs dpkg, so it should, but that is inference) — and whether `linux-version sort --reverse` orders
|
||||
a mixed backports/stable kernel set the way I assume when a 6.12 and a 7.1 are both installed. It
|
||||
produced the right answer with the current set; a future set could differ.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
The honest list. Most of this needs an actual upgrade cycle, which I did not and could not run.
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
---
|
||||
date: 2026-10-18
|
||||
branch: fix/pihole-local-ttl
|
||||
title: "The split-horizon answer was uncacheable while the wrong answer cached for 5 minutes"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
The operator reported DNS "flapping between the cluster ingress and the router (public IP)".
|
||||
PiHole was not at fault — both replicas answer correctly and consistently. The bug is an
|
||||
**asymmetry in cacheability** that we created.
|
||||
|
||||
Measured on prod 2026-10-18:
|
||||
|
||||
```
|
||||
dig home.keithjackson.dev @192.168.2.23 (pihole) -> 192.168.2.20 TTL=0
|
||||
dig home.keithjackson.dev @1.1.1.1 (public) -> wan.home.keithjackson.dev TTL=300
|
||||
dig home.keithjackson.dev @192.168.2.1 (router) -> wan.... TTL=153 -> 148 -> 143
|
||||
```
|
||||
|
||||
dnsmasq answers `address=` entries with **TTL 0** unless told otherwise. The SOHO router forwards
|
||||
LAN queries to PiHole and **caches** the result — the decrementing TTL above is the proof it caches
|
||||
rather than forwarding live. So:
|
||||
|
||||
* the **correct** answer is **uncacheable** (TTL 0): the router must re-query PiHole on *every*
|
||||
lookup, which is maximum exposure to any momentary gap;
|
||||
* the **wrong** answer is **cacheable for 300s**: the instant the router falls back to its upstream
|
||||
(standard forwarder behaviour when the forward target does not answer) it latches the public WAN
|
||||
address and serves it to **every LAN client** for five minutes.
|
||||
|
||||
So **one sub-second PiHole gap becomes five minutes of the whole house resolving to the WAN IP** —
|
||||
then it expires, the router re-queries PiHole, gets TTL 0 again, and is immediately re-exposed.
|
||||
That asymmetry, not the gap itself, is why this is visible as flapping rather than a blip.
|
||||
|
||||
Fix: `local-ttl=60` prepended to `FTLCONF_misc_dnsmasq_lines` in `catalog/pihole.yaml`, so the
|
||||
private answer is cacheable and a short gap is invisible to clients. `apps/wave-0/pihole.yaml`
|
||||
regenerated with the canonical render command.
|
||||
|
||||
## What I verified
|
||||
|
||||
* **PiHole is NOT the fault.** Both replicas, queried directly by pod IP, answer `192.168.2.20`;
|
||||
the VIP returned it 6/6; this workstation 8/8. DHCP hands out only `192.168.2.23`.
|
||||
* **The router caches rather than forwards live** — three successive queries returned TTL 153, 148,
|
||||
143 (decrementing), which a pure forwarder could not do.
|
||||
* **The TTL asymmetry**, as quoted above: 0 from PiHole, 300 from public DNS.
|
||||
* **The router does answer DNS and returns the public chain** (`wan.home.keithjackson.dev` → the WAN
|
||||
address), so a fallback always has a wrong answer available.
|
||||
* **`local-ttl` is the right directive**: dnsmasq applies it to replies made from configuration,
|
||||
`/etc/hosts` and DHCP leases, which covers all five `address=` names with one directive. It is a
|
||||
standalone option, not a sub-field of `address=`, so prepending it to the `;`-separated
|
||||
`FTLCONF_misc_dnsmasq_lines` list is syntactically correct.
|
||||
* **The render is clean and scoped**: `catalog/render.py --all catalog -o apps --wave-layout
|
||||
--no-image-automation` (the canonical command `ci/render-golden.sh` asserts against, and
|
||||
deliberately **not** `--emit-waves`) changed exactly two files — `catalog/pihole.yaml` and
|
||||
`apps/wave-0/pihole.yaml` — and the rendered diff is the single `local-ttl=60;` prefix.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
* **UNAPPLIED.** Nothing has been reconciled, so I have not observed PiHole serving a non-zero TTL,
|
||||
nor the router caching the private answer, nor the flap stopping. **Verify after merge with
|
||||
`dig home.keithjackson.dev @192.168.2.23` (expect `TTL=60`, not 0) and then the same query at the
|
||||
router (expect `192.168.2.20` with a decrementing TTL, not `wan.…`).**
|
||||
* **I did not confirm that PiHole/FTL accepts `local-ttl` via `FTLCONF_misc_dnsmasq_lines`.** The
|
||||
directive is valid dnsmasq and the field is a passthrough, but `catalog/pihole.yaml`'s own header
|
||||
records that FTL has previously written a bogus line into `dnsmasq.conf` and reported "0 invalid"
|
||||
— so a malformed passthrough fails *silently*. If the TTL is still 0 after merge, that is the
|
||||
cause, not this reasoning. Check the pod's `/etc/pihole/dnsmasq.conf`.
|
||||
* **60s is chosen, not derived.** Long enough for a pod restart and a lease re-election (both
|
||||
seconds); it does **nothing** for an outage longer than the TTL, where the cache expires and the
|
||||
public answer latches again. That is a real availability problem, not a caching one.
|
||||
* **This does not remove the trigger.** `192.168.2.23` is served by **two** LoadBalancer Services
|
||||
(`pihole-dns-tcp`, `pihole-dns-udp`) with **two independently-elected L2 announcement leases**, so
|
||||
two nodes can ARP for the same address at once — they sat on different nodes earlier the same day
|
||||
(tcp on prod-2, udp on prod-1). The same pattern applies to crafty (`.22`) and unifi (`.21`).
|
||||
Merging each pair into one mixed-protocol Service would remove the class; the operator has parked
|
||||
that decision, and it is deliberately **not** in this change.
|
||||
* **Client-side bypasses are unexamined.** A device with its own resolver, or a browser with
|
||||
DNS-over-HTTPS, bypasses PiHole entirely and always gets the public answer; this fix cannot help
|
||||
there. Only this one workstation's resolver chain was inspected.
|
||||
* **No alert fires if the split-horizon answer regresses to the public address.** Nothing probes
|
||||
`home.keithjackson.dev` from a LAN vantage point and compares it against `${GATEWAY_VIP}`, so the
|
||||
next recurrence would again be found by a human noticing. That gap is not closed here.
|
||||
@@ -20,10 +20,19 @@ restarted, and reboots are delegated to **kured** (`platform/kured`).
|
||||
|
||||
Deploying either alone produces a mechanism that looks healthy and does nothing:
|
||||
|
||||
- **kured alone** watches a file that nothing creates. Measured 2026-10-17:
|
||||
`update-notifier-common` — the package that owns the hook which writes
|
||||
`/var/run/reboot-required` — was **not installed on any node**. kured would sit at 1/1 Running
|
||||
reporting `kured_reboot_required=0` forever.
|
||||
- **kured alone** watches a file that **nothing on Debian ever creates**. Measured 2026-10-18,
|
||||
after a first attempt to install Ubuntu's `update-notifier-common` failed on all three nodes
|
||||
with *"No package matching 'update-notifier-common' is available"*:
|
||||
- `apt-cache policy update-notifier-common` → `Candidate: (none)` — the package does not exist
|
||||
in Debian; it is Ubuntu-only.
|
||||
- `/usr/share/update-notifier/notify-reboot-required` → absent.
|
||||
- **No `linux-image-*` postinst references `notify-reboot-required`** — calling it is an Ubuntu
|
||||
patch, so even dropping a script at that path would never be invoked.
|
||||
- No hook under `/etc/apt/apt.conf.d` or `/etc/kernel/` mentions `reboot-required`.
|
||||
|
||||
So **this role ships the writer itself** (`/usr/local/sbin/k3s-notify-reboot-required` plus a
|
||||
`DPkg::Post-Invoke` hook) rather than installing a package. Without it kured would sit at 1/1
|
||||
Running reporting `kured_reboot_required=0` forever.
|
||||
- **this role alone** installs patches that a long-running process never picks up, and when a
|
||||
reboot *is* needed there is nothing safe to take it. `Automatic-Reboot "true"` is not the
|
||||
answer: three control-plane nodes with etcd quorum 2/3, each rebooting at its own time, is how
|
||||
@@ -36,7 +45,7 @@ Measured on all three prod nodes, 2026-10-17:
|
||||
| | |
|
||||
|---|---|
|
||||
| `unattended-upgrades` | **not installed** |
|
||||
| `update-notifier-common` | **not installed** |
|
||||
| `update-notifier-common` | **not installable** — Ubuntu-only, `Candidate: (none)` on trixie |
|
||||
| `needrestart` | **not installed** |
|
||||
| `apt-mark showhold` | **empty** — no package held on any node |
|
||||
| `/etc/apt/preferences.d/` | **empty** — no apt pin declares the backports-kernel intent |
|
||||
@@ -157,7 +166,9 @@ first two):
|
||||
```bash
|
||||
ssh [email protected] 'apt-config dump | grep -iE "unattended-upgrade::(automatic-reboot|package-blacklist|origins)"'
|
||||
ssh [email protected] 'systemctl list-timers apt-daily-upgrade.timer --no-pager'
|
||||
ssh [email protected] 'ls -la /var/run/reboot-required 2>&1; dpkg -l update-notifier-common needrestart unattended-upgrades | tail -4'
|
||||
ssh [email protected] 'ls -la /var/run/reboot-required* 2>&1 | head -2
|
||||
ls -l /usr/local/sbin/k3s-notify-reboot-required /etc/apt/apt.conf.d/99-k3s-reboot-required 2>&1
|
||||
dpkg -l needrestart unattended-upgrades | tail -3'
|
||||
```
|
||||
|
||||
Expect: `Automatic-Reboot "false"`, a blacklist containing `^linux-`, an `Origins-Pattern`
|
||||
|
||||
@@ -92,13 +92,11 @@ unattended_upgrades_enabled: false
|
||||
unattended_upgrades_packages:
|
||||
# The upgrader itself.
|
||||
- unattended-upgrades
|
||||
# THE SENTINEL WRITER, AND THE WHOLE REASON kured CAN WORK. On Debian, /var/run/reboot-required is
|
||||
# not created by the kernel, by apt, or by unattended-upgrades — it is created by a hook shipped in
|
||||
# update-notifier-common. Measured ABSENT on all three nodes. Without this package the file is
|
||||
# never written, kured correctly never reboots, every dashboard stays green, and the reboot half of
|
||||
# this PR is an elaborate no-op. The role asserts the hook exists after install rather than
|
||||
# assuming the package shipped one (see tasks/main.yml).
|
||||
- update-notifier-common
|
||||
# NOTE: `update-notifier-common` was listed here and REMOVED (2026-10-18). It is an UBUNTU
|
||||
# package and does not exist in Debian at all -- `apt-cache policy update-notifier-common` on
|
||||
# trixie returns `Candidate: (none)`, so the play failed outright with "No package matching
|
||||
# 'update-notifier-common' is available" on all three nodes. The sentinel writer is now provided
|
||||
# by this role itself; see unattended_upgrades_sentinel_* below and tasks/main.yml.
|
||||
# Restarts services still mapped to a deleted library after a security upgrade. Without it a
|
||||
# security-only policy patches bytes on disk that no running process reads — see DECISION 3.
|
||||
- needrestart
|
||||
@@ -170,3 +168,32 @@ unattended_upgrades_needrestart_blacklist:
|
||||
unattended_upgrades_download_time: "03:30"
|
||||
unattended_upgrades_upgrade_time: "04:00"
|
||||
unattended_upgrades_randomised_delay: 60m
|
||||
|
||||
|
||||
# ── THE REBOOT SENTINEL, PROVIDED BY THIS ROLE ──────────────────────────────────────────────────
|
||||
# /var/run/reboot-required is the ONLY thing kured watches, and on Debian trixie **nothing creates
|
||||
# it**. Measured on all three nodes 2026-10-18:
|
||||
# * `update-notifier-common` (which owns that hook on Ubuntu) has `Candidate: (none)` — the
|
||||
# package does not exist in Debian. Installing it is what broke the first run of this play.
|
||||
# * `/usr/share/update-notifier/notify-reboot-required` is absent.
|
||||
# * NO `linux-image-*` postinst references `notify-reboot-required` — calling it is an Ubuntu
|
||||
# patch, not Debian behaviour, so merely dropping a script at that path would never be invoked.
|
||||
# * No hook under /etc/apt/apt.conf.d or /etc/kernel/ mentions reboot-required.
|
||||
# So the writer has to be ours. Without it kured is a permanent, INVISIBLE no-op: it stays 1/1
|
||||
# Running and reports kured_reboot_required=0 forever, which is the "failure mode is a green tick"
|
||||
# shape this repo keeps getting burned by.
|
||||
#
|
||||
# MECHANISM: a dpkg Post-Invoke hook runs a script that compares the RUNNING kernel against the
|
||||
# newest INSTALLED one and touches the sentinel when they differ. Deliberately kernel-only, which
|
||||
# matches the design: unattended-upgrades is security-only with `^linux-` blacklisted, so it will
|
||||
# essentially never set this — the sentinel appears when the OPERATOR deliberately runs the
|
||||
# node_baseline kernel play, and kured then automates only the mechanical reboot.
|
||||
#
|
||||
# It uses `linux-version` (from `linux-base`, a dependency of every linux-image package, so
|
||||
# guaranteed present — measured `linux-base 4.12.1` installed) rather than parsing `dpkg -l`
|
||||
# output, because kernel version ordering is not string ordering.
|
||||
#
|
||||
# /run is a tmpfs (measured), so the sentinel self-clears on boot: after a reboot the running
|
||||
# kernel IS the newest and the script writes nothing. No cleanup task is needed, and none exists.
|
||||
unattended_upgrades_sentinel_script: /usr/local/sbin/k3s-notify-reboot-required
|
||||
unattended_upgrades_sentinel_hook: /etc/apt/apt.conf.d/99-k3s-reboot-required
|
||||
@@ -35,9 +35,9 @@
|
||||
tags: [packages, config, verify]
|
||||
|
||||
# ── PACKAGES ────────────────────────────────────────────────────────────────────────────────
|
||||
- name: Install unattended-upgrades, the reboot-sentinel writer and needrestart
|
||||
# update-notifier-common is NOT optional padding: it owns the hook that creates
|
||||
# /var/run/reboot-required, which is the only thing kured watches. See defaults/main.yml.
|
||||
- name: Install unattended-upgrades and needrestart
|
||||
# The reboot-sentinel writer is NOT a package here. On Debian nothing ships one -- see
|
||||
# defaults/main.yml -- so this role installs its own script and dpkg hook, two tasks below.
|
||||
ansible.builtin.apt:
|
||||
name: "{{ unattended_upgrades_packages }}"
|
||||
state: present
|
||||
@@ -45,6 +45,60 @@
|
||||
cache_valid_time: 3600
|
||||
tags: [packages]
|
||||
|
||||
# ── THE REBOOT SENTINEL WRITER (ours, because Debian ships none) ────────────────────────────
|
||||
# Full rationale and the four measurements behind it: defaults/main.yml. Short version: kured
|
||||
# watches /var/run/reboot-required and nothing on Debian creates it, so without these two files
|
||||
# the entire reboot half is an invisible no-op.
|
||||
- name: Install the reboot-sentinel writer
|
||||
ansible.builtin.copy:
|
||||
dest: "{{ unattended_upgrades_sentinel_script }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
content: |
|
||||
#!/bin/sh
|
||||
# MANAGED BY ANSIBLE -- infra/ansible/roles/unattended_upgrades. Local edits are reverted.
|
||||
#
|
||||
# Touch /var/run/reboot-required when the RUNNING kernel is not the newest INSTALLED one.
|
||||
# kured watches that path; see the role's defaults/main.yml for why this is ours to ship.
|
||||
#
|
||||
# `linux-version` comes from linux-base (a dependency of every linux-image package), and
|
||||
# is used rather than sorting `dpkg -l` output because kernel version ordering is not
|
||||
# string ordering. If it is somehow missing, exit 0 and write nothing: a dpkg hook must
|
||||
# never fail an apt transaction, and a missing sentinel is caught by the verify assert.
|
||||
set -eu
|
||||
command -v linux-version >/dev/null 2>&1 || exit 0
|
||||
|
||||
running="$(uname -r)"
|
||||
newest="$(linux-version list 2>/dev/null | linux-version sort --reverse 2>/dev/null | head -n1)"
|
||||
|
||||
# No newest (nothing installed we can read) -> nothing to say.
|
||||
[ -n "$newest" ] || exit 0
|
||||
|
||||
if [ "$running" != "$newest" ]; then
|
||||
touch /var/run/reboot-required 2>/dev/null || exit 0
|
||||
# kured only needs the file, but the .pkgs companion is what an operator reads to find
|
||||
# out WHY a node is waiting, so write a reason rather than an empty marker.
|
||||
printf 'linux-image-%s\n' "$newest" > /var/run/reboot-required.pkgs 2>/dev/null || true
|
||||
fi
|
||||
exit 0
|
||||
tags: [packages, config]
|
||||
|
||||
- name: Install the dpkg hook that runs the sentinel writer
|
||||
# Post-Invoke, so it runs after dpkg has finished and a newly installed kernel is already
|
||||
# visible to `linux-version list`. `|| true` so a failure here can never abort an apt
|
||||
# transaction -- an unwritten sentinel is a missed reboot, but a broken apt is a broken node.
|
||||
ansible.builtin.copy:
|
||||
dest: "{{ unattended_upgrades_sentinel_hook }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: |
|
||||
// MANAGED BY ANSIBLE -- infra/ansible/roles/unattended_upgrades. Local edits reverted.
|
||||
// Writes /var/run/reboot-required when the running kernel is not the newest installed.
|
||||
DPkg::Post-Invoke { "{{ unattended_upgrades_sentinel_script }} || true"; };
|
||||
tags: [packages, config]
|
||||
|
||||
# ── CONFIG ──────────────────────────────────────────────────────────────────────────────────
|
||||
# Written as a whole file rather than patched in place, for the reason node_baseline gives for
|
||||
# its own apt source: an idempotent full-file write with a managed-by header is reviewable and
|
||||
@@ -246,21 +300,24 @@
|
||||
Effective apt config confirmed: kernel blacklisted, automatic reboot disabled.
|
||||
tags: [verify]
|
||||
|
||||
- name: Find the hooks that can create /var/run/reboot-required
|
||||
# THE SENTINEL IS THE SINGLE POINT OF FAILURE FOR THE ENTIRE REBOOT HALF OF THIS DESIGN, and
|
||||
# it is the one part that cannot be asserted from the repo — it depends on what
|
||||
# update-notifier-common actually ships on this Debian release. Rather than hard-code a path
|
||||
# that may not exist, discover the candidates and assert that at least one is present.
|
||||
ansible.builtin.find:
|
||||
paths:
|
||||
- /etc/kernel/postinst.d
|
||||
- /etc/apt/apt.conf.d
|
||||
patterns:
|
||||
- "*update-notifier*"
|
||||
- "*unattended-upgrades*"
|
||||
- "99update-notifier"
|
||||
file_type: any
|
||||
register: unattended_upgrades_sentinel_hooks
|
||||
# THE SENTINEL IS THE SINGLE POINT OF FAILURE FOR THE ENTIRE REBOOT HALF OF THIS DESIGN.
|
||||
# This was originally a `find` over /etc/kernel/postinst.d and /etc/apt/apt.conf.d, on the
|
||||
# theory that the writer came from a package and the role could only discover whatever that
|
||||
# package happened to ship. That theory was wrong: no such package exists on Debian (see
|
||||
# defaults/main.yml), so the writer is now OURS and lives at two known paths. Checking two
|
||||
# exact paths is strictly stronger than a glob -- a glob would have matched Debian's own
|
||||
# unattended-upgrades apt conffile and passed while no sentinel writer existed at all.
|
||||
- name: Stat the sentinel writer script
|
||||
ansible.builtin.stat:
|
||||
path: "{{ unattended_upgrades_sentinel_script }}"
|
||||
register: unattended_upgrades_sentinel_script_stat
|
||||
changed_when: false
|
||||
tags: [verify]
|
||||
|
||||
- name: Stat the sentinel dpkg hook
|
||||
ansible.builtin.stat:
|
||||
path: "{{ unattended_upgrades_sentinel_hook }}"
|
||||
register: unattended_upgrades_sentinel_hook_stat
|
||||
changed_when: false
|
||||
tags: [verify]
|
||||
|
||||
@@ -272,18 +329,21 @@
|
||||
# before, so it is an assert and not a debug message.
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- unattended_upgrades_sentinel_hooks.matched | int > 0
|
||||
- unattended_upgrades_sentinel_script_stat.stat.exists
|
||||
- unattended_upgrades_sentinel_script_stat.stat.executable | default(false)
|
||||
- unattended_upgrades_sentinel_hook_stat.stat.exists
|
||||
fail_msg: >-
|
||||
NOTHING ON {{ inventory_hostname }} CAN CREATE /var/run/reboot-required. No
|
||||
update-notifier / unattended-upgrades hook was found under /etc/kernel/postinst.d or
|
||||
/etc/apt/apt.conf.d. kured watches that path and reboots only when it exists, so with no
|
||||
writer the reboot automation is permanently and INVISIBLY idle -- kured stays 1/1 Running
|
||||
and reports kured_reboot_required=0 on every node forever. Confirm
|
||||
update-notifier-common is installed (`dpkg -l update-notifier-common`) and inspect
|
||||
`dpkg -L update-notifier-common | grep -E 'postinst.d|apt.conf.d'`.
|
||||
NOTHING ON {{ inventory_hostname }} CAN CREATE /var/run/reboot-required.
|
||||
{{ unattended_upgrades_sentinel_script }} (executable) and
|
||||
{{ unattended_upgrades_sentinel_hook }} must both exist -- this role ships both, because
|
||||
Debian provides no sentinel writer of its own (update-notifier-common is an UBUNTU
|
||||
package and is not installable here). kured watches that path and reboots only when it
|
||||
exists, so with no writer the reboot automation is permanently and INVISIBLY idle --
|
||||
kured stays 1/1 Running and reports kured_reboot_required=0 on every node forever.
|
||||
Re-run this role without `--tags verify` to install them.
|
||||
success_msg: >-
|
||||
Reboot sentinel writer present ({{ unattended_upgrades_sentinel_hooks.matched }} hook(s)
|
||||
found) -- a kernel or flagged package upgrade will signal kured.
|
||||
Reboot sentinel writer present ({{ unattended_upgrades_sentinel_script }} + dpkg hook)
|
||||
-- a kernel change will signal kured on the next apt transaction.
|
||||
tags: [verify]
|
||||
|
||||
- name: Report the pending-reboot state of this node
|
||||
|
||||
Reference in new issue
Block a user