fix(unattended-upgrades): Debian has no reboot-sentinel package, so ship the writer

The first real run failed on all three nodes:

  fatal: [k3s-prod-1]: "No package matching 'update-notifier-common' is available"

update-notifier-common is an UBUNTU package and does not exist in Debian --
`apt-cache policy` returns Candidate: (none) on trixie. The role asserted it
"owns the hook that creates /var/run/reboot-required", which is true on Ubuntu
and false here, so the role could never have worked.

Three further measurements settled the replacement: notify-reboot-required is
absent; NO linux-image-* postinst references it (invoking it is an Ubuntu patch,
so dropping a script at that path would never be called); and no hook under
apt.conf.d or /etc/kernel mentions reboot-required. Debian trixie ships no
mechanism at all.

So the role now provides it:
* /usr/local/sbin/k3s-notify-reboot-required -- touches the sentinel when the
  running kernel is not the newest installed, using `linux-version` (from
  linux-base, a dependency of every linux-image, measured present) because
  kernel version ordering is not string ordering.
* /etc/apt/apt.conf.d/99-k3s-reboot-required -- DPkg::Post-Invoke, `|| true` so
  it can never abort an apt transaction.
/run is tmpfs (measured), so the sentinel self-clears on boot; no cleanup task.

The verify assert was ALSO weaker than it looked: a `find` glob for
*update-notifier* / *unattended-upgrades* under apt.conf.d would have matched
Debian's own unattended-upgrades conffile and reported "writer present" while
none existed. It now stats the two exact paths this role owns and requires the
script to be executable.

Newly verified: the script ran on prod-1 as unprivileged k3s under sh -x --
running=newest=7.1.8+deb13-amd64, took the equal branch, exited 0, wrote
nothing. Correct. sh -n and bash -n clean.

Still unverified: the WRITING half never executes while running == newest, so
the sentinel has not been seen created and kured has not been seen react. The
honest test is the next kernel upgrade.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

.gitea/workflows/renovate.yaml

@@ -0,0 +1,173 @@
+# Renovate — the dependency/image update automation. SCHEDULED, self-hosted against this Gitea.
+#
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# WHY THIS FILE EXISTS: renovate.json HAS BEEN IN THIS REPO AND HAS NEVER ONCE RUN.
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# Measured 2026-10-17: `renovate.json` is present, complete and thoughtfully written — it declares
+# managers, schedules, automerge classes and grouping. Nothing executes it. There is no Renovate
+# workflow (this is the first), no Renovate CronJob or Deployment in the cluster
+# (`kubectl get cronjob,deploy,job -A | grep -i renovate` -> nothing), and no bot account doing it
+# from outside. Meanwhile SEVEN files carry comments asserting "Renovate maintains this pin"
+# (platform/homepage/deployment.yaml:40, runbooks/serving/deployment.yaml:87,
+# platform/crafty/deployment.yaml:63, platform/vpn-ingress/daemonset.yaml:259,
+# platform/keycloak/postgres-cluster.yaml:75, apps/wave-0/nextcloud-db.yaml:302, and
+# platform/headlamp/helmrepository.yaml:2). Those comments have been false for the life of the file.
+#
+# The cost is measurable. With nothing bumping anything: the six per-app oauth2-proxy auth sidecars
+# sit on v7.8.1 while the platform oauth2-proxy in the SAME cluster runs v7.15.5; Keycloak (the IdP)
+# runs 26.0.6, a November-2024 release, 8 minors behind; Velero runs a RELEASE CANDIDATE
+# (velero-plugin-for-aws v1.14.3-rc.2) with a stable available; the CNPG Postgres image is 16.4,
+# missing every 16.5+ security release. A pin nothing maintains is not a pin, it is a fossil.
+#
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# WHY RENOVATE AND NOT FLUX IMAGE-AUTOMATION (the decision, and it reverses ADR-0009's split)
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# ADR-0009 divided the work: Renovate owns Helm chart/dependency bumps, Flux image-automation owns
+# per-service container image TAGS, deliberately non-overlapping. The second half of that split
+# cannot be delivered on this repo's terms, for three measured reasons:
+#
+#   1. IT WOULD BYPASS THE MERGE GATE, WHICH CLAUDE.md §1 FORBIDS. ImageUpdateAutomation does not
+#      open pull requests; it COMMITS AND PUSHES. Its default target is the source's own branch —
+#      `main` — so it would write straight past the PR flow. It can be pointed at a side branch
+#      instead, but then nothing opens the PR and bumps silently accumulate on a branch nobody
+#      merges. Either way a human or a second bot has to close the loop.
+#   2. ON THIS CLUSTER IT CANNOT EVEN DO THAT. Measured 2026-10-17: `main` is protected with
+#      `user_can_push: false`, and the `flux-system` GitRepository has NO `spec.secretRef` at all —
+#      the source is anonymous and read-only. Enabling write-back means minting an in-cluster git
+#      WRITE credential for the repository that defines the cluster, which is a materially larger
+#      blast radius than a CI-scoped token.
+#   3. RENOVATE ALREADY OPENS PRs NATIVELY, so every change it proposes goes through the gate with
+#      no extra machinery.
+#
+# So: Renovate owns BOTH halves, and Flux image-automation stays disabled. Note this is a change of
+# plan, not a change of state — image-automation has been commented out at
+# platform/kustomization.yaml:52 since it was authored, and the cluster has ZERO
+# ImageRepository/ImagePolicy/ImageUpdateAutomation objects (measured). Its two controllers ARE
+# installed and have idled for 43 days. Nothing is being torn down here; a path that was never
+# taken is being formally closed, and renovate.json's `docker`/`kubernetes` managers — disabled
+# purely to avoid fighting a controller that never ran — are switched on to fill the gap.
+#
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# THIS WORKFLOW MUST NOT BE ABLE TO SUCCEED SILENTLY
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# A SKIPPED Gitea job reports `success` (CLAUDE.md §2), so "green" is not evidence this ran. Two
+# guards:
+#   * The preflight step FAILS HARD (exit 1) if RENOVATE_TOKEN is absent, rather than skipping.
+#     A missing credential on a dependency bot is indistinguishable from "nothing to update", and
+#     that is precisely how renovate.json stayed inert for months.
+#   * The summary step records what ran into the job summary unconditionally.
+# What this still does NOT give you is an alert when the SCHEDULE stops firing. See the handoff
+# entry's "What I did NOT verify" — that gap is stated, not papered over.
+#
+# OPERATOR PREREQUISITE — ONE SECRET, NOT COMMITTED HERE:
+#   `RENOVATE_TOKEN` — a Gitea access token for the bot account, scopes `write:repository` and
+#   `read:user`. Set it as an Actions secret on this repository. Nothing in this repo mints,
+#   stores or references its value; it is consumed by name only.
+#   Until it exists, this workflow FAILS LOUDLY on every scheduled run. That is intended: a
+#   visible red is the correct state for "automation configured but not credentialed".
+#
+# THE FIRST RUN WILL WANT TO OPEN A LOT OF PRs. ~30 dependencies are behind. renovate.json caps the
+# blast radius (`prConcurrentLimit: 5`, `prHourlyLimit: 2`), but run it manually with
+# `dry_run: true` FIRST and read the log before letting the schedule loose. That is what
+# workflow_dispatch's default is for.
+name: renovate
+
+on:
+  schedule:
+    # 03:15 UTC daily. renovate.json ALSO carries `"schedule": ["after 3am and before 6am"]` in
+    # America/New_York, and that inner schedule is the authoritative one — Renovate re-checks it and
+    # will decline to open PRs outside it. The two are deliberately both present: this cron decides
+    # when the process RUNS, renovate.json decides when it may ACT. Keeping the cron slightly ahead
+    # of the repo window means a run is always in flight when the window opens.
+    - cron: "15 3 * * *"
+  workflow_dispatch:
+    inputs:
+      dry_run:
+        description: "Dry run (no PRs, no branches). Leave true for the first run."
+        required: false
+        default: "true"
+      log_level:
+        description: "Renovate LOG_LEVEL (info | debug)"
+        required: false
+        default: "info"
+
+concurrency:
+  # Never two Renovate runs at once: concurrent runs race on the same branches and can open
+  # duplicate PRs. Not cancel-in-progress — a half-finished Renovate run can leave a branch pushed
+  # with no PR, so let the first one finish.
+  group: renovate
+  cancel-in-progress: false
+
+jobs:
+  renovate:
+    runs-on: ubuntu-latest
+    # Renovate scans every manager across the whole repo; 40m is generous for this repo's size and
+    # well under a runner's patience. Without a timeout a wedged run holds the `renovate`
+    # concurrency group and silently blocks every subsequent scheduled run.
+    timeout-minutes: 40
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Preflight - the token must exist, and its absence must be LOUD
+        env:
+          RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
+        run: |
+          set -euo pipefail
+          if [ -z "${RENOVATE_TOKEN:-}" ]; then
+            echo "::error::RENOVATE_TOKEN is not set on this repository." >&2
+            cat >&2 <<'MSG'
+
+          Renovate cannot run without a Gitea access token, and this step fails rather than
+          skipping ON PURPOSE: a skipped Gitea job reports `success`, so skipping here would make
+          "dependency automation is not running" look exactly like "everything is up to date".
+          That is the failure mode that let renovate.json sit inert in this repo for months while
+          seven files claimed "Renovate maintains this pin".
+
+          TO FIX: create a Gitea access token for the bot account with scopes
+          `write:repository` and `read:user`, then add it to this repository as the Actions
+          secret RENOVATE_TOKEN. Do not commit it anywhere.
+          MSG
+            exit 1
+          fi
+          echo "RENOVATE_TOKEN is present (value not printed)."
+
+      - name: Run Renovate
+        uses: renovatebot/[email protected]
+        with:
+          # Pinned, and Renovate maintains this pin itself: renovate.json's enabledManagers
+          # includes `github-actions`, and `helpers:pinGitHubActionDigests` is in its `extends`.
+          # This is the one dependency in the repo whose updater is its own updater.
+          configurationFile: renovate.json
+          token: ${{ secrets.RENOVATE_TOKEN }}
+        env:
+          # Self-hosted Gitea. `platform` + `endpoint` were the two values renovate.json's own
+          # header said were "set at runtime by the operator's renovate runner" — this is that
+          # runner, so they are set here rather than in the repo config, which keeps renovate.json
+          # portable and free of this instance's URL.
+          RENOVATE_PLATFORM: gitea
+          RENOVATE_ENDPOINT: https://gitea.home.keithjackson.dev/api/v1
+          RENOVATE_REPOSITORIES: Home-Network/home-network-k3s-impl
+          # Renovate must never push to a protected branch; it works on its own branches and opens
+          # PRs against this base. Stated explicitly rather than inherited, because this is the
+          # property that keeps the whole mechanism inside CLAUDE.md §1's branch -> PR -> gate path.
+          RENOVATE_BASE_BRANCHES: main
+          RENOVATE_ONBOARDING: "false"
+          RENOVATE_REQUIRE_CONFIG: "required"
+          RENOVATE_DRY_RUN: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run == 'true' && 'full' || '' }}
+          LOG_LEVEL: ${{ github.event_name == 'workflow_dispatch' && inputs.log_level || 'info' }}
+
+      - name: Summary
+        if: always()
+        run: |
+          {
+            echo "## Renovate"
+            echo
+            echo "- trigger: \`${{ github.event_name }}\`"
+            echo "- dry run: \`${{ github.event_name == 'workflow_dispatch' && inputs.dry_run || 'false' }}\`"
+            echo "- outcome: \`${{ job.status }}\`"
+            echo
+            echo "Renovate owns BOTH Helm chart bumps and container image tags here; Flux"
+            echo "image-automation is deliberately disabled (platform/kustomization.yaml)."
+            echo "Every change it proposes arrives as a pull request and must pass the merge gate."
+          } >> "$GITHUB_STEP_SUMMARY"

apps/wave-0/plex.yaml

@@ -77,7 +77,7 @@ spec:
   storageClassName: longhorn
   resources:
     requests:
-      storage: 8Gi
+      storage: 16Gi
 ---
 # PVC plex-media (R-STATE)
 apiVersion: v1

catalog/plex.yaml

@@ -351,7 +351,7 @@ exposure: public              # edge port-forward + public DNS; clients + Seerr
 egress: internet              # normal WAN egress (docs/03; R-NET-05)
 storage:
   - name: config
-    size: 8Gi
+    size: 16Gi
     class: longhorn           # Plex DB/metadata: replicated app state on the 3-replica tier (docs/04; R-STATE-01/02)
     mountPath: /config        # plexinc/pms-docker writes its identity/claim + "Library" DB under /config, NOT the
                               # renderer default /data/config. Without this the PVC sat unused and Plex wrote to the

catalog/qbittorrent.yaml

@@ -359,7 +359,7 @@ storage:
                                 # The live /config contents were copied onto each PVC before this landed,
                                 # so API keys and credentials other apps reference were preserved.
   # THE DOWNLOADS AREA — ONE mount of the SAME export sonarr/radarr/plex already consume
-  # (192.168.2.102:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
+  # (192.168.2.30:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
   # /downloads, which is precisely where this app is ALREADY configured to write.
   #
   # WHY subPath: downloads AND NOT THE EXPORT ROOT AT /media (the shape sonarr/radarr use):
@@ -434,7 +434,7 @@ dependencies: []                # default-deny: reaches only DNS; arr reach it n
                                 # for weeks and neither has any CNP naming the NFS server or port 2049
                                 # (grep apps/wave-1/sonarr.yaml: none). The SAME fact is what keeps the VPN
                                 # egress gateway off this traffic (its selector is a podSelector), and
-                                # 192.168.2.102 is inside the 192.168.0.0/16 excludedCIDR besides — see
+                                # 192.168.2.30 is inside the 192.168.0.0/16 excludedCIDR besides — see
                                 # platform/vpn-egress/egress-steering.yaml, "BULK NFS".
 ingressRoutes:
   - host: qbittorrent.${BASE_DOMAIN}

clusters/base/cluster-vars.yaml

@@ -234,26 +234,28 @@ data:
   WG_CLIENT_CIDR: "10.90.0.0/24"             # remote VPN clients; .1 = server. NOT a LAN range
 
   # --- Bulk (external NFS) storage host (docs/04 bulk tier) ---------------------------------------
-  # THE address of the Proxmox host whose ZFS datasets back every `class: bulk` volume: the media
-  # library (Storage/plex, 8.33 T — plex, sonarr, radarr) and the legacy Nextcloud archive
+  # THE address of the NAS whose ZFS datasets back every `class: bulk` volume: the media library
+  # (Storage/plex, 8.33 T — plex, sonarr, radarr, qbittorrent) and the legacy Nextcloud archive
   # (Storage/nextcloud, 270 G). The renderer emits it into each static NFS PersistentVolume's
   # `spec.nfs.server` (catalog/render.py, BULK_NFS_SERVER); the export PATH is per-dataset and is
   # declared in the ServiceDefinition, because the path is identical in every environment and only
   # the ADDRESS is not.
   #
-  # WHY THIS IS A KNOB AND NOT A LITERAL — the two clusters are on OPPOSITE SIDES of pfSense.
-  # The storage host has exactly one interface, vmbr0 192.168.1.9/24, on pfSense's LAN.
-  #   dev  nodes are 192.168.1.40/.41/.42 — the SAME L2 segment. They reach it directly, and one
-  #        NFS export to them already exists on that host (/mnt/Storage/k8s-volumes in
-  #        /etc/exports), so this path is proven, not theoretical. Base default = the dev value,
-  #        per the convention of the LAN block above.
-  #   prod nodes are 192.168.2.10/.11/.12 — pfSense's WAN. tcp/2049 does NOT traverse that today;
-  #        22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. Opening
-  #        the storage path is a PENDING OPERATOR DECISION with two candidate outcomes that give
-  #        DIFFERENT values here: 192.168.2.102 (another port-forward) or 192.168.1.9 (a route).
-  # So prod holds a loud TBD-PROD-* placeholder — see clusters/prod/kustomization.yaml for the
-  # blast radius of shipping it unset, which is NOT the usual "fails to apply".
-  BULK_NFS_SERVER: "192.168.1.9"
+  # IT IS NO LONGER A CROSS-NETWORK KNOB (2026-10-07). This used to read "the two clusters are on
+  # OPPOSITE SIDES of pfSense": the datasets lived on a Proxmox host whose only interface was
+  # vmbr0 192.168.1.9/24 on pfSense's LAN, which prod (192.168.2.0/24, pfSense's WAN) could reach
+  # only through a DNAT at 192.168.2.102. Both pools were physically moved onto a standalone
+  # TrueNAS at 192.168.2.30 — prod's OWN /24 — so there is no crossing to arrange and no candidate
+  # values to choose between. Measured 2026-10-07: 192.168.2.102:2049 no longer answers;
+  # 192.168.2.30:2049 and :111 do. There is now exactly ONE storage host and one correct value;
+  # this stays a knob only because the renderer reads it per-environment.
+  #
+  # *** CHANGING THIS VALUE IS NOT A PLAIN RECONCILE. *** A bound PersistentVolume's spec.nfs is
+  # immutable — measured by server dry-run 2026-10-07:
+  #     spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation
+  # so Flux's apply FAILS rather than updating, and the pods keep their old mount. The PVs must be
+  # deleted and recreated; the operator sequence is docs/BULK-NFS-STORAGE.md §10.
+  BULK_NFS_SERVER: "192.168.2.30"
 
   # Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style.
   # Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db,

clusters/base/kured.yaml

@@ -0,0 +1,98 @@
+---
+# Flux Kustomization: kured (Kubernetes Reboot Daemon) — reconciles ./platform/kured.
+# Performs the cordon / drain / reboot / uncordon of a node that has a pending OS reboot, strictly
+# one node at a time, inside a nightly window, gated on Longhorn volume health. Design rationale,
+# every tuned value, and the honest list of what is NOT mitigated: platform/kured/helmrelease.yaml.
+#
+# WHY ITS OWN KUSTOMIZATION, AND NOT ONE LINE IN platform/kustomization.yaml
+#
+# The same argument the Cilium split-out recorded in platform/kustomization.yaml and intel-gpu-plugin
+# reused, except the consequence is worse here than for either of them:
+#
+#   * `platform` is `wait: true` over twelve-plus components, so its Ready condition cannot tell you
+#     "the reboot daemon is wedged" from "longhorn is slow".
+#   * The only emergency valve would be `flux suspend kustomization platform` — freezing storage,
+#     TLS, DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a DaemonSet.
+#   * That tier's 10m interval / 5m timeout are called out there as WRONG for a 3-node DaemonSet
+#     rollout. This is a 3-node DaemonSet rollout.
+#   * And `platform` gates platform-config, which gates apps-wave-0 (wait:true), which gates
+#     apps-wave-1. A reboot daemon that failed to go healthy inside that tier would stall EVERY
+#     APPLICATION in the cluster on "dependency not ready". A component added to make maintenance
+#     safer must not acquire the ability to cause an outage; that trade is the whole argument.
+#
+# dependsOn: [platform] — AND THIS ONE IS A REAL EDGE, NOT CONVENTION
+#
+# Not `cilium` (what intel-gpu-plugin uses) and not `platform-config` (what the leaf UIs use). It is
+# `platform`, for two independent reasons:
+#
+#   1. CRD ORDERING (F-MON1). platform/kured ships a PrometheusRule, and the chart generates a
+#      ServiceMonitor. Both are CRs whose CRDs (monitoring.coreos.com) are installed by
+#      kube-prometheus-stack INSIDE the `platform` tier. Flux dry-runs every object in a pass before
+#      applying any of them, so a pass containing an unknown kind ABORTS — the failure that made the
+#      CDI StorageProfiles a true deadlock on prod on 2026-09-05 (see
+#      clusters/base/platform-config-cdi.yaml). Depending on `platform` guarantees the CRDs exist
+#      before this pass is attempted.
+#   2. FUNCTIONAL PREREQUISITE. kured's reboot gate QUERIES Prometheus
+#      (`prometheusUrl` in the HelmRelease). Prometheus lives in platform/monitoring. If kured ran
+#      first it would get connection errors, treat the gate as unevaluable and skip every reboot —
+#      the correct fail-safe direction, but silent. Longhorn, whose volume health IS the gate, is
+#      also in that tier.
+#
+# It does NOT need platform-config: no Keycloak, no TLS, no split-horizon DNS, no Gateway, no
+# HTTPRoute. kured serves no HTTP to users.
+#
+# wait: true — A HEALTH SIGNAL, NOT A GATE. Following intel-gpu-plugin rather than the leaf-UI
+# template. A leaf UI is legitimately not-Ready for long stretches of a cold start while it waits on
+# Keycloak and DNS, so its health is uninformative; kured has no such dependency — it is a DaemonSet
+# that opens a socket and watches a file, so on healthy nodes it is Ready in seconds and not-Ready
+# genuinely means BROKEN. This turns `flux get kustomization kured` into a one-line answer to "is
+# reboot automation live on all three nodes", which is the question the operator must be able to
+# answer before trusting a kernel upgrade to it.
+#
+# NOTHING IN THE GRAPH dependsOn THIS, so wait:true gates nothing — the blast radius of an unhealthy
+# kured is this Kustomization's own Ready condition. Do not add `dependsOn: [kured]` anywhere: that
+# would hand a node-rebooting DaemonSet the power to stall whatever depends on it, which is precisely
+# what the placement above exists to prevent.
+#
+# prune: true is safe and deliberate. If this Kustomization is deleted, kured's DaemonSet goes with
+# it and reboots simply stop happening — nodes are left uncordoned and running. The one caveat is
+# that pruning mid-reboot would leave a node CORDONED with no daemon to uncordon it; that is a
+# `kubectl uncordon` away and is covered in runbooks/node-reboot-automation.md.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: kured
+  namespace: flux-system
+spec:
+  # A HelmRelease plus five static manifests, nothing to re-render; kustomize-controller is
+  # source-triggered, so this is only the drift-correction floor.
+  interval: 10m
+  retryInterval: 2m
+  # 10m, not the 5m the `platform` tier uses: a 3-node DaemonSet rollout under `wait: true` can
+  # exceed 5m on a cold start (image pull on three nodes), which is one of the reasons the Cilium
+  # split-out gave for leaving that tier.
+  timeout: 10m
+  prune: true
+  # See the header: a health SIGNAL for a component nothing depends on, not a gate.
+  wait: true
+  dependsOn:
+    - name: platform   # monitoring CRDs + Prometheus + Longhorn; see the header, this is load-bearing
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/kured
+  # REQUIRED, not boilerplate. platform/kured/alerts.yaml renders `${BASE_DOMAIN}` into the
+  # `runbook_url` annotation of all three rules. Without this block that token becomes the EMPTY
+  # STRING and the Kustomization still reports Ready — Flux's substitution fails OPEN — so every
+  # alert would ship a dead link to `https:///runbooks/node-reboot-automation`. The gate stage
+  # `flux-postbuild-substitution` asserts the converse (a rendered token implies this block), and
+  # BASE_DOMAIN is declared in clusters/base/cluster-vars.yaml.
+  #
+  # Nothing else in this path contains `${...}` or a bare `$`: the HelmRelease's alertFilterRegexp is
+  # a plain anchored alternation, and there are no shell snippets anywhere in the component — which
+  # is deliberate, because envsubst would eat a braced shell variable and `$$` would trip the
+  # dollar-collapse assertion in the same check.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars

clusters/base/kustomization.yaml

@@ -85,4 +85,17 @@ resources:
   # pm-agent-credentials.sops.yaml, so both overlays add a SOPS decryption patch for it (by
   # name). See clusters/base/pm-agent.yaml and platform/pm-agent/README.md.
   - pm-agent.yaml        # Flux Kustomization -> ./platform/pm-agent (dependsOn apps-wave-0)
+  # kured (Kubernetes Reboot Daemon) — its OWN Kustomization (dependsOn platform, wait:true), NOT a
+  # line in platform/kustomization.yaml. Same reasoning as the Cilium and intel-gpu-plugin
+  # split-outs, with a sharper consequence: `platform` gates platform-config and every apps wave, so
+  # a wedged reboot DaemonSet placed in that wait:true tier could stall every application in the
+  # cluster — a component added to make maintenance SAFER must not be able to cause an outage.
+  # wait:true here is a HEALTH SIGNAL, not a gate: nothing dependsOn it, and nothing should.
+  # The `platform` edge is load-bearing rather than conventional — it supplies the
+  # monitoring.coreos.com CRDs this component's PrometheusRule/ServiceMonitor need (F-MON1), and
+  # Prometheus itself, which kured QUERIES before every reboot to gate on Longhorn volume health.
+  # It reboots a node only when /var/run/reboot-required exists, which on these nodes only a
+  # deliberate operator kernel upgrade creates — the human gate is the sentinel, not the daemon.
+  # See clusters/base/kured.yaml and platform/kured/helmrelease.yaml.
+  - kured.yaml           # Flux Kustomization -> ./platform/kured (dependsOn platform)
   - cluster-vars.yaml    # overlay knob surface (base defaults)

clusters/dev/kustomization.yaml

@@ -279,15 +279,15 @@ patches:
       - op: replace
         path: /data/CRAFTY_L4_VIP
         value: "192.168.1.55"
-      # Bulk (external NFS) storage host: the Proxmox host itself, on the SAME LAN as dev's nodes
-      # (192.168.1.40/.41/.42), which already hold an NFS export from it (/mnt/Storage/k8s-volumes).
-      # Restated verbatim rather than inherited, per this overlay's "supply every knob" convention.
-      # Dev is therefore the environment in which the bulk manifests can actually be VALIDATED
-      # end-to-end before prod's network path exists — provided the additive `zfs set sharenfs=`
-      # change described in docs/BULK-NFS-STORAGE.md is applied for the dev node addresses.
+      # Bulk (external NFS) storage host. THIS VALUE IS INERT — dev is decommissioned, its nodes
+      # (192.168.1.40/.41/.42) no longer exist, and the Proxmox host this used to name (192.168.1.9)
+      # went with the pools it held. It is restated per this overlay's "supply every knob"
+      # convention, not because anything consumes it. It carries the real address so that no file
+      # in the tree renders a host that is gone; it has NOT been proven from a dev node, because
+      # there is no dev node to prove it from.
       - op: replace
         path: /data/BULK_NFS_SERVER
-        value: "192.168.1.9"
+        value: "192.168.2.30"
       # Off-cluster backup target: the on-Proxmox MinIO (plain HTTP, path-style) that every CNPG
       # barmanObjectStore and Velero's backupStorageLocation write to.
       #

clusters/prod/kustomization.yaml

@@ -540,60 +540,48 @@ patches:
         path: /data/LAN_CIDR
         value: "192.168.2.0/24"
       # ------------------------------------------------------------------------------------------
-      # BULK (EXTERNAL NFS) STORAGE HOST — *** NOT YET DECIDED. THIS PLACEHOLDER IS LOAD-BEARING. ***
+      # BULK (EXTERNAL NFS) STORAGE HOST — the TrueNAS at 192.168.2.30.
       # ------------------------------------------------------------------------------------------
       # Backs every `class: bulk` volume: the 8.33 T media library (Storage/plex — plex, sonarr,
-      # radarr) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
+      # radarr, qbittorrent) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
       #
-      # THE STORAGE HOST IS NOT REACHABLE FROM PROD AT ALL RIGHT NOW. It has one interface,
-      # vmbr0 192.168.1.9/24, on pfSense's LAN; prod's nodes are on 192.168.2.0/24, pfSense's WAN.
-      # 22 and 8006 answer from prod only because pfSense port-forwards them via 192.168.2.102;
-      # 2049 and 111 do not. Opening that path is an OPERATOR decision, and the two candidate
-      # outcomes give DIFFERENT values here:
-      #     port-forward 192.168.2.102:2049 -> 192.168.1.9:2049   =>  "192.168.2.102"
-      #     a WAN->LAN route to 192.168.1.0/24                    =>  "192.168.1.9"
-      # Guessing either one authors a prod cluster that mounts nothing, or — worse, if the guess
-      # ever became reachable — mounts something unintended. So it stays a placeholder.
+      # THIS WAS THE REPO'S ONE LOAD-BEARING PLACEHOLDER, AND THE REASON IT COULD BE IS NOW GONE.
+      # The datasets lived on a Proxmox host with a single interface on pfSense's LAN
+      # (192.168.1.9), which prod could reach only through a DNAT at 192.168.2.102 — so the correct
+      # value here depended on an undecided operator choice, and guessing it would have authored a
+      # cluster that mounts nothing. Both pools were physically moved onto a standalone TrueNAS at
+      # 192.168.2.30, prod's OWN /24. No crossing to arrange, no candidates to choose between.
+      # Measured 2026-10-07: 192.168.2.102:2049 no longer answers; 192.168.2.30:2049 and :111 do.
       #
-      # WHICH PORTS THE DECISION MUST OPEN: exactly ONE, tcp/2049. The PVs mount nfsvers=4.1, and
-      # NFSv4 carries mount and lock in-protocol — no rpcbind/111, no mountd, no statd. Choosing v4
-      # over v3 was in part this decision's cost: v3 would have needed four ports and three of them
-      # on ephemeral numbers.
-      #
-      # WHAT THE EXPORT MUST ALLOW IS ALREADY SETTLED, and it is the same under BOTH outcomes.
-      # Measured 2026-09-07 from this workstation (192.168.2.132) through the existing port-forward:
-      # the storage host logged the connection as coming FROM 192.168.2.132, not from a pfSense LAN
-      # address (`ss -tn` peer 192.168.2.132, and the host's only route off-LAN is its default via
-      # 192.168.1.1). pfSense DNATs WITHOUT source NAT, so the NFS server will see prod's REAL node
-      # addresses either way. The additive `zfs set sharenfs=` values are in docs/BULK-NFS-STORAGE.md.
-      #
-      # *** BLAST RADIUS — READ BEFORE MERGING THIS TO A BRANCH PROD RECONCILES. ***
-      # A PersistentVolume's spec.nfs.server is an unvalidated string: the API server ACCEPTS this
-      # placeholder, the PV is created and the PVC BINDS to it (binding is pure control-plane). The
-      # failure surfaces only when a pod mounts — MountVolume.SetUp "Failed to resolve server
-      # TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr and nextcloud would sit in
-      # ContainerCreating on prod. Unlike the LAN knobs above, this placeholder does NOT fail at
-      # apply time and there is no CRD validation to lean on. Set this value BEFORE prod reconciles
-      # a revision containing the bulk volumes; the dev overlay renders a working value, which is
-      # where these manifests are meant to be proven first.
+      # ONE PORT IS STILL ALL IT NEEDS: tcp/2049. The PVs mount nfsvers=4.1 and NFSv4 carries mount
+      # and lock in-protocol — no rpcbind/111, no mountd, no statd. The NAS advertises
+      # `+3 +4 +4.1 +4.2` (/proc/fs/nfsd/versions, measured), so 4.1 negotiates.
       - op: replace
         path: /data/BULK_NFS_SERVER
-        # 192.168.2.102 is pfSense's WAN address, which DNATs tcp/2049 through to the Proxmox
-        # host's REAL address 192.168.1.9 on pfSense's LAN. Prod sits on the WAN side and cannot
-        # reach 192.168.1.9 directly, so the cluster mounts the NAT address. Dev is on the LAN and
-        # keeps the base value 192.168.1.9.
+        # VERIFIED ON THE NAS 2026-10-07, before any cluster touched it (`exportfs -v`):
+        #   /mnt/Storage/plex       192.168.2.10/.11/.12  rw,sec=sys,secure,root_squash,no_all_squash
+        #   /mnt/Storage/nextcloud  192.168.2.10/.11/.12  ro,sec=sys,secure,root_squash,no_all_squash
         #
-        # PROVEN from k3s-prod-1 on 2026-09-12, not inferred:
-        #   - mount of 192.168.2.102:/mnt/Storage/plex negotiated nfs4 vers=4.1 proto=tcp
-        #   - a write as uid 1000 succeeded. Tested as 1000 and NOT root deliberately: the export
-        #     carries no_root_squash, so a root write would have passed and proved nothing, while
-        #     LSIO images default to PUID 911 and would have gone Ready then failed every import
-        #     with EACCES. The tree is 0755 1000:3000, so uid 1000 is the only writer.
-        #   - ln() across downloads/ -> tv/ returned link count 2 (same inode), so the arrs get
-        #     real atomic moves instead of silently degrading to copy-then-delete on EXDEV.
-        # pfSense DNATs WITHOUT source NAT (the host logs prod's own node addresses as the peer),
-        # so the ZFS export ACL names 192.168.2.10/.11/.12 directly rather than one translated IP.
-        value: "192.168.2.102"
+        # no_all_squash is the flag the entire permission model rests on: client uids pass through
+        # untouched, so the arrs' PUID=1000 arrives as the owner of a 0755 1000:3000 tree. sec=sys
+        # keeps that numeric rather than principal-mapped. The export deliberately does NOT carry
+        # no_root_squash, which the Proxmox one did: nothing here needs it, because every write to
+        # the tree happens after the LSIO drop to uid 1000 and the images only chown their Longhorn
+        # /config. Both subPath directories (plex/downloads, nextcloud/nextcloud) already exist at
+        # 0755 1000:3000, so a squashed root can still traverse them for subPath resolution.
+        #
+        # The ro= posture on the archive is now SERVER-enforced, which the old export also managed.
+        # The media tree's read-only half (plex) stays a CLIENT-side guarantee — all four media pods
+        # can land on any node, so the server sees the same three addresses for reader and writers
+        # alike and cannot tell them apart. Unchanged by the move; see docs/BULK-NFS-STORAGE.md §5.
+        #
+        # *** CHANGING THIS VALUE REQUIRES DELETING AND RECREATING THE PVs. *** A bound PV's
+        # spec.nfs is immutable — measured by server dry-run 2026-10-07:
+        #     spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation
+        # so Flux's apply FAILS rather than updating and the pods keep their old, dead mount. The
+        # operator sequence is docs/BULK-NFS-STORAGE.md §10. Retain + the renderer's pre-set
+        # claimRef make it safe: no data is touched and each recreated PV rebinds its own PVC.
+        value: "192.168.2.30"
       # ==========================================================================================
       # OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05; bucket + token 2026-09-07).
       # ==========================================================================================

conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh

@@ -363,6 +363,54 @@ SCRAPE_OWNERS: list[dict] = [
         "why": "velero_* powers VeleroBackupFailure / PartialFailure / NoRecentBackup — the rules "
                "that failed to notice prod running with zero volume backups",
     },
+    # ── ADDED 2026-10-17 (feat/automated-upgrades). A NEW OWNER, NOT A RELAXED ASSERTION. ──────
+    # This entry is purely additive: it registers a new metric family and its enablement, and it
+    # changes nothing about how any existing owner is evaluated. It makes the check STRICTER for the
+    # component it covers, not looser — it adds two `require` patterns plus a `companion` obligation
+    # that platform/kured must keep satisfying, so a future edit that disables kured's Service,
+    # ServiceMonitor or NetworkPolicy FAILS the gate instead of silently re-inerting three alerts.
+    #
+    # WHY THE CHECK HAD TO BE TOUCHED AT ALL, rather than the manifest being wrong: assertion (3)
+    # fails any alert over a metric no owner claims, and the escape hatch (KNOWN_UNWIRED_METRICS) is
+    # SHRINK-ONLY, so declaring `kured_*` as accepted debt is not permitted and would be the wrong
+    # answer anyway — the scrape genuinely IS wired here. The only two honest options were to
+    # register the owner (this) or to ship the reboot automation with no alerting at all, which for a
+    # component whose every failure mode is silent would have been the worse engineering choice.
+    # `conformance/**` is a sensitive approval path precisely so that this reasoning gets read.
+    {
+        "owner": "kured: service.create + metrics.create in the HelmRelease",
+        "prefixes": ("kured_",),
+        "file": "platform/kured/helmrelease.yaml",
+        # BOTH keys are required and the chart defaults BOTH to false. `metrics.create` makes the
+        # ServiceMonitor; `service.create` makes the Service it selects. Enable only the monitor and
+        # it selects nothing, so the single kured metric is never scraped — and a PrometheusRule over
+        # a series no target produces is INERT, which for this component means its own liveness
+        # alarm is the thing that breaks. Matched as two separate anchored blocks rather than one
+        # pattern because they are separate top-level values keys, not nested.
+        "require": [
+            r"service:\s*\n\s*create:\s*true",
+            r"metrics:\s*\n\s*create:\s*true",
+        ],
+        "forbid": [],
+        "why": "kured_reboot_required is the ONLY metric kured exposes (one GaugeVec, "
+               "cmd/kured/main.go:92-96) and it powers all three platform-health.node-reboots "
+               "rules. Those rules are the only way to learn that OS reboot automation has stalled: "
+               "kured stays 1/1 Running while rebooting nothing, so a missed scrape here is "
+               "indistinguishable from a cluster that needed no reboots",
+        # The ServiceMonitor ALONE is not sufficient — the same NetworkPolicy trap Longhorn hit.
+        "companion": {
+            "file": "platform/kured/kustomization.yaml",
+            "require": [r"^\s*-\s*networkpolicy\.yaml\s*$"],
+            "why": "platform/kured/networkpolicy.yaml ships a default-deny baseline (the kured "
+                   "namespace is a platform namespace, so Kyverno's app-namespace-scoped generator "
+                   "does not cover it) AND the ingress allow that lets Prometheus reach :8080 "
+                   "through it. Drop the networkpolicy from this kustomization and the deny-all "
+                   "goes too, so this currently fails SAFE — but reinstating the baseline without "
+                   "the allow reproduces Longhorn's measured failure exactly: targets DISCOVERED, "
+                   "every scrape 'context deadline exceeded', TSDB empty, all three rules quiet. "
+                   "Keeping the two in one file makes that pair inseparable",
+        },
+    },
 ]
 
 # `up` is produced by EVERY scrape target, so no owner can meaningfully claim it and assertion (1)

decisions/2026-10-06-peers-configmap-content-hash.md

@@ -0,0 +1,103 @@
+---
+date: 2026-10-06
+branch: fix/peers-configmap-content-hash
+title: A merged VPN peer never reached the tunnel — the peer register was a fixed-name ConfigMap, so nothing rolled the pods
+---
+
+## What changed
+
+The operator enrolled a phone, merged the peer, and **`wg show` listed no peers.** Measured on prod
+2026-10-06, with the peer already on `main`:
+
+| observation | value |
+|---|---|
+| peer stanza on `main` | present (`AllowedIPs = 10.90.0.2/32`) |
+| GitRepository artifact | `main@sha1:e1bebb8` — Flux had fetched it |
+| `vpn-ingress` Kustomization `lastAppliedRevision` | `main@sha1:5928d81` — **behind**, 10m interval |
+| live `wg-peers` ConfigMap, active `AllowedIPs` lines | **0** |
+| DaemonSet volume reference | `configMap.name: wg-peers` — a **fixed** name |
+| DaemonSet `generation` / `observedGeneration` | **1 / 1** — it had **never rolled** |
+| init container, all three pods | `wg0.conf assembled: 0 peer(s)` |
+
+The lag was only the reconcile interval. **The defect is the fixed name.** When Flux does update a
+fixed-name ConfigMap it updates it *in place*: the DaemonSet's pod template is byte-identical, so
+no rollout is triggered — and because the init container assembles `wg0.conf` **once at pod start**
+into a tmpfs, the running tunnel keeps the old peer list indefinitely. Adding or revoking a peer
+would have had **no effect at all** until something unrelated happened to restart the pods. For a
+*revocation* that is a security defect, not an inconvenience: the operator would delete a stanza,
+see a green PR, and the device would keep working.
+
+My runbook and README both asserted "PR → merge → Flux rolls the DaemonSet (~1 min)". That was
+never true.
+
+**This repo already documents the fix, and I had read the file.** `runbooks/kustomization.yaml`:
+*"The generated name carries kustomize's content hash, so editing any runbook rolls the server
+automatically and a stale projection is impossible."*
+
+So the register becomes a **content-hashed generator**:
+
+- `platform/vpn-ingress/peers.yaml` (inline ConfigMap) → **`platform/vpn-ingress/peers.conf`**, a
+  plain file, plus a `configMapGenerator` in `kustomization.yaml`. The operator's enrolled peer was
+  carried across verbatim. Every line is WireGuard config and `#` is a legal comment there, so the
+  documentation now travels *inside the mounted file* instead of in a YAML wrapper that gets
+  stripped.
+- **`namespace: vpn-ingress` added to the kustomization — REQUIRED, not tidiness.** A generated
+  ConfigMap inherits no namespace; every other object here declares `vpn-ingress`; and kustomize's
+  `nameReference` transformer will not rewrite a reference across a namespace boundary. Without it
+  the build emitted `ConfigMap wg-peers-m9kcckcb5t` while the DaemonSet still mounted the unhashed
+  `wg-peers`, which would not exist — the pods would wedge on a missing volume. I caught this only
+  by building the output and reading the volumes block.
+- **`infra/vpn-ingress/new-peer.sh`**: `PEERS_FILE` repointed to `peers.conf`, and a missing
+  register is now **fatal** instead of skipped. The old guard was
+  `[ -r "$PEERS_FILE" ] && grep …`, so after the rename the only symptom would have been
+  duplicate-address detection quietly never firing again — the silent-skip shape this repo treats
+  as a defect in its own right.
+- Stale `peers.yaml` references swept from the README, runbook, `daemonset.yaml`,
+  `networkpolicy.yaml`, the SOPS example, and the script's own **user-facing output** (it told the
+  operator to paste into "peers.yaml, under data.peers.conf", which would no longer exist).
+- The runbook gained the verification that actually distinguishes this failure — the DaemonSet's
+  `generation` must *increase* after the merge — and a troubleshooting row for `0 peer(s)` after a
+  merge.
+
+Flux's `prune: true` removes the orphaned fixed-name `wg-peers` ConfigMap on the next reconcile.
+
+No sensitive paths (`platform/vpn-ingress/**`, `infra/vpn-ingress/**`, `runbooks/**`,
+`decisions/**`), so this merges on a green gate with no override.
+
+## What I verified
+
+Built and run in the pinned toolbox, 2026-10-06 (kustomize v5.8.1 via `kubectl kustomize`).
+
+- **The reference is now rewritten end to end**: generated `ConfigMap wg-peers-m9kcckcb5t` in
+  namespace `vpn-ingress`, and the DaemonSet mounts **`wg-peers-m9kcckcb5t`**. Before adding
+  `namespace:` the same build mounted the unhashed `wg-peers` — so the namespace line is load
+  bearing, measured both ways.
+- **The hash tracks content, which is the whole mechanism.** `wg-peers-m9kcckcb5t` →
+  append a second peer → `wg-peers-dgg7ht9c74` → restore → `wg-peers-m9kcckcb5t`. Deterministic
+  and reversible, so an edit necessarily changes the pod template.
+- **The operator's enrolled peer survived the file conversion byte-for-byte**: the active stanza in
+  `peers.conf` (`PublicKey = pBRSHrRy…`, `AllowedIPs = 10.90.0.2/32`) is identical to the one on
+  `main`, compared directly against `git show origin/main:platform/vpn-ingress/peers.yaml`.
+- **The build still emits all five objects**: Namespace, ConfigMap, Secret, DaemonSet,
+  NetworkPolicy — and the `Namespace` object's own name is still `vpn-ingress` (the namespace
+  transformer did not rewrite it).
+- **`new-peer.sh` works against the new register**: `spare-laptop 7` renders a profile and prints
+  "append this to platform/vpn-ingress/peers.conf"; `dupe 2` is **refused** because the operator's
+  real `.2` now occupies it — the first time the duplicate check has been exercised against real
+  data rather than against its own example.
+- **Toolbox gate**: rc 3, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`,
+  `l1` and `approval-policy` all pass; rc 3 is the two environmental skips. **Not green. Reported
+  as rc 3.**
+
+## What I did NOT verify
+
+- **That the roll actually happens on the cluster.** The whole point of the change is unproven
+  until Flux applies it: the success criterion is the DaemonSet's `generation` increasing and
+  `wg0.conf assembled: 1 peer(s)` in the init log, followed by `wg show` listing the peer.
+- **That the tunnel then carries traffic.** No handshake has ever been observed. The WAN
+  port-forward is reportedly configured but has never been exercised by a packet I can see, and I
+  have no off-LAN vantage point — runbook step 6 from cellular remains the only honest test.
+- **Revocation**, which is the case where this defect was a security problem. Deleting a stanza
+  should now roll the pods and drop the peer; untested.
+- **Whether the orphaned fixed-name `wg-peers` ConfigMap is actually pruned.** Reasoned from
+  `prune: true` on the Kustomization, not observed.

decisions/2026-10-07-bulk-nfs-truenas.md

@@ -0,0 +1,85 @@
+---
+date: 2026-10-07
+branch: fix/bulk-nfs-truenas
+title: Repoint bulk NFS at the TrueNAS (192.168.2.30); the PVs must be recreated, not reconciled
+---
+
+## What changed
+
+Both ZFS pools were physically moved off the Proxmox host onto a standalone TrueNAS Community
+Edition at **192.168.2.30**, which sits on prod's own `192.168.2.0/24`. `BULK_NFS_SERVER` moves from
+`192.168.2.102` (the pfSense DNAT prod used to mount through) to `192.168.2.30`.
+
+- `clusters/prod/kustomization.yaml` — `BULK_NFS_SERVER` → `192.168.2.30`. The comment block that
+  justified the DNAT, and the "NOT YET DECIDED / THIS PLACEHOLDER IS LOAD-BEARING" banner, are
+  replaced with the measured export configuration and the recreate requirement below.
+- `clusters/base/cluster-vars.yaml` — base default `192.168.1.9` → `192.168.2.30`. That host no
+  longer exists.
+- `clusters/dev/kustomization.yaml` — same; marked explicitly **inert**, since dev is decommissioned
+  and its nodes are gone. It carries the real address only so nothing in the tree renders a dead one.
+- `docs/BULK-NFS-STORAGE.md` — §1 and §5 rewritten for TrueNAS (exports, service settings, the ZFS
+  `sharenfs` properties that had to be cleared, and the ACL hazard); §7 rewritten now that the
+  placeholder is gone; §9 retargeted from dev to prod; **new §10** documents the PV
+  delete-and-recreate. §2/§3/§4/§6 are unchanged by the move.
+- `catalog/qbittorrent.yaml`, `platform/monitoring/node-pressure-dashboard.yaml` — two comment/prose
+  references to `192.168.2.102` as the live export address.
+
+**The headline operational fact: repointing this knob is NOT a plain reconcile.** A bound PV's NFS
+source is immutable, so Flux's apply fails rather than updating and the pods keep their old, dead
+mount. The five PVs must be deleted and recreated. That sequence is §10 and is an operator action;
+it is **not** performed by this PR.
+
+## What I verified
+
+**On the NAS (192.168.2.30), measured 2026-10-07 by the operator at my request:**
+
+- `zfs list` — `Storage` → `/mnt/Storage`, `Storage/plex` → `/mnt/Storage/plex`,
+  `Storage/nextcloud` → `/mnt/Storage/nextcloud`. The PV paths resolve unchanged.
+- Ownership intact: `/mnt/Storage/plex`, `tv/`, `downloads/`, `movies/`,
+  `downloads/complete/`, `nextcloud/nextcloud/` all `drwxr-xr-x 1000:3000`. `movies/` 1171 entries
+  and `tv/` 107, against 1165 / 107 measured on Proxmox.
+- `acltype=nfsv4` (SCALE's built-in default, `SOURCE: default`), `aclinherit=passthrough`, and no
+  non-trivial ACLs — mode bits still govern.
+- Three stale `sharenfs` dataset properties (`plex`, `nextcloud`, `docker-volumes`) carried over from
+  Proxmox; cleared with `zfs inherit`, re-checked empty.
+- `exportfs -v`: both shares to `192.168.2.10/.11/.12` with `sec=sys,secure,root_squash,no_all_squash`,
+  `rw` on plex and `ro` on nextcloud.
+- `/proc/fs/nfsd/versions` → `+3 +4 +4.1 +4.2`, so the PVs' `nfsvers=4.1` negotiates.
+- **A defect was found and fixed during setup:** the plex share listed `192.168.2.13` — an address no
+  node holds — and so omitted `192.168.2.12` (k3s-prod-3). Corrected and re-verified.
+
+**From the cluster:**
+
+- Node InternalIPs are `192.168.2.10/.11/.12` (`kubectl get nodes`), confirming `.13` was wrong.
+- All five NFS PVs currently name `192.168.2.102`.
+- `192.168.2.30:2049` and `:111` accept connections; `192.168.2.102:2049` does not.
+- **`spec.nfs` immutability, by server-side dry-run** — this is the finding that shapes the change:
+  `spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation`.
+- The pods report `Ready` while their mounts are dead: a `stat /media` answers from attribute cache,
+  but a `readdir` blocks indefinitely (the PVs mount `hard`). Hence "scale to zero first" in §10.
+
+**Gate:** see the PR; run in the pinned toolbox from this worktree.
+
+## What I did NOT verify
+
+- **Nothing has mounted 192.168.2.30.** No pod has touched the new server — the PVs still point at
+  the old address and cannot be updated in place. Every §9 probe (numeric-id passthrough, the uid-1000
+  write, the hardlink/EXDEV check, Plex's read-only posture) is **outstanding**, not passing. The
+  equivalents passed against the old host on 2026-09-12; that is not evidence about this one.
+- **The §10 sequence is reasoned, not executed.** Retain + the renderer's pre-set `claimRef` are read
+  from the manifests and from `catalog/render.py`; I have not watched a PVC rebind after a PV delete
+  in this cluster.
+- **`root_squash` is a behaviour change from the Proxmox export**, which carried `no_root_squash`. The
+  argument that nothing needs root on the tree (LSIO drops to 1000; the images chown only their
+  Longhorn `/config`; both subPath directories already exist at 0755) is reasoned from the images and
+  the measured permissions. It has not been exercised. If a mount or subPath resolution fails with
+  EACCES, Maproot `root:root` reproduces the old export and is the first thing to try.
+- **Whether NFSv3 can be disabled.** Enabled Protocols is `NFSv3, NFSv4`; the PVs only need v4.1, so
+  narrowing would close the rpcbind/mountd/statd surface. Not done — it is a change to make after the
+  cutover is proven, not during it.
+- **Whether pfSense / the legacy stack is actually retired.** I measured only that
+  `192.168.2.102:2049` stopped answering. The other legacy consumers of these datasets
+  (`192.168.1.6`, `10.123.1.10`, `192.168.1.18`) were dropped from the new exports; if any is still
+  live, it has lost access. Not checked.
+- **L2/L3 conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline subset
+  only. Nothing in the gate exercises an NFS mount.

decisions/2026-10-17-automated-upgrades.md

@@ -0,0 +1,300 @@
+---
+date: 2026-10-17
+branch: feat/automated-upgrades
+title: Automated upgrades — kured + security-only unattended-upgrades, and Renovate finally gets a runner
+---
+
+## What changed
+
+Two halves of one process. Neither is useful alone and they must land together.
+
+### (a) OS upgrades and reboots
+
+**`platform/kured/`** — the Kubernetes Reboot Daemon, chart `6.1.0` / app `1.23.0`, in its own Flux
+Kustomization (`clusters/base/kured.yaml`, `dependsOn: [platform]`, `wait: true`), registered in
+`clusters/base/kustomization.yaml`. Not a line in `platform/kustomization.yaml`: that tier is
+`wait: true` and gates `platform-config` → every apps wave, so a wedged reboot daemon placed there
+could stall every application in the cluster. A component added to make maintenance safer must not
+be able to cause an outage.
+
+**`infra/ansible/roles/unattended_upgrades/`** + `playbooks/unattended-upgrades.yml`, enabled for
+prod in `inventory/group_vars/k3s_prod/vars.yml`. A **new** role rather than an extension of
+`node_baseline` — different lifecycle, and other open branches touch that role.
+
+**The design, and how it answers this repo's two standing objections to reboot automation.**
+`roles/node_baseline/tasks/main.yml:61-67` says "IT DOES NOT REBOOT" (k3s-prod-2 hung on a warm
+reboot 2026-09-03; physical power button required), and `docs/plans/cluster-upgrade-plan.md:291-299`
+rejected system-upgrade-controller for "automating away the human gate between nodes". Both still
+stand. **The human gate is the sentinel.** unattended-upgrades is security-only with the kernel
+blacklisted, so it essentially never produces `/var/run/reboot-required`; that file appears when an
+operator deliberately runs `node-baseline.yml --tags kernel`. kured then performs only the
+*mechanical* half the operator would otherwise type by hand — including the ordering rule
+`node_baseline`'s own output has to shout ("NEVER node 2 first"), which the operator keeps by
+running the kernel play with `--limit <host>` one node at a time. `cluster-upgrade-plan.md:329-330`
+names the condition under which its own rejection reverses (fast disks), and prod is that cluster.
+
+**Quorum: concurrent reboots are structurally impossible.** All three nodes are control-plane+etcd
+(quorum 2/3). `concurrency: 1` plus kured's single cluster-wide lock annotation on its own DaemonSet
+means a node reboots only while holding that lock, enforced by API-server optimistic concurrency
+rather than by timing. **`lockTtl: 0`** (expiry disabled) is the deliberate choice: a TTL exists so
+reboots continue after the lock holder dies, which on 3 etcd nodes is exactly the catastrophe — a
+hung node's lock expiring would let a second node take the cluster to 1/3. So a hang stops the cycle
+forever and pages a human. `lockReleaseDelay: 30m` holds the lock after a node returns.
+
+**The Longhorn gate.** Measured: 25 volumes, all `numberOfReplicas: 3`, one replica per node, with
+`replica-soft-anti-affinity: false` and `disable-scheduling-on-cordoned-node: true`. So every drain
+takes **all 25 volumes to degraded 2/3**, and recovery is serialized —
+`replica-replenishment-wait-interval: 600`, `concurrent-replica-rebuild-per-node-limit: 1`. Reboot
+the next node too early and volumes reach 1/3, where `node-drain-policy:
+block-if-contains-last-replica` can block a drain forever. The inter-node gate is therefore **volume
+robustness, not pod readiness**, and it is doubled: `lockReleaseDelay: 30m` (unconditional, covers
+the 600s wait) plus kured's Prometheus alert gate (semantic, covers a rebuild longer than 30m).
+
+**`alertFilterMatchOnly: true` is load-bearing.** The flag inverts the regexp's meaning: `false`
+(chart default) makes it an *ignore* list and kured blocks on **any** firing alert. Measured
+2026-10-17, five alerts were firing with no end in sight (`CNPGPrimariesCoLocated`,
+`KubeDeploymentReplicasMismatch`, `KubeDeploymentRolloutStuck`, `KubePodNotReady`,
+`OAuth2ProxyGateUnobserved` — the last three all tracing to `pm-agent`, ImagePullBackOff on a
+placeholder digest for 12 days). Under the default, kured would have been blocked on day one and
+never rebooted anything, indistinguishable from "no reboots needed". The allow-list is anchored:
+`^(LonghornVolumeDegraded|LonghornVolumeFaulted|KubeNodeNotReady|NodeExporterDown)$`.
+`NodeEtcdRequestsSlow` and the `NodeIOPressure*` family are **deliberately excluded** despite being
+exactly what you'd want to pause for — prod-1 has a documented etcd WAL I/O starvation history and
+including them risks a permanent wedge that looks like safety.
+
+**The kernel floor (the highest-risk item), handled with two independent exclusions.** Measured: all
+three nodes run `7.1.8+deb13-amd64` (`7.1.8-1~bpo13+1`, the backports metapackage), far above the
+6.13 memattr floor — **but `linux-image-6.12.107` and `linux-image-6.12.94` are also still
+installed, below the floor, bootable, and `apt-mark showhold` is empty on every node.** So: (1) the
+allowed-origins list is `label=Debian-Security` only, and the running kernel comes from
+`o=Debian Backports` (pin 100, the `NotAutomatic`+`ButAutomaticUpgrades` signature), so u-u cannot
+*see* a kernel upgrade; (2) `^linux-` is blacklisted anyway, so widening origins later — the most
+likely future edit — cannot re-arm the hazard. The role **asserts both against `apt-config dump`**,
+apt's own parse of the whole directory, rather than trusting the file it just wrote. Also blacklisted:
+`intel-microcode`, `amd64-microcode`, `grub-*`. `Remove-Unused-Kernel-Packages` is **false** against
+the Debian default, because those older kernels are the documented rollback path. None of this stops
+the kernel moving — the operator's deliberate Ansible run still does that, unchanged.
+
+**The silent-no-op trap this closes.** Measured: `update-notifier-common`, `needrestart` and
+`unattended-upgrades` were **absent on all three nodes**, so *nothing could create
+`/var/run/reboot-required`*. kured deployed alone would have been permanently idle while 1/1 Running
+and reporting `kured_reboot_required=0` — green forever. The role installs the sentinel writer and
+**asserts a hook exists** by discovery (not a hard-coded path), failing loudly if not.
+`needrestart` runs in auto mode with `k3s.service` blacklisted — auto-restart is what makes a
+security-only policy effective, but restarting the control plane unattended is worse than a delayed
+library fix. (k3s is not an apt package at all here: no `k3s`, no `containerd` in dpkg.)
+
+**Alerts + runbook.** `platform/kured/alerts.yaml` — three rules over kured's *only* metric
+(`kured_reboot_required`, one GaugeVec, read from `cmd/kured/main.go:92-96`) plus `kube_node_info`:
+`KuredNotReporting` (`absent()` — the pipeline is dead and the other two are inert),
+`KuredNodeCoverageGap` (`count(kured) < count(nodes)` — a node silently uncovered, which `absent()`
+cannot see), `KuredRebootPending` (`for: 36h` — a reboot was offered and blocked). Companion
+promtool suite `alerts.promtool-test.yaml` pins all three both ways, including that the rule
+**stops** when a reboot succeeds. `runbooks/node-reboot-automation.md` registered in
+`runbooks/kustomization.yaml`.
+
+**The window is DAILY 03:00-06:00, and that choice is driven by the alert.** A weekly window means a
+pending reboot legitimately waits 6 days, forcing `for: 8d`+ — longer than this cluster's 7d
+Prometheus retention and reset by any restart past the 1h for-outage-tolerance. Daily caps the
+legitimate wait under 24h and makes `for: 36h` both robust and meaningful.
+
+### (b) Image / dependency automation
+
+**Renovate was aspirational. It now has a runner.** `.gitea/workflows/renovate.yaml` — scheduled
+daily 03:15 UTC, `workflow_dispatch` with `dry_run` defaulting to **true** for a safe first run.
+Preflight **fails hard (exit 1)** if `RENOVATE_TOKEN` is missing rather than skipping, because a
+skipped Gitea job reports `success` and that is precisely how `renovate.json` stayed inert.
+
+**`renovate.json`: Renovate now owns both halves, and Flux image-automation is formally closed.**
+ADR-0009 gave container image tags to Flux image-automation; that cannot be delivered on this
+repo's terms. (1) `ImageUpdateAutomation` **pushes commits, it does not open PRs**, so it bypasses
+the CLAUDE.md §1 gate; pointed at a side branch instead, nothing opens the PR. (2) It cannot work
+here anyway: `main` is protected `user_can_push: false` and the `flux-system` GitRepository has **no
+`spec.secretRef`** — the source is anonymous and read-only. (3) Enabling it means an in-cluster git
+**write** credential for the repo that defines the cluster. So `docker`/`kubernetes` — disabled
+*only* to avoid fighting that controller — are enabled, with `kubernetes.managerFilePatterns`
+covering `platform/`, `platform-config/`, `apps/`, **`catalog/`** (apps are generated from it; a bump
+only to the rendered output is reverted by the next render and fails `ci/render-golden.sh`),
+`runbooks/serving/` and `infra/agent/`. Added a no-automerge `blast-radius` rule for `kured`,
+`longhorn`, `cilium`, `cloudnative-pg`, `kube-prometheus-stack`, because the gate is static and does
+not boot a node or form a quorum. `platform/kyverno/tests/**` added to `ignorePaths`.
+
+### Sensitive paths — this PR needs a non-author approving review
+
+It touches **`clusters/**`**, **`conformance/**`** and **`.gitea/**`**, all in
+`infra/gitea/approval-policy.json`. It will **not** merge on a green gate alone. Per CLAUDE.md §1 I
+have **not** posted `/approve-override` and must not — that is the self-approval the stage exists to
+prevent.
+
+**The `conformance/` edit, justified.** `+34/-0` in
+`alert-metrics-have-scrape-targets/check.sh`: one new `SCRAPE_OWNERS` entry claiming `kured_`. It
+**weakens nothing** — no existing owner's evaluation changes — and it *tightens* the check for my
+own component, adding two `require` patterns (`service.create: true`, `metrics.create: true`; the
+chart defaults **both** to false) plus a `companion` requirement that `networkpolicy.yaml` stay in
+the kustomization. It had to be touched: assertion (3) fails any alert over an unclaimed metric, and
+the escape hatch `KNOWN_UNWIRED_METRICS` is **shrink-only**, so declaring `kured_*` as debt was not
+permitted and would have been wrong anyway. The alternative was shipping reboot automation with no
+alerting, which for a component whose every failure mode is silent is the worse choice. The entry
+carries a dated comment saying all of this.
+
+## What I verified
+
+Measured against the live cluster and nodes, read-only, on 2026-10-17.
+
+- **Flux image-automation controllers ARE installed and idle.** `-n flux-system get deploy` shows
+  `image-automation-controller` and `image-reflector-controller` both `1/1`, age **43d**; all three
+  `image.toolkit.fluxcd.io` CRDs exist. And **zero** `ImageRepository`/`ImagePolicy`/
+  `ImageUpdateAutomation` objects — "No resources found" for each. The controllers have run for 43
+  days with nothing to do. `platform/kustomization.yaml:52` has the component commented out.
+- **Renovate is aspirational, confirmed three ways**: no workflow in `.gitea/` or `ci/` (grep),
+  no Renovate CronJob/Deployment/Job in any namespace, and `renovate.json` itself disabled the
+  `docker`+`kubernetes` managers — so every plain `image:` tag was maintained by **nothing**, while
+  **seven files** claim "Renovate maintains this pin".
+- **`flux-system` GitRepository has no `secretRef`** (empty), URL is the in-cluster Gitea, and its
+  artifact revision was `main@sha1:1a0b208…`, matching my base.
+- **Nodes**: 3× Debian 13 trixie, all `control-plane,etcd`, k3s `v1.36.4+k3s1`, kernel
+  `7.1.8+deb13-amd64`. `apt-mark showhold` **empty**; `/etc/apt/preferences.d/` **empty**;
+  `/var/run/reboot-required` absent; `unattended-upgrades`/`update-notifier-common`/`needrestart`
+  all **absent**; `linux-image-6.12.107` and `6.12.94` installed alongside the backports kernel;
+  no `k3s`/`containerd` apt package; `apt-cache policy` origins and pins as quoted above.
+- **No node taints** on any of the three (so no tolerations needed), and `system-node-critical`
+  exists.
+- **Longhorn**: 1.12.1; `node-drain-policy: block-if-contains-last-replica`;
+  `replica-soft-anti-affinity: false`; `disable-scheduling-on-cordoned-node: true`;
+  `replica-replenishment-wait-interval: 600`; `concurrent-replica-rebuild-per-node-limit: 1`;
+  25 volumes all 3-replica/healthy; 25 replicas per node.
+- **6 PDBs with 0 allowed disruptions**: 3× `longhorn-system/instance-manager-*` (structurally 0 —
+  single-node selector, `minAvailable: 1`) and 3× CNPG `*-primary`. All three Postgres primaries
+  were on `k3s-prod-1`.
+- **kured drain flags do not exist.** Read `cmd/kured/main.go:438-449`: `Force`,
+  `DeleteEmptyDirData` and `IgnoreAllDaemonSets` are **hardcoded true**. Passing
+  `--ignore-daemonsets`/`--delete-emptydir-data`/`--force` as extraArgs would CrashLoop the
+  DaemonSet. All three are genuinely needed (12 DaemonSets, 135 emptyDir pods measured).
+- **kured exposes exactly one metric** — the only `MustRegister` in the project.
+- **Prometheus selectors are all `{}`** (`serviceMonitorSelector`, `serviceMonitorNamespaceSelector`,
+  `ruleSelector`, `ruleNamespaceSelector`), so a `PrometheusRule`/`ServiceMonitor` in a new namespace
+  is picked up with no label plumbing.
+- **`count(kube_node_info) == 3`** and `count(longhorn_volume_robustness) == 100`, queried through a
+  read-only port-forward. The 5 chronically-firing alerts were read from `/api/v1/alerts`.
+- **`LonghornVolumeDegraded`** is `longhorn_volume_robustness{state="degraded"} == 1`, `for: 10m` —
+  shorter than Longhorn's own 600s replenishment wait, so degradation reliably outlasts the latch.
+- **kured chart 6.1.0 → appVersion 1.23.0** is the newest of 80 entries in the chart index.
+- **Kyverno/PSA will not block it**: every policy is scoped
+  `app-namespace: "true"`; the namespace sets it explicitly `"false"`, and
+  `restrict-host-namespaces.yaml`'s own header states that carve-out. No cluster-wide PSA baseline.
+- **`renovate.json` parses** as valid JSON after my edits (`json.load`).
+- **kured is not installed** and the string `kured` appears nowhere in the repo — greenfield.
+
+### CORRECTION 2026-10-18 — the first real run FAILED, and the sentinel design was wrong
+
+`ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod` failed on all three
+nodes at the package task:
+
+```
+fatal: [k3s-prod-1]: FAILED! => "No package matching 'update-notifier-common' is available"
+```
+
+**`update-notifier-common` is an UBUNTU package and does not exist in Debian.** Measured on trixie:
+`apt-cache policy update-notifier-common` → `Candidate: (none)`. The original entry asserted this
+package "owns the hook that creates /var/run/reboot-required" — true on Ubuntu, false here, and the
+role could never have worked. This was the exact risk the first version of this entry flagged ("the
+`find` patterns are a guess at which package ships what") landing in a worse place than predicted:
+not a wrong glob, a nonexistent package.
+
+Three further measurements settled what to do instead:
+
+* `/usr/share/update-notifier/notify-reboot-required` is **absent**.
+* **No `linux-image-*` postinst references `notify-reboot-required`** — invoking it is an Ubuntu
+  patch, so merely placing a script at that path would never be called.
+* No hook under `/etc/apt/apt.conf.d` or `/etc/kernel/` mentions `reboot-required`.
+
+So Debian trixie has **no stock mechanism at all**, and the role now ships the writer itself:
+`/usr/local/sbin/k3s-notify-reboot-required` (compares the running kernel against the newest
+installed via `linux-version`, from `linux-base`, which every `linux-image` depends on — measured
+`linux-base 4.12.1` present) plus a `DPkg::Post-Invoke` hook at
+`/etc/apt/apt.conf.d/99-k3s-reboot-required`. `/run` is a tmpfs (measured), so the sentinel
+self-clears on boot and needs no cleanup task.
+
+The verify assert changed shape too: it was a `find` glob over `/etc/kernel/postinst.d` and
+`/etc/apt/apt.conf.d` for `*update-notifier*` / `*unattended-upgrades*`. That glob was **weaker
+than it looked** — it would have matched Debian's own `unattended-upgrades` apt conffile and
+reported "sentinel writer present" while no writer existed. It now stats the two exact paths this
+role owns and requires the script to be executable.
+
+**Newly verified (this correction):** the sentinel script was executed on prod-1 as the
+unprivileged `k3s` user under `sh -x`. It resolved `running=7.1.8+deb13-amd64`,
+`newest=7.1.8+deb13-amd64`, took the equal branch, exited 0 and wrote nothing — correct, since the
+running kernel *is* the newest. `sh -n` and `bash -n` clean.
+
+**Still NOT verified after this correction:** the *writing* half. Because running == newest on all
+three nodes, the `touch` path has never executed, so I have not seen the sentinel actually created,
+nor seen kured react to it. The honest test is the operator's next kernel upgrade. Also unverified:
+whether a `DPkg::Post-Invoke` hook fires in the `unattended-upgrades` code path specifically (it
+runs dpkg, so it should, but that is inference) — and whether `linux-version sort --reverse` orders
+a mixed backports/stable kernel set the way I assume when a 6.12 and a 7.1 are both installed. It
+produced the right answer with the current set; a future set could differ.
+
+## What I did NOT verify
+
+The honest list. Most of this needs an actual upgrade cycle, which I did not and could not run.
+
+- **I never ran an OS upgrade, never rebooted a node, and never watched kured take a lock.** No part
+  of the runtime behaviour of either half has been observed. `sudo` needs a password I do not have,
+  so I could not run `apt`, `unattended-upgrade --dry-run`, or install anything. **Everything about
+  (a) working is inference from configuration and measured cluster state, not observation.**
+- **The Ansible role has never been executed.** Not syntax-checked by `ansible-playbook --syntax-check`
+  either (no inventory access from the gate). Its asserts, its `apt-config dump` parsing, and the
+  sentinel-hook discovery are all unproven. The `find` patterns for the sentinel hook are a *guess at
+  which package ships what* — I could not read `update-notifier-common`'s file list (packages.debian.org
+  returned no such package for that suite/arch), which is exactly why the role discovers rather than
+  hard-codes. **If no hook matches, the role fails loudly — but it may fail on a correct node.**
+- **Whether CNPG actually switches the primary in response to a drain-induced eviction.** All three
+  `*-primary` PDBs are structurally 0-disruption and `.spec.nodeMaintenanceWindow` is unset on all
+  three clusters. If the operator does not switch over, draining `k3s-prod-1` blocks on three PDBs at
+  once and hits `drainTimeout`. **This is the most likely way the first real reboot fails.** It
+  should be tested with one manual, watched `kubectl drain` before trusting the timer.
+- **Whether `drainTimeout: 15m` is long enough.** Unmeasured. The `unifi` VMI has
+  `evictionStrategy: LiveMigrate` with `migrationMethod: BlockMigration`, i.e. a full root-disk copy
+  over the network on every `k3s-prod-1` drain. I do not know how long that takes.
+- **Whether the alert gate actually blocks as intended.** I never saw kured query Prometheus. The
+  assumption that it treats a query error as "do not proceed" is from its documented behaviour, not
+  observed. The NetworkPolicy allowing kured→Prometheus:9090 and Prometheus→kured:8080 is
+  **unproven**; if either is wrong the symptom is silence, which is why `KuredNotReporting` exists —
+  but that alert is itself unproven against a live scrape.
+- **The alert expressions have not run against real `kured_*` data**, because the metric does not
+  exist in this cluster yet. They pass promtool unit tests against synthetic series whose shape I
+  read from kured's source. `absent()` and the `count() < count()` comparison are the two most
+  likely to behave differently against real label sets.
+- **`enabledManagers: ["regex"]` may be the wrong name.** Modern Renovate expects `custom.regex` for
+  `customManagers`. I left the pre-existing value alone rather than guess, but **the inline
+  `# renovate:` markers may simply not run.** Unverifiable without executing Renovate. Two markers
+  are additionally known-broken independent of this: `external-snapshotter`'s sits two comment lines
+  above its `newTag`, and Keycloak's `kustomization.yaml` has no marker at all despite a comment
+  claiming Renovate maintains it.
+- **Renovate has still never run.** I added the runner; `RENOVATE_TOKEN` does not exist yet, so the
+  first scheduled run **will fail** (by design). Nothing about the config is proven correct until a
+  `dry_run: true` dispatch is read by a human. Expect it to want to open ~30 PRs.
+- **No alert for "automatic patching silently stopped".** u-u's success lives in
+  `/var/log/unattended-upgrades/` on each node and nothing reads it. Likewise **no alert for the
+  Renovate schedule ceasing to fire**. Both are real gaps I did not close.
+- **`coredns` is 1 replica and I did not fix it.** It is k3s-managed, out of this PR's scope, and it
+  means every `k3s-prod-1` reboot causes a cluster-wide in-cluster DNS gap. Recommended follow-up.
+- **kured cannot honour "NEVER node 2 first".** It has no ordering control. The mitigation — run the
+  kernel play with `--limit` one host at a time — is a *procedure*, documented in the runbook, not an
+  enforced property. If the operator flags all three nodes at once, kured picks the order.
+- **kured cannot press a power button.** The 2026-09-03 warm-reboot hang is unmitigated. Bounded, not
+  prevented: quorum holds at 2/3, the lock is held forever so nothing cascades, and
+  `KuredRebootPending` eventually fires.
+- **Upstream version claims are a snapshot, partly from search results**, not all from authoritative
+  release APIs. I could not determine latest for versitygw, crafty, nginx-unprivileged, act_runner,
+  `virt-template-*`, the linuxserver wireguard date-tag, or the exact latest PostgreSQL 16 patch
+  (sources conflicted 16.10 vs 16.15). **The gitea `1.24.3 → v28.1.0` gap is almost certainly an
+  upstream versioning-scheme change, not 26 majors — do not act on that framing.**
+- **I did not verify Renovate's `kustomize` manager still processes `images:`/`newTag`** now that
+  `docker` is enabled, which decides whether `external-snapshotter` and `intel-gpu-plugin` are
+  managed at all.
+- **I did not raise any HelmRelease version ceiling**, so the 9 minor-locking ranges still cap
+  Renovate at patches. Bumping `kube-prometheus-stack` in particular will fail the gate until
+  `ci/versions.env`'s exact promtool/Prometheus `3.14.0` lockstep moves with it.
+- **L2/L3 conformance did not run** (`vars.CLUSTER_AVAILABLE` unset), so nothing here is proven
+  against a cluster by CI either.

decisions/2026-10-17-oauth2-proxy-oidc-startup-race.md

@@ -0,0 +1,99 @@
+---
+date: 2026-10-17
+branch: fix/oauth2-proxy-oidc-startup-race
+title: "oauth2-proxy dies on a cold start because OIDC discovery runs once, before Keycloak serves"
+---
+
+## What changed
+
+Exposed by the 2026-10-17 whole-cluster shutdown for the UPS move. oauth2-proxy performs OIDC
+discovery **once, at startup**, and `os.Exit(1)`s if it fails. On a cold boot Keycloak is not
+serving yet, so the shared Gateway answers 503 and the proxy dies:
+
+```
+[provider.go:55] Performing OIDC Discovery...
+[main.go:73] ERROR: Failed to initialise OAuth2 Proxy: error initialising provider: ...
+             failed to discover OIDC configuration: unexpected status "503": no healthy upstream
+```
+
+It then sits in **exponential CrashLoopBackOff long after Keycloak is healthy** — one of the two
+replicas was still down minutes later and only a manual `kubectl delete pod` cleared it. There is no
+retry flag; the process does not survive a cold start. A readinessProbe cannot help (the container
+is already dead), and CrashLoopBackOff backoff is not tunable per-pod.
+
+`platform/oauth2-proxy/helmrelease.yaml` gains an `extraInitContainers` gate (`wait-for-idp`) that
+polls Keycloak's discovery doc until it answers, then lets the proxy start. A pod in `Init` does not
+accrue CrashLoopBackOff backoff and retries promptly, which is the point.
+
+**Bounded, and fails open.** After ~5 minutes it gives up and starts the proxy anyway, so the worst
+case is exactly today's behaviour and never something new. An unbounded wait would trade a CrashLoop
+for a pod wedged in `Init` forever, which is not an improvement.
+
+## Why this shape and not the alternatives
+
+* **`extraInitContainers` is the chart's own documented hook for precisely this.** Its commented
+  example in `values.yaml` is literally `- name: wait-for-idp`. The key was **confirmed against the
+  pinned chart** (`helm show values oauth2-proxy/oauth2-proxy --version 10.7.1` → `extraInitContainers: []`
+  at line 428), not assumed — a wrong Helm values key is silently ignored, which is the
+  "reviewed, merged, apparently done, still broken" shape this repo keeps getting bitten by.
+* **No new base image.** The chart already runs an `alpine` init container for its built-in
+  `initContainers.waitForRedis` gate, so alpine is a base this chart uses regardless. Pinned to
+  `3.22` (verified to exist upstream) rather than the chart's own `latest`.
+* **Flux `postRenderers` rejected:** it would work independently of the chart schema, but nothing in
+  this tree uses postRenderers, so it would introduce a mechanism with no precedent for a problem
+  the chart already has a hook for.
+* **`--skip-oidc-discovery` rejected:** it removes the startup fetch entirely, which is arguably
+  more elegant, but it means hardcoding four Keycloak endpoints and I could not verify offline that
+  oauth2-proxy then fetches JWKS lazily rather than at startup. oauth2-proxy gates SSO for PiHole,
+  the \*arr apps, qBittorrent, Homepage, Prometheus, Alertmanager and Hubble — a wrong guess locks
+  the operator out of all of them, so the reversible, additive option wins.
+
+The `$$`-free convention matters here: shell variables are written bare (`$i`, `$url`, `$(( ))`)
+with **no braced form anywhere**, because `platform` declares `postBuild.substituteFrom cluster-vars`
+and envsubst consumes braced expansions — a braced shell variable renders to the empty string and
+silently breaks the loop. `${BASE_DOMAIN}` is braced on purpose. Same hazard and same convention as
+the gitea and nextcloud OIDC bootstrap hooks, and `ci: flux-postbuild-substitution` enforces it.
+
+## What I verified
+
+* **The failure is measured, not theorised.** The log above is from the live pod
+  (`oauth2-proxy-5d6c74876b-bzb2p`, `lastState.terminated: {reason: Error, exitCode: 1}`), captured
+  with `--previous` after the cold boot. The surviving replica's log shows the same startup
+  succeeding once Keycloak was up, which is what identifies this as a race rather than a config
+  fault.
+* **Deleting the pod is a sufficient workaround** — done on prod 2026-10-17, the replacement came up
+  clean and both replicas have been healthy since.
+* **The chart key is real**, at the pinned version range: `helm search repo` resolves
+  `>=10.0.0 <11.0.0` to chart `10.7.1` (app `7.15.5`), whose values carry `extraInitContainers: []`,
+  `extraContainers: []` and the built-in `initContainers.waitForRedis` (which uses `alpine`).
+* **`docker.io/library/alpine:3.22` exists** — confirmed against the registry tag list rather than
+  assumed from memory.
+* The edited HelmRelease parses as YAML.
+
+## What I did NOT verify
+
+* **UNAPPLIED. The init container has never run.** I did not reconcile the HelmRelease, so I have not
+  seen `wait-for-idp` execute, exit 0, or gate the proxy. Applying it rolls both oauth2-proxy
+  replicas, which is a mutating action (CLAUDE.md §5) — and it rolls the component that gates SSO for
+  seven apps, so **do it when you can watch it**, not unattended.
+* **I did not re-create the cold-start race to prove the fix.** That needs Keycloak down while
+  oauth2-proxy starts; the honest test is the next whole-cluster boot. Until then this is reasoned
+  from the measured failure, not demonstrated against it.
+* **busybox `wget`'s TLS behaviour here is assumed, not tested.** The claim is that alpine's
+  `ca-certificates-bundle` lets it validate the Gateway's LE-prod certificate, so a genuine TLS
+  failure exits non-zero rather than passing falsely. If that is wrong in either direction the gate
+  degrades safely — a never-succeeding probe just burns the 5-minute budget and then starts the proxy
+  anyway — but the fast path would be silently dead and this would be doing nothing. **Check the
+  init container's log on first apply**: it should print "Keycloak discovery reachable" within
+  seconds on a warm cluster, not the give-up message.
+* **Whether a pod can reach the Gateway hostname at all from this namespace is inherited, not
+  re-proven.** The main container does exactly this fetch today and succeeds, so the path is known
+  good — but note the standing trap that an arbitrary pod gets 403 from every Gateway hostname
+  (`decisions/2026-10-04-plex-advertise-plexdirect.md`); oauth2-proxy is exempt only because
+  Keycloak's ingress policy admits it. An init container in the same pod inherits that identity, so
+  this should hold — unverified.
+* **60 attempts x 5s is not tuned.** It is "comfortably longer than Keycloak's observed cold start",
+  not a measured bound.
+* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline
+  subset only.
+* I did not re-measure branch protection or runner state (CLAUDE.md §2b).

docs/BULK-NFS-STORAGE.md

@@ -1,61 +1,43 @@
 # Bulk NFS storage — the media library and the legacy Nextcloud archive
 
-The operator-facing companion to the `class: bulk` storage entries in `catalog/{plex,sonarr,radarr,nextcloud}.yaml`.
-Everything here was **measured** on the storage host on 2026-09-07, over read-only SSH; nothing is
-assumed. It resolves the long-standing **OQ-2** stub (`platform/longhorn/storageclasses.yaml`) for the
+The operator-facing companion to the `class: bulk` storage entries in
+`catalog/{plex,sonarr,radarr,qbittorrent,nextcloud}.yaml`.
+
+**Rewritten 2026-10-07 for the move off Proxmox onto a standalone TrueNAS at 192.168.2.30.** The
+layout (§2), the mount design (§3, §4) and the uid model (§6) are unchanged by the move and were
+measured on the old host on 2026-09-07. Everything about the storage host itself — §1, §5, §10 —
+was re-measured on the NAS on 2026-10-07. Nothing here is assumed. It resolves the long-standing
+**OQ-2** stub (`platform/longhorn/storageclasses.yaml`) for the
 case that actually exists here — attaching datasets that already hold data — and leaves dynamic bulk
 provisioning unimplemented, because nothing needs it.
 
 ---
 
-## 1. The storage host, and why prod cannot reach it
+## 1. The storage host
 
-The bulk data lives on the **Proxmox host's ZFS pool**. That host has exactly one interface:
+The bulk data lives on a **standalone TrueNAS** (Community Edition) at **192.168.2.30** — the same
+`192.168.2.0/24` as prod's nodes (`192.168.2.10/.11/.12`), so it is reached directly, with no NAT
+and no routing decision.
 
-```
-vmbr0  192.168.1.9/24        default via 192.168.1.1   (pfSense LAN)
-```
-
-| | nodes | reaches 192.168.1.9? |
-|---|---|---|
-| **dev** | 192.168.1.40 / .41 / .42 | **Yes** — same L2 segment, and it already holds an NFS export from this host (`/mnt/Storage/k8s-volumes`, in `/etc/exports`) |
-| **prod** | 192.168.2.10 / .11 / .12 | **No** — 192.168.2.0/24 is pfSense's **WAN**. Ports 22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. 2049 and 111 do not. |
-
-Opening the storage path for prod is a **pending operator decision**. Its two candidate outcomes give
-different values for the `BULK_NFS_SERVER` cluster-var:
-
-| outcome | `BULK_NFS_SERVER` for prod |
-|---|---|
-| port-forward `192.168.2.102:2049` → `192.168.1.9:2049` | `192.168.2.102` |
-| a WAN→LAN route to 192.168.1.0/24 | `192.168.1.9` |
-
-Until it lands, `clusters/prod` holds `TBD-PROD-BULK-NFS-SERVER`. **See §7 for why that placeholder
-behaves differently from the repo's other ones.**
+**This section used to be the hard part of this page.** The datasets were on the Proxmox host, whose
+only interface was `vmbr0 192.168.1.9/24` on pfSense's **LAN**, while prod sits on pfSense's **WAN**.
+tcp/2049 did not traverse that, so prod mounted through a DNAT at `192.168.2.102`, and which address
+belonged in `BULK_NFS_SERVER` depended on an undecided operator choice. Both pools were physically
+moved on 2026-10-07 (`docs/plans/proxmox-to-n5-air-migration.md`, Phase C/D) and all of that is
+simply gone. Measured 2026-10-07: **192.168.2.102:2049 no longer answers; 192.168.2.30:2049 and
+:111 do.**
 
 ### It only needs ONE port
 
 The PVs mount `nfsvers=4.1`. **NFSv4 carries mount and lock in-protocol**, so the only thing that
 must be reachable is **tcp/2049** — no rpcbind/111, no mountd, no statd. NFSv3 would have needed
-four ports, three of them on ephemeral numbers, which is a materially worse firewall story. The
-server advertises `+3 +4 +4.1 +4.2` (`/proc/fs/nfsd/versions`), so v4.1 is available and is the
-conservative floor.
+four ports, three of them on ephemeral numbers. The NAS advertises `+3 +4 +4.1 +4.2`
+(`/proc/fs/nfsd/versions`, measured 2026-10-07), so 4.1 is available and is the conservative floor.
 
-### The export's client list is the SAME under both outcomes — measured, not assumed
-
-**pfSense DNATs the existing port-forward WITHOUT source NAT.** Connecting from this workstation
-(192.168.2.132) to `192.168.2.102:22`, the storage host logged the peer as `192.168.2.132`:
-
-```
-SSH_CONNECTION=192.168.2.132 39768 192.168.1.9 22
-ss -tn  ->  192.168.1.9:22   192.168.2.132:39768
-```
-
-The original WAN-side source address survives the translation (return traffic finds its way back via
-the host's default route, 192.168.1.1). So under **either** outcome the NFS server will see prod's
-**real** node addresses — 192.168.2.10/.11/.12 — and the `sharenfs` change in §5 is the same either
-way. **It does not have to wait for the routing decision.**
-
----
+**NFSv4 is not on by default and its absence does not look like a storage problem.** TrueNAS ships
+NFSv3-only until NFSv4 is added to *Services → NFS → Enabled Protocols*. Without it the mount never
+completes and the pods sit in `ContainerCreating` — no permission error, nothing in the app logs,
+just no volume.
 
 ## 2. The layout, as found
 
@@ -107,16 +89,18 @@ why it is mounted read-only; see §6.
 | plex | `/mnt/Storage/plex` | `/media` | — | **ro** | ReadOnlyMany |
 | sonarr | `/mnt/Storage/plex` | `/media` | — | rw | ReadWriteMany |
 | radarr | `/mnt/Storage/plex` | `/media` | — | rw | ReadWriteMany |
+| qbittorrent | `/mnt/Storage/plex` | `/downloads` | `downloads` | rw | ReadWriteMany |
 | nextcloud | `/mnt/Storage/nextcloud` | `/mnt/legacy-nextcloud` | `nextcloud` | **ro** | ReadOnlyMany |
 
 In-app paths that follow: Plex libraries at `/media/movies`, `/media/tv`, `/media/photos`; Sonarr
 Root Folder `/media/tv`; Radarr Root Folder `/media/movies`; both importing from
 `/media/downloads/complete/{tv-sonarr,radarr}`.
 
-**All three media consumers use the identical mountPath on purpose.** When qbittorrent is eventually
-given the same export (it is not in this change), the arrs and the download client will already agree
-on absolute paths, which is the second precondition for hardlinked imports — otherwise every import
-needs a Remote Path Mapping and quietly falls back to copying when one is missing.
+**The three library consumers use the identical mountPath on purpose**, and qbittorrent has since
+been given the same export (narrowed by `subPath: downloads`, presented at `/downloads` to match its
+existing `Session\DefaultSavePath`). The arrs and the download client therefore agree on absolute
+paths, which is the second precondition for hardlinked imports — otherwise every import needs a
+Remote Path Mapping and quietly falls back to copying when one is missing.
 
 **`/media` and not `/data`, and that is not cosmetic.** The arrs' Longhorn config PVC renders at the
 renderer default `/data/config`. A media mount at `/data` would be mounted **over** it and mask the
@@ -147,62 +131,123 @@ Narrowing that would cost hardlinks, which is the worse trade.
 
 ---
 
-## 5. The export change the operator must apply — STRICTLY ADDITIVE
+## 5. The exports, as configured on TrueNAS — and the settings that decide permissions
 
-**Do not apply this from the repo.** It is an operator step on the storage host, recorded here so the
-exact value is reviewable. The existing consumers **must keep working**: 192.168.1.6, 10.123.1.10 and
-192.168.1.18 on the media tree, and 192.168.1.6 and 192.168.1.5 on the Nextcloud dataset, are the
-live non-k3s prod stack.
+**Do not apply this from the repo.** It is operator configuration on the NAS, recorded here so the
+exact values are reviewable. Everything below was **measured on 192.168.2.30 on 2026-10-07**, after
+the move.
 
-Current values (`zfs get sharenfs Storage`):
+### The exports
 
 ```
-Storage/plex       rw=@192.168.1.6:@10.123.1.10:@192.168.1.18,no_subtree_check,no_root_squash
-Storage/nextcloud  [email protected]:@192.168.1.5,no_subtree_check,no_root_squash
+/mnt/Storage/plex       192.168.2.10 192.168.2.11 192.168.2.12
+                        (sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,root_squash,no_all_squash)
+/mnt/Storage/nextcloud  192.168.2.10 192.168.2.11 192.168.2.12
+                        (sync,wdelay,hide,no_subtree_check,sec=sys,ro,secure,root_squash,no_all_squash)
 ```
 
-### `Storage/plex` — append the six k3s node addresses to the existing `rw=` list
+Per-node addresses rather than `@192.168.2.0/24`, following the precedent of the old host. A `/24`
+would work and would survive adding a node; it also grants the whole subnet.
+
+**All three nodes must appear on BOTH shares.** Any media pod can be scheduled on any node, so a
+missing node is a latent failure, not an immediate one: it surfaces the first time a pod reschedules
+there, as `access denied by server while mounting`, with the cause weeks stale. One was caught
+exactly this way during setup — the plex share listed `192.168.2.13`, an address no node holds, and
+so was missing `192.168.2.12` (k3s-prod-3). Nothing was failing at the time.
+
+### The option flags ARE the permission contract
+
+| flag | why it matters here |
+|---|---|
+| **`no_all_squash`** | The one the whole model rests on. Client uids pass through untouched, so the arrs' `PUID=1000` arrives as the owner of a `0755 1000:3000` tree. In the UI this is **Mapall User/Group left EMPTY** — setting either squashes every client to one identity and erases the reader/writer distinction entirely. |
+| **`sec=sys`** | AUTH_SYS. Numeric uid/gid, not Kerberos principals. §6 is only true because of this. |
+| **`root_squash`** | Maproot left empty. The Proxmox export carried `no_root_squash`; **nothing here needs it** — every write to the tree happens after the LSIO drop to uid 1000, and the images only chown their Longhorn `/config`. This is a deliberate tightening over the old export, not parity. |
+| **`rw` / `ro`** | The archive's read-only posture is **server-enforced**, as it was before. The media tree cannot get the same treatment — see below. |
+| **`secure`** | Reserved source ports required; "Allow non-root mount" off. The kubelet mounts as root from a reserved port and does not need the exception. |
+
+**Plex's read-only posture is still a CLIENT-side guarantee.** Plex is ro and the arrs are rw, but
+all four can land on any node, so the server sees the same three source addresses for all of them
+and cannot distinguish the reader from the writers. `spec.nfs.readOnly: true` on `plex-media` plus
+the volumeMount flag is the whole enforcement. Unchanged by the move; an honest limitation, not an
+oversight. If it ever needs to be server-enforced, the lever is a dedicated node pool for Plex.
+
+### Service settings that are NOT obvious from the export list
+
+- **Enabled Protocols must include NFSv4** (above).
+- **Require Kerberos for NFSv4: off.** The numeric AUTH_SYS pass-through is the model.
+- **Manage Groups Server-side: off.** This is `rpc.mountd --manage-gids`: the server *discards the
+  client's supplied group list* and looks that uid's groups up in TrueNAS's own user database
+  instead. With no local uid 1000 defined, the arrs' `PGID=3000` would simply vanish. Writes would
+  survive anyway — uid 1000 **owns** the tree, so the owner bits grant access without the group —
+  but it is a free way to lose group semantics.
+- **NFSv4 DNS Domain: blank.** It only feeds `idmapd.conf`, which numeric mapping bypasses.
+- **Bind IP 192.168.2.30** means loopback does not answer: check with `showmount -e 192.168.2.30`,
+  not `localhost`.
+
+There is **no** "NFSv3 ownership model for NFSv4" control on current TrueNAS — it was an older
+TrueNAS/CORE option, and no replacement is needed: on a Linux client against a Linux server over
+AUTH_SYS, both the `nfs` and `nfsd` modules default to `nfs4_disable_idmapping=Y`, so NFSv4 carries
+raw numeric ids rather than `user@domain` principals. §9's `ls -n` probe is how you confirm it.
+
+### The ZFS `sharenfs` properties must be CLEARED, not edited
+
+The pools arrived with the old host's exports still set as dataset properties:
+
+```
+Storage/plex              [email protected]:@10.123.1.10:@192.168.1.18:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash
+Storage/nextcloud         [email protected]:@192.168.1.5,[email protected]:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash
+Storage/docker-volumes    [email protected]:@192.168.1.5:@192.168.1.10:@192.168.1.18,no_subtree_check,no_root_squash
+```
+
+TrueNAS generates `/etc/exports` from its own middleware database and will fight a property it did
+not set — the failure mode is two differently-configured exports of one path, with no indication
+which is in force. Cleared 2026-10-07:
 
 ```sh
-zfs set sharenfs='[email protected]:@10.123.1.10:@192.168.1.18:@192.168.1.40:@192.168.1.41:@192.168.1.42:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash' Storage/plex
+zfs inherit sharenfs Storage/plex
+zfs inherit sharenfs Storage/nextcloud
+zfs inherit sharenfs Storage/docker-volumes
 ```
 
-Every pre-existing host is preserved verbatim and no option changes; only clients are added.
+`Storage` is `off`, so each resolves to `off`.
 
-### `Storage/nextcloud` — add the k3s nodes to a **`ro=`** list, not to `rw=`
+### Do NOT touch the dataset's permissions or ACLs
 
-```sh
-zfs set sharenfs='[email protected]:@192.168.1.5,[email protected]:@192.168.1.41:@192.168.1.42:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash' Storage/nextcloud
-```
+The tree's ownership **is** the access story (§6), and TrueNAS offers two ways to destroy it
+irreversibly across 8.33 T.
 
-This is the one place read-only can be enforced **server-side**, and it is worth taking: only
-Nextcloud consumes this dataset from k3s, so the export itself can refuse writes. Even a
-misconfigured client cannot then damage the archive.
+`Storage/plex` carries `acltype=nfsv4` (SCALE's built-in default, `SOURCE: default`) and
+`aclinherit=passthrough` (inherited). Measured 2026-10-07: the directories carry **trivial ACLs
+only** — `nfs4xdr_getfacl` shows just `owner@`/`group@`/`everyone@` matching the mode — so `0755
+1000:3000` governs exactly as it did under Proxmox.
 
-**The media tree cannot get the same treatment.** Plex is read-only and the arrs are read-write, but
-all three may be scheduled on any node, so the server sees the same source addresses for all of
-them and cannot distinguish the reader from the writers. **Plex's read-only posture is therefore a
-client-side guarantee** (`spec.nfs.readOnly: true` on `plex-media`, plus the volumeMount flag) — an
-honest limitation, not an oversight. If it ever needs to be server-enforced, the lever is a
-dedicated node pool for Plex, which is a much larger change.
+- **Stay out of the dataset's Edit Permissions / Edit ACL screen.** On an `acltype=nfsv4` dataset
+  that is a full ACL editor, and its presets write *real* NFSv4 ACLs, recursively if permitted.
+  There is no undo.
+- `aclinherit=passthrough` means a single ACL written at `/mnt/Storage/plex` would be **inherited by
+  everything created under it afterwards** — the blast radius is the whole future library, not one
+  directory.
+- Note GNU `ls`'s `+` marker is **not** a reliable ACL indicator here: it looks for a POSIX ACL
+  xattr, and this dataset stores NFSv4 ACLs elsewhere. Use `nfs4xdr_getfacl`.
 
-Per-node addresses rather than `@192.168.2.0/24` follows the precedent already on this host
-(`/etc/exports` lists the dev nodes individually, as `/32`). A `/24` would work and would survive
-adding a node; it also grants the whole subnet. Per-node is the tighter default.
+`xattr=on` rather than `sa` is a performance detail, not a permissions one, and changing it would
+not affect existing files.
 
 ### Verifying, without changing anything
 
 ```sh
-exportfs -v                  # the new client lines appear, existing ones unchanged
-zfs get sharenfs Storage/plex Storage/nextcloud
+cat /proc/fs/nfsd/versions          # +4.1 must be present
+exportfs -v                         # the two shares, three nodes each, flags as tabulated above
+showmount -e 192.168.2.30           # NOT localhost -- the service is bound to one address
+zfs get -r -o name,property,value,source sharenfs Storage    # every row off or -
 ```
 
----
-
 ## 6. uid / gid — the failure this design exists to avoid
 
-The export uses `sec=sys` and `no_root_squash`, so **the client's numeric uid is passed through
-verbatim** and the server's on-disk ownership is the entire access story. The media tree is
+The export uses `sec=sys` and `no_all_squash`, so **the client's numeric uid is passed through
+verbatim** and the server's on-disk ownership is the entire access story. (It no longer carries
+`no_root_squash`, which the Proxmox export did — see §5. Nothing here needs it, and the uid that
+matters is 1000, not 0.) The media tree is
 `0755 1000:3000` throughout (some directories `1000:1000`), so **only uid 1000 can write it**.
 
 **`fsGroup` cannot fix this, and reaching for it is how this gets misdiagnosed.** The in-tree NFS
@@ -233,24 +278,27 @@ Nextcloud needs no uid work at all, which is one more reason its mount is read-o
 
 ---
 
-## 7. The prod placeholder does NOT fail at apply time — read this before merging
+## 7. Why this knob is unvalidated at create AND immutable at update
+
+**The `TBD-PROD-BULK-NFS-SERVER` placeholder is gone** (2026-10-07) — prod now carries a real
+address, because the NAS is on prod's own subnet and there is no longer a decision to defer. The
+hazard that made it load-bearing is worth keeping on the page, because it is the half that explains
+why §10 exists.
 
 Every other `TBD-PROD-*` in `clusters/prod/kustomization.yaml` is rejected loudly by CRD validation
-(Cilium, LB-IPAM, CNPG, Velero) because it is not a valid IP or URL. **This one is not.**
+(Cilium, LB-IPAM, CNPG, Velero) because it is not a valid IP or URL. **This one was not.**
 
-A PersistentVolume's `spec.nfs.server` is an **unvalidated string**. The API server accepts
-`TBD-PROD-BULK-NFS-SERVER`, the PV is created, and the PVC **binds** to it — binding is pure
-control-plane and never touches the server. The failure appears only when a pod mounts:
+A PersistentVolume's `spec.nfs.server` is an **unvalidated string**. The API server accepted
+`TBD-PROD-BULK-NFS-SERVER`, the PV was created, and the PVC **bound** to it — binding is pure
+control-plane and never touches the server. The failure appeared only when a pod mounted:
 
 ```
 MountVolume.SetUp failed ... Failed to resolve server TBD-PROD-BULK-NFS-SERVER
 ```
 
-so **plex, sonarr, radarr and nextcloud would sit in ContainerCreating on prod.**
-
-**Consequently: set `BULK_NFS_SERVER` in `clusters/prod/kustomization.yaml` BEFORE prod reconciles a
-revision that carries these bulk volumes.** Dev renders a working value and is where these manifests
-are meant to be proven first.
+**The same field is immutable on UPDATE** (§10), which is the other half of the same awkwardness: a
+wrong value is accepted silently at create, and then cannot be corrected in place. Together they are
+why repointing this knob is a delete-and-recreate rather than a reconcile.
 
 ### The static guard this convention leans on is missing from `main` — but the gate now refuses the value anyway
 
@@ -287,10 +335,9 @@ Without the arm, HALF A is what holds the line, by design.
    EPHEMERAL layer"). Plex was fixed; sonarr, radarr, qbittorrent and jackett were not. Fixing it
    means those apps start from an empty config volume, so it belongs in its own change with the
    operator present. **This change does not make it worse** — that is why the media mount is `/media`.
-2. **qbittorrent has no bulk mount.** Today the arrs' import source is written by the **legacy
-   non-k3s** download client, which is coherent while the legacy stack stays live. Giving the k3s
-   qbittorrent the same export at the same `/media` path is the natural follow-up, and the paths were
-   chosen so it needs no Remote Path Mapping.
+2. **RESOLVED — qbittorrent now has the bulk mount** (`subPath: downloads` at `/downloads`), so the
+   import source is written by the k3s download client rather than the legacy non-k3s one. See
+   `catalog/qbittorrent.yaml`.
 3. **`/mnt/Storage/plex/downloads/downloads` is a symlink to `/shared/downloads`**, which is outside
    the export. Over NFS it is a dangling symlink on every client. Harmless, but expect Sonarr to log
    it while scanning `downloads/`.
@@ -300,10 +347,10 @@ Without the arm, HALF A is what holds the line, by design.
 5. **`nconnect` was not set.** It would likely help throughput on the media tree, but it is a tuning
    change that should be measured, not guessed, and it changes client behaviour under server
    failure.
-6. **Nothing was verified against a live cluster.** No `kubectl` context is usable from this
-   workstation (browser-OIDC only), and this change is manifest-authoring under an explicit
-   do-not-apply constraint. Every claim about the *host* is measured; every claim about *binding and
-   mounting* is reasoned from the manifests.
+6. **Every claim about the NAS in §5 is measured (2026-10-07); the post-move MOUNT is not.** The PVs
+   still have to be recreated (§10) before any pod mounts 192.168.2.30, so the §9 probes are the
+   outstanding verification, not a formality. The pre-move equivalents did pass against the old host
+   on 2026-09-12.
 7. **The bulk-tier requirement ID is deliberately not typed on any non-`.md` line of this change.**
    `ci/touched-requirements.sh` attributes requirements from R-IDs appearing in added lines, and the
    in-cluster conformance test for the bulk tier is still a gap, so typing it would select a
@@ -312,36 +359,95 @@ Without the arm, HALF A is what holds the line, by design.
 
 ---
 
-## 9. Validating on dev, once the export is extended
+## 9. Validating, after the PVs are recreated
 
-Dev can prove all of this before prod's network path exists. Read-only checks first:
+These run against **prod**; dev is decommissioned. They are the outstanding verification for the
+move — nothing below has been run against 192.168.2.30. Read-only checks first:
 
 ```sh
-infra/devcontainer/dev.sh run kubectl --context k3s-dev get pv
-infra/devcontainer/dev.sh run kubectl --context k3s-dev get pvc -A | grep -E 'media|legacy-archive'
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr get pod -o wide
+./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pv -o custom-columns=\
+NAME:.metadata.name,SERVER:.spec.nfs.server,PATH:.spec.nfs.path,STATUS:.status.phase'
 ```
 
-Expect **four** PVs — `plex-media`, `sonarr-media`, `radarr-media`, `nextcloud-legacy-archive` (three
-of them naming the *same* export, one per namespace, which is the cross-namespace model) — each
-`Bound` to the same-named PVC, with `RECLAIM POLICY Retain` and an empty `STORAGECLASS`.
+Expect **five** NFS PVs — `plex-media`, `sonarr-media`, `radarr-media`, `qbittorrent-downloads`,
+`nextcloud-legacy-archive` (four of them naming the *same* export, one per namespace, which is the
+cross-namespace model) — each `Bound`, `RECLAIM POLICY Retain`, empty `STORAGECLASS`, and
+**`SERVER 192.168.2.30`**. A PV still showing `192.168.2.102` means the recreate in §10 did not
+happen; Flux will NOT have updated it in place.
 
-Then the two things that actually matter, from inside a pod:
+Then the four things that actually matter, from inside a pod:
 
 ```sh
-# (a) the writer can write — this is the uid check, and it is the one that fails in practice
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr exec deploy/sonarr -- \
-  sh -c 'id; touch /media/tv/.k3s-write-probe && rm /media/tv/.k3s-write-probe && echo WRITE-OK'
+# (a) numeric ids survived the wire -- the idmapping check, new since the TrueNAS move
+kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- sh -c 'ls -n /media | head -3'
+#     expect uid 1000 / gid 3000 on the entries. 65534 / nobody means ids are being mapped, not
+#     passed through, and every import will fail EACCES while the pod still reports Ready.
 
-# (b) hardlinks work BETWEEN the library and the download path — the root-mount check
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr exec deploy/sonarr -- \
+# (b) the writer can write -- the uid check, and the one that fails in practice
+kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- \
+  sh -c 'touch /media/tv/.k3s-write-probe && rm /media/tv/.k3s-write-probe && echo WRITE-OK'
+
+# (c) hardlinks work BETWEEN the library and the download path -- the root-mount check
+kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- \
   sh -c 'set -e; d=/media/downloads/complete/tv-sonarr/.probe; t=/media/tv/.probe-link;
          echo x > "$d"; ln "$d" "$t"; ls -i "$d" "$t"; rm -f "$d" "$t"; echo HARDLINK-OK'
 
-# (c) the reader really is read-only
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n plex exec deploy/plex -- \
-  sh -c 'touch /media/.probe 2>&1 | head -1'      # expect "Read-only file system"
+# (d) the reader really is read-only
+kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c 'touch /media/.probe 2>&1 | head -1'
+#     expect "Read-only file system"
 ```
 
-`(b)` printing two identical inode numbers is the proof that §4's argument holds in practice. If it
+`(c)` printing two identical inode numbers is the proof that §4's argument holds in practice. If it
 prints `EXDEV`, something has split the single mount back into two.
+
+Reading the failures together is what localises the fault: **(b) EACCES with (a) showing `nobody`**
+is idmapping (§5 service settings). **(b) EACCES with (a) showing `1000`** is on-disk ownership —
+stop and check `ls -ldn` on the NAS before changing anything else.
+
+---
+
+## 10. Repointing BULK_NFS_SERVER means DELETING AND RECREATING THE PVs
+
+**A bound PersistentVolume's NFS source is immutable.** Measured by server-side dry-run on
+2026-10-07:
+
+```
+$ kubectl patch pv sonarr-media --type=json --dry-run=server \
+    -p '[{"op":"replace","path":"/spec/nfs/server","value":"192.168.2.30"}]'
+The PersistentVolume "sonarr-media" is invalid: spec.persistentvolumesource: Forbidden:
+  spec.persistentvolumesource is immutable after creation
+```
+
+Note the pairing with §7: `spec.nfs.server` is **unvalidated at create** (any string is accepted and
+the PVC binds) and **immutable at update**. So editing `BULK_NFS_SERVER` alone does not fail loudly
+and does not work either — Flux's apply errors on the five PVs and the pods keep their previous,
+now-dead mount.
+
+This is safe to do because `persistentVolumeReclaimPolicy: Retain` means deleting a PV never touches
+the data, and `catalog/render.py` emits a **pre-set `claimRef`** on each one, so a recreated PV
+rebinds its own PVC with no manual step.
+
+**Operator sequence** (mutating `kubectl` is an operator action, not an agent action):
+
+```sh
+# 1. Stop the consumers. This also releases the hung hard-mounts against the old server:
+#    the pods report Ready, but any real I/O on /media blocks indefinitely once 2049 stops
+#    answering, because the PVs mount `hard`.
+for ns in plex sonarr radarr qbittorrent nextcloud; do
+  kubectl --context k3s-prod -n $ns scale deploy --all --replicas=0
+done
+
+# 2. Delete the five PVs. Retain + no live claim means nothing is reclaimed or erased.
+kubectl --context k3s-prod delete pv plex-media sonarr-media radarr-media \
+  qbittorrent-downloads nextcloud-legacy-archive
+
+# 3. Let Flux reconcile the revision carrying BULK_NFS_SERVER=192.168.2.30, which recreates them.
+#    Confirm .status.lastAppliedRevision matches the GitRepository artifact before continuing.
+
+# 4. Bring the consumers back, then run every probe in §9.
+```
+
+The PVCs pass through `Lost` between steps 2 and 3; that is expected and resolves when the PV with
+the matching `claimRef` reappears. If a PVC is still `Lost` after Flux reconciles, the PV name or
+its `claimRef` namespace does not match — do **not** delete the PVC to "fix" it.
+

infra/ansible/inventory/group_vars/k3s_prod/vars.yml

@@ -420,3 +420,34 @@ flux_bootstrap_env_path: "clusters/prod"
 # rather than leaving an inert drop-in behind, so this group-wide `true` is still correct
 # for it.
 etcd_io_priority_enabled: true
+
+# --- automatic security patching (roles/unattended_upgrades) -------------------------
+# TURNS ON roles/unattended_upgrades FOR PROD ONLY. Role default is false. Same seam as
+# node_baseline_backports_kernel and etcd_io_priority_enabled above.
+#
+# WHAT WAS THERE BEFORE: nothing at all. Measured on all three prod nodes 2026-10-17 —
+# `unattended-upgrades`, `update-notifier-common` and `needrestart` were ALL absent,
+# `apt-mark showhold` was empty on every node, /etc/apt/preferences.d/ was empty, and
+# nothing on the node could create /var/run/reboot-required. These boxes have had ZERO
+# automatic patching since provisioning, and no way to signal that a reboot was needed.
+#
+# WHAT ENABLING THIS DOES *NOT* DO: nothing, until the operator runs
+# playbooks/unattended-upgrades.yml. It is not applied by site.yml (deliberately — see
+# that playbook's header). And once run, it still never upgrades the KERNEL, never
+# restarts k3s, and never reboots:
+#   * kernel — blacklisted `^linux-`, AND the trixie-backports archive the running
+#     kernel comes from (o=Debian Backports, measured) is not an allowed origin. Two
+#     independent exclusions, because this is the EFI-memattr constraint: these HP minis
+#     need >= 6.13 or they can boot with a fraction of their RAM
+#     (PROD-PROVISIONING.md §0b), and 6.12 kernels are still installed and bootable here.
+#     The kernel continues to move ONLY under a deliberate
+#     `node-baseline.yml --tags kernel` run, exactly as before.
+#   * k3s — needrestart blacklist. k3s is not even an apt package on these nodes
+#     (measured: no `k3s`, no `containerd` in dpkg), so the control plane and runtime are
+#     outside apt's reach regardless.
+#   * reboots — Automatic-Reboot "false", asserted by the role against apt's own
+#     effective config. Reboots belong to kured (platform/kured): one node at a time,
+#     under a cluster-wide lock, inside 03:00-06:00, after cordon + drain. Three
+#     control-plane nodes with etcd quorum 2/3 must never reboot themselves
+#     independently, which is what Automatic-Reboot "true" would mean.
+unattended_upgrades_enabled: true

infra/ansible/playbooks/unattended-upgrades.yml

@@ -0,0 +1,36 @@
+---
+# Automatic Debian SECURITY patching for the k3s nodes — the package half of the automated upgrade
+# process. The reboot half is kured (platform/kured), reconciled by Flux; the two are designed
+# together and neither is useful alone. Full rationale for every decision:
+# roles/unattended_upgrades/defaults/main.yml and that role's README.md.
+#
+# TARGET (R-ENV-02): `k3s_hosts` defaults to dev's `k3s_servers`; prod is
+#   ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod
+# Same pattern node-baseline.yml and node-health.yml established — `hosts:` is templated before
+# group_vars exist, so the target is an extra var, never a group_vars value.
+#
+# The role is a NO-OP unless `unattended_upgrades_enabled` is true, which is set for prod only in
+# inventory/group_vars/k3s_prod/vars.yml. Running this against dev changes nothing.
+#
+# DELIBERATELY NOT IN site.yml. site.yml is the bring-up chain (node-baseline -> k3s ->
+# etcd-io-priority -> kube-vip -> cilium -> verify -> flux-bootstrap) and is "not run against prod
+# casually". Automatic patching is not a prerequisite for anything in that chain and nothing in the
+# chain depends on it, so adding it there would only widen what a cluster rebuild touches. It is
+# applied on its own, once, and re-applied when its configuration changes.
+#
+# SAFE TO RUN ON A LIVE CLUSTER, AT ANY TIME. This playbook installs and configures the upgrader. It
+# never runs `apt upgrade`, never restarts k3s, and never reboots — the role asserts the last of
+# those against apt's own effective config rather than trusting the file it just wrote. The first
+# unattended upgrade happens at the next timer firing (04:00 +/-60m), and any reboot that upgrade
+# implies is kured's to schedule, one node at a time, inside 03:00-06:00.
+#
+# TAGS. `--tags verify` re-runs only the read-only assertions and the per-node summary, which is the
+# way to answer "is automatic patching actually working on these boxes, and is anything waiting on a
+# reboot" without rewriting any configuration:
+#   ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify
+- name: Automatic security upgrades on the k3s nodes (kernel excluded; reboots delegated to kured)
+  hosts: "{{ k3s_hosts | default('k3s_servers') }}"
+  become: true
+  gather_facts: true
+  roles:
+    - unattended_upgrades

infra/ansible/roles/unattended_upgrades/README.md

@@ -0,0 +1,195 @@
+# roles/unattended_upgrades
+
+Automatic Debian **security** patching on the k3s nodes. The kernel is excluded, k3s is never
+restarted, and reboots are delegated to **kured** (`platform/kured`).
+
+- **Playbook:** `playbooks/unattended-upgrades.yml`
+- **Enabled by:** `unattended_upgrades_enabled` — role default `false`, set `true` for prod in
+  `inventory/group_vars/k3s_prod/vars.yml`
+- **Companion component:** `platform/kured` (the reboot half). **Neither is useful alone** — see
+  "The two halves" below.
+- **Every decision and its reasoning:** `defaults/main.yml`. That file is the argument; this one is
+  the orientation.
+
+## The two halves, and why they ship together
+
+| | what it does | what it must NOT do |
+|---|---|---|
+| **this role** (Ansible, host) | installs security updates continuously; leaves `/var/run/reboot-required` behind when one needs a reboot | upgrade the kernel; restart k3s; reboot |
+| **kured** (Flux, cluster) | cordons, drains, reboots and uncordons **one node at a time** when that file exists | decide *whether* a reboot is needed |
+
+Deploying either alone produces a mechanism that looks healthy and does nothing:
+
+- **kured alone** watches a file that **nothing on Debian ever creates**. Measured 2026-10-18,
+  after a first attempt to install Ubuntu's `update-notifier-common` failed on all three nodes
+  with *"No package matching 'update-notifier-common' is available"*:
+  - `apt-cache policy update-notifier-common` → `Candidate: (none)` — the package does not exist
+    in Debian; it is Ubuntu-only.
+  - `/usr/share/update-notifier/notify-reboot-required` → absent.
+  - **No `linux-image-*` postinst references `notify-reboot-required`** — calling it is an Ubuntu
+    patch, so even dropping a script at that path would never be invoked.
+  - No hook under `/etc/apt/apt.conf.d` or `/etc/kernel/` mentions `reboot-required`.
+
+  So **this role ships the writer itself** (`/usr/local/sbin/k3s-notify-reboot-required` plus a
+  `DPkg::Post-Invoke` hook) rather than installing a package. Without it kured would sit at 1/1
+  Running reporting `kured_reboot_required=0` forever.
+- **this role alone** installs patches that a long-running process never picks up, and when a
+  reboot *is* needed there is nothing safe to take it. `Automatic-Reboot "true"` is not the
+  answer: three control-plane nodes with etcd quorum 2/3, each rebooting at its own time, is how
+  the cluster is lost.
+
+## What it was replacing: nothing
+
+Measured on all three prod nodes, 2026-10-17:
+
+| | |
+|---|---|
+| `unattended-upgrades` | **not installed** |
+| `update-notifier-common` | **not installable** — Ubuntu-only, `Candidate: (none)` on trixie |
+| `needrestart` | **not installed** |
+| `apt-mark showhold` | **empty** — no package held on any node |
+| `/etc/apt/preferences.d/` | **empty** — no apt pin declares the backports-kernel intent |
+| `/var/run/reboot-required` | absent, and uncreatable |
+
+These nodes had received **no automatic patching since provisioning**.
+
+## The three decisions
+
+### 1. Security only
+
+Allowed origins are `label=Debian-Security` and nothing else. `trixie-updates` is excluded
+deliberately (not overlooked) — it carries non-security fixes that are useful but are the archive
+through which an unreviewed change would most plausibly arrive on a control-plane node. The knob to
+add it is `unattended_upgrades_include_updates`.
+
+### 2. The kernel is excluded twice over
+
+**This is the highest-risk part of the change.** These HP EliteDesk 800 G5 minis publish a corrupt
+EFI Memory Attributes Table on R21 firmware; Linux `< 6.13` does not validate it before reserving
+memory from its contents, and `k3s-prod-2` twice booted with **2.6 GiB of 16 GB**. Upstream
+`8fbe4c49c0cc` (first in v6.13) is the only fix — there is no kernel cmdline workaround. Full
+write-up: `infra/PROD-PROVISIONING.md` §0b and `roles/node_baseline/defaults/main.yml:21-51`.
+
+Measured 2026-10-17: all three nodes run `7.1.8+deb13-amd64` (`7.1.8-1~bpo13+1`, i.e. the backports
+metapackage), comfortably above the floor — **but `linux-image-6.12.107+deb13-amd64` and
+`linux-image-6.12.94+deb13-amd64` are also still installed, below the floor, bootable, and held by
+nothing.**
+
+So an unattended process must not be able to touch the kernel at all:
+
+1. **Origin exclusion (structural).** The running kernel comes from `o=Debian Backports` (measured
+   via `apt-cache policy`, pin priority 100 — the `NotAutomatic` + `ButAutomaticUpgrades`
+   signature). That origin is not in `Origins-Pattern`, so unattended-upgrades cannot see a kernel
+   upgrade at all.
+2. **Package blacklist (belt and braces).** `^linux-` is blacklisted, so even if the origin list is
+   later widened — the most likely future edit to this role, and the one that would silently re-arm
+   the hazard — the kernel is still untouchable.
+
+The role **asserts both** against `apt-config dump` (apt's own parse of the whole
+`/etc/apt/apt.conf.d/` directory) rather than trusting the file it just wrote.
+
+**This does not stop the kernel moving.** The operator still advances it deliberately with
+
+```
+ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel --limit <host>
+```
+
+which is the only thing that maintains the backports source, the metapackage and the on-node
+provenance stamp. `node_baseline`'s own note that `apt upgrade` keeps the backports kernel current
+still holds for an operator-run apt. What changed is that a **new unattended actor** is forbidden
+from it.
+
+Also blacklisted, for the same "operator-only" reason: `intel-microcode`, `amd64-microcode`,
+`grub-*`. Microcode and bootloader changes take effect on reboot and alter boot behaviour on the
+very firmware that is already misbehaving here.
+
+`Remove-Unused-Kernel-Packages` is **false**, against the Debian default: those older kernels are
+the documented rollback path, and an unattended process pruning them could remove the fallback an
+operator needs in front of a box that will not boot.
+
+### 3. needrestart restarts services — but never k3s
+
+A security-only policy that restarts nothing patches bytes on disk that no running process reads.
+`needrestart` in mode `a` restarts services still mapped to a deleted library, which is what makes
+the policy actually effective.
+
+`k3s.service` is blacklisted. Restarting the Kubernetes control plane unattended at 04:00 is a
+worse outcome than a delayed library fix; that fix lands at the next kured reboot instead. Measured
+2026-10-17, `k3s.service` is the **only** matching unit — k3s bundles its own containerd and
+kubelet, and neither `containerd` nor `containerd.io` is an apt package on these nodes — so this
+one entry is complete today.
+
+## Schedule
+
+`apt-daily` at 03:30 and `apt-daily-upgrade` at 04:00, both `±60m` randomised, via systemd drop-ins.
+Chosen to land **inside** kured's 03:00–06:00 reboot window so that a sentinel written by an upgrade
+can be acted on the same night rather than ~24h later.
+
+Two traps the role handles explicitly:
+
+- `/etc/apt/apt.conf.d/20auto-upgrades` must set the four `APT::Periodic::*` counters. Without
+  them the apt-daily timers fire and **do nothing** — installed, configured, green, and inert.
+- The timer drop-ins clear `OnCalendar=` before setting it. systemd **accumulates** `OnCalendar`
+  entries across drop-ins, so omitting the reset keeps the shipped schedule *in addition* to ours.
+
+## Why a new role rather than extending `node_baseline`
+
+`node_baseline` is the repo's only other apt surface and would have been a defensible home. Two
+reasons against:
+
+1. **Single responsibility.** `node_baseline` is the *first-boot floor* — it runs before k3s exists
+   and is the thing that makes a node installable. Continuous patching policy is a different
+   lifecycle with a different blast radius, and it wants to be runnable (and re-runnable) without
+   going anywhere near that role's data-disk tasks.
+2. **Concurrent work.** Other open branches touch `roles/node_baseline/`. A new role has nothing to
+   merge.
+
+Nothing here duplicates `node_baseline`: it does not add apt sources, does not touch the kernel, and
+does not manage disks.
+
+## Post-merge verification
+
+Running the role is safe on a live cluster at any time — it installs and configures the upgrader, and
+never runs `apt upgrade`, restarts k3s, or reboots.
+
+```bash
+# Apply (prod). Idempotent; re-runnable.
+cd infra/ansible && ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod
+
+# Re-run ONLY the read-only assertions + the per-node pending-reboot summary.
+cd infra/ansible && ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify
+```
+
+Then, on a node, confirm the three properties that matter (all read-only, no sudo needed for the
+first two):
+
+```bash
+ssh [email protected] 'apt-config dump | grep -iE "unattended-upgrade::(automatic-reboot|package-blacklist|origins)"'
+ssh [email protected] 'systemctl list-timers apt-daily-upgrade.timer --no-pager'
+ssh [email protected] 'ls -la /var/run/reboot-required* 2>&1 | head -2
+  ls -l /usr/local/sbin/k3s-notify-reboot-required /etc/apt/apt.conf.d/99-k3s-reboot-required 2>&1
+  dpkg -l needrestart unattended-upgrades | tail -3'
+```
+
+Expect: `Automatic-Reboot "false"`, a blacklist containing `^linux-`, an `Origins-Pattern`
+containing `Debian-Security` and **not** `Backports`, a next-elapse around 04:00, and all three
+packages `ii`.
+
+A dry run of what the upgrader would actually do (needs root, so this one is an operator step):
+
+```bash
+sudo unattended-upgrade --dry-run --debug
+```
+
+The kernel must appear in its blacklist output, not its upgrade list.
+
+## What this role cannot tell you
+
+- **That a security upgrade has ever succeeded.** That is in
+  `/var/log/unattended-upgrades/unattended-upgrades.log` on each node, and nothing in this repo or
+  in Prometheus reads it. There is no alert for "automatic patching has silently stopped working".
+  That is a real, known gap — see the handoff entry
+  `decisions/2026-10-17-automated-upgrades.md`.
+- **That a reboot actually happened.** That is kured's half, and it *is* alerted on
+  (`KuredRebootPending`, `KuredNodeCoverageGap`, `KuredNotReporting` in
+  `platform/kured/alerts.yaml`).

infra/ansible/roles/unattended_upgrades/defaults/main.yml

@@ -0,0 +1,199 @@
+---
+# roles/unattended_upgrades — automatic Debian SECURITY patching on the k3s nodes, with the kernel
+# deliberately excluded and with reboots delegated to kured (platform/kured).
+#
+# OFF BY DEFAULT, ON FOR PROD. Same seam as node_baseline_backports_kernel / etcd_io_priority_enabled:
+# a plain boolean, default false, turned on in inventory/group_vars/k3s_prod/vars.yml. Not a hostname
+# test — that would encode the environment into the role (see node_baseline's defaults for the full
+# argument, which this role follows rather than restates).
+#
+# ═════════════════════════════════════════════════════════════════════════════════════════════════
+# STATE BEFORE THIS ROLE (measured on all three prod nodes, 2026-10-17, via `dpkg-query -W`)
+# ═════════════════════════════════════════════════════════════════════════════════════════════════
+#   unattended-upgrades      NOT INSTALLED
+#   update-notifier-common   NOT INSTALLED
+#   needrestart              NOT INSTALLED
+#   apt-mark showhold        EMPTY (no package is held on any node)
+#   /var/run/reboot-required ABSENT, and nothing on the node can create it
+#   /etc/apt/preferences.d/  EMPTY (no apt pin declares the backports-kernel intent)
+# So: zero automatic patching, and no mechanism by which a reboot could ever be signalled. That last
+# point is why this role and platform/kured must land together — kured watches
+# /var/run/reboot-required, and on these nodes today NOTHING WRITES IT. Deploying kured alone would
+# produce a daemon that is permanently, silently idle while looking perfectly healthy.
+#
+# ═════════════════════════════════════════════════════════════════════════════════════════════════
+# DECISION 1 — SECURITY ONLY, NOT FULL UPGRADES
+# ═════════════════════════════════════════════════════════════════════════════════════════════════
+# Measured `apt-cache policy` on k3s-prod-1, 2026-10-17, gives the four archives and their pins:
+#   trixie            o=Debian,            a=stable,            l=Debian          pin 500
+#   trixie-updates    o=Debian,            a=stable-updates,    l=Debian          pin 500
+#   trixie-security   o=Debian,            a=stable-security,   l=Debian-Security pin 500
+#   trixie-backports  o=Debian Backports,  a=stable-backports,  l=Debian Backports pin 100
+#
+# Only `Debian-Security` is allowed below. The reasoning, in order of weight:
+#
+#   * Security updates are the class where the cost of NOT applying them is unbounded and the
+#     change itself is minimal by Debian policy (targeted fixes against the frozen stable version,
+#     no feature churn). It is the one class worth applying with nobody watching.
+#   * A full `apt upgrade` on these nodes would also pull trixie point-release changes, which is a
+#     larger surface than anyone has reviewed for a 3-node etcd cluster, for no urgency.
+#   * `trixie-updates` is DELIBERATELY EXCLUDED, not overlooked. It carries non-security fixes that
+#     are too urgent to wait for a point release — useful, but it is the archive through which an
+#     unreviewed change would most plausibly arrive, and it is not what "unattended" should mean on
+#     a control-plane node. Add it consciously later if the operator wants it; the knob is below.
+#   * `trixie-backports` is EXCLUDED, AND THAT IS THE KERNEL ARGUMENT — see DECISION 2.
+#
+# ═════════════════════════════════════════════════════════════════════════════════════════════════
+# DECISION 2 — THE KERNEL IS EXCLUDED TWICE OVER. THIS IS THE HIGHEST-RISK PART OF THE CHANGE.
+# ═════════════════════════════════════════════════════════════════════════════════════════════════
+# The constraint: HP's R21 firmware on these EliteDesk 800 G5 minis publishes a corrupt EFI Memory
+# Attributes Table, and Linux < 6.13 does not validate it before reserving memory from its contents
+# — k3s-prod-2 twice booted with 2.6 GiB of 16 GB. Upstream 8fbe4c49c0cc (first in v6.13) is the
+# ONLY fix; there is no kernel cmdline workaround. So these nodes run the backports metapackage
+# `linux-image-amd64` (measured 2026-10-17: version 7.1.8-1~bpo13+1 installed and running on all
+# three, well above the floor). Full write-up: infra/PROD-PROVISIONING.md §0b, and the rationale
+# block in roles/node_baseline/defaults/main.yml:21-51.
+#
+# A NODE THAT COMES BACK FROM AN UNATTENDED UPGRADE ON A 6.12 KERNEL IS A NODE THAT MAY COME BACK
+# WITH A FRACTION OF ITS RAM. Note that 6.12 kernels are not hypothetical here — measured
+# 2026-10-17, `linux-image-6.12.107+deb13-amd64` and `linux-image-6.12.94+deb13-amd64` are BOTH
+# still installed on all three nodes, below the floor, bootable, and nothing holds them.
+#
+# Two independent mechanisms keep unattended-upgrades away from the kernel, and the role asserts
+# both rather than trusting either:
+#
+#   1. ORIGIN EXCLUSION (structural). The running kernel comes from `o=Debian Backports`, which is
+#      not in Allowed-Origins. u-u only ever considers packages from allowed origins, so it cannot
+#      see a backports kernel upgrade at all. This also means the backports kernel receives no
+#      automatic patching — accepted, and in any case backports carries no security guarantees
+#      (node_baseline's defaults state that cost plainly: "no security-tracker commitment, no DSAs").
+#   2. PACKAGE BLACKLIST (belt and braces). `linux-*` is blacklisted below, so even if the origin
+#      list is later widened — the most likely future edit to this file, and the one that would
+#      silently re-arm the hazard — the kernel is still untouchable. Without this, widening origins
+#      to include backports would hand an unattended process the EFI-memattr-critical package.
+#
+# WHAT THIS DOES *NOT* DO, AND MUST NOT BE READ AS DOING: it does not stop the kernel moving. The
+# operator still advances it deliberately with
+#   ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel --limit <host>
+# which is the only thing that maintains the backports source, the metapackage and the on-node
+# provenance stamp. node_baseline's own comment ("`ButAutomaticUpgrades: yes` ... means `apt upgrade`
+# keeps it current once installed") still holds for an operator-run apt; the measured pin priority of
+# 100 is exactly the NotAutomatic + ButAutomaticUpgrades signature that produces that behaviour.
+# Nothing here changes it. What changes is that a NEW unattended actor is forbidden from the kernel.
+#
+# NOT BLACKLISTED, BECAUSE THEY ARE NOT APT PACKAGES AT ALL — measured 2026-10-17: `dpkg-query` finds
+# no `k3s`, no `containerd` and no `containerd.io`. k3s is a single static binary at
+# /usr/local/bin/k3s installed by roles/k3s_server, and it bundles its own containerd and kubelet.
+# So the entire Kubernetes control plane and container runtime are OUTSIDE apt's reach and cannot be
+# upgraded by this role even in principle. That is worth knowing before worrying about it.
+unattended_upgrades_enabled: false
+
+# The packages this role installs. Each one is load-bearing; see the notes.
+unattended_upgrades_packages:
+  # The upgrader itself.
+  - unattended-upgrades
+  # NOTE: `update-notifier-common` was listed here and REMOVED (2026-10-18). It is an UBUNTU
+  # package and does not exist in Debian at all -- `apt-cache policy update-notifier-common` on
+  # trixie returns `Candidate: (none)`, so the play failed outright with "No package matching
+  # 'update-notifier-common' is available" on all three nodes. The sentinel writer is now provided
+  # by this role itself; see unattended_upgrades_sentinel_* below and tasks/main.yml.
+  # Restarts services still mapped to a deleted library after a security upgrade. Without it a
+  # security-only policy patches bytes on disk that no running process reads — see DECISION 3.
+  - needrestart
+
+# --- Allowed origins -----------------------------------------------------------------------------
+# Verbatim apt origin expressions, substituted into 50unattended-upgrades. The ${distro_*} tokens
+# here are UNATTENDED-UPGRADES' OWN placeholders, expanded by it at runtime — they are not Ansible
+# or Flux substitutions, which is why they are single-quoted and why this role's files never travel
+# through Flux envsubst.
+#
+# `l=Debian-Security` plus the `-security` codename is the form Debian's own shipped config uses, and
+# both halves were confirmed against the measured archive metadata above (l=Debian-Security,
+# n=trixie-security). Matching on the LABEL rather than the suite name is what makes this survive the
+# next Debian release.
+unattended_upgrades_allowed_origins:
+  - 'origin=Debian,codename=${distro_codename},label=Debian-Security'
+  - 'origin=Debian,codename=${distro_codename}-security,label=Debian-Security'
+# Set this to add 'origin=Debian,codename=${distro_codename}-updates,label=Debian' — a deliberate
+# widening, kept as a knob so that turning it on is a reviewable one-line change rather than an edit
+# to a heredoc. See DECISION 1 for why it is off.
+unattended_upgrades_include_updates: false
+
+# --- Package blacklist ---------------------------------------------------------------------------
+# Regular expressions matched against package names. See DECISION 2: `linux-` is the EFI-memattr
+# guard and must not be removed without re-reading PROD-PROVISIONING.md §0b.
+unattended_upgrades_blacklist:
+  # Every kernel package: linux-image-*, linux-headers-*, linux-image-amd64 (the metapackage that
+  # actually tracks backports), plus linux-kbuild / linux-libc-dev. Anchored at the start so it
+  # cannot accidentally match an unrelated package containing the substring.
+  - '^linux-'
+  # Microcode changes take effect only on reboot and can alter boot behaviour on the very firmware
+  # that is already misbehaving here. Operator-only, same argument as the kernel.
+  - '^intel-microcode$'
+  - '^amd64-microcode$'
+  # The bootloader. A grub upgrade rewrites boot configuration on a machine with a documented
+  # warm-reboot hang (k3s-prod-2, 2026-09-03). Never unattended.
+  - '^grub-'
+  - '^grub2-common$'
+
+# --- Reboot policy -------------------------------------------------------------------------------
+# FALSE, AND THIS IS THE DIVISION OF LABOUR WITH kured, NOT AN OVERSIGHT.
+# unattended-upgrades can reboot a node itself (Automatic-Reboot "true"). It must not here: it has
+# no notion of a cluster, so it would not cordon, would not drain, would not evict the 25 attached
+# Longhorn volumes cleanly, would not check whether another node is already down, and — on three
+# control-plane nodes where etcd quorum is 2 of 3 — three nodes each deciding independently to
+# reboot at their own `Automatic-Reboot-Time` is precisely how you lose the cluster.
+# u-u therefore only ever leaves the sentinel behind; kured owns every reboot, one node at a time,
+# under a cluster-wide lock. See platform/kured/helmrelease.yaml.
+unattended_upgrades_automatic_reboot: false
+
+# --- needrestart (DECISION 3) --------------------------------------------------------------------
+# See the role README for the full argument. Summary: 'a' (restart services automatically) is what
+# makes a security-only policy actually effective — a patched libssl on disk does nothing for a
+# process still mapping the old one — but k3s.service is EXCLUDED, because restarting the
+# Kubernetes control plane unattended at 06:00 is a worse outcome than a delayed library fix.
+# Measured 2026-10-17: the only matching unit on these nodes is `k3s.service` (no separate
+# containerd or kubelet unit exists; k3s bundles both), so this blacklist is complete today.
+unattended_upgrades_needrestart_mode: a
+unattended_upgrades_needrestart_blacklist:
+  - '^k3s'
+
+# --- Schedule ------------------------------------------------------------------------------------
+# apt's own systemd timers (apt-daily, apt-daily-upgrade) drive u-u; these set their windows.
+# 04:00 with a 1h randomised delay, so the upgrade lands INSIDE kured's 03:00-06:00 reboot window
+# (platform/kured/helmrelease.yaml) rather than hours away from it: if an upgrade ever does set the
+# sentinel, the reboot can be taken the same night instead of waiting ~24h. The randomised delay
+# keeps the three nodes from hitting the mirror simultaneously; it does NOT serialise anything, and
+# it does not need to — u-u never reboots, so three concurrent upgrades are harmless.
+unattended_upgrades_download_time: "03:30"
+unattended_upgrades_upgrade_time: "04:00"
+unattended_upgrades_randomised_delay: 60m
+
+
+# ── THE REBOOT SENTINEL, PROVIDED BY THIS ROLE ──────────────────────────────────────────────────
+# /var/run/reboot-required is the ONLY thing kured watches, and on Debian trixie **nothing creates
+# it**. Measured on all three nodes 2026-10-18:
+#   * `update-notifier-common` (which owns that hook on Ubuntu) has `Candidate: (none)` — the
+#     package does not exist in Debian. Installing it is what broke the first run of this play.
+#   * `/usr/share/update-notifier/notify-reboot-required` is absent.
+#   * NO `linux-image-*` postinst references `notify-reboot-required` — calling it is an Ubuntu
+#     patch, not Debian behaviour, so merely dropping a script at that path would never be invoked.
+#   * No hook under /etc/apt/apt.conf.d or /etc/kernel/ mentions reboot-required.
+# So the writer has to be ours. Without it kured is a permanent, INVISIBLE no-op: it stays 1/1
+# Running and reports kured_reboot_required=0 forever, which is the "failure mode is a green tick"
+# shape this repo keeps getting burned by.
+#
+# MECHANISM: a dpkg Post-Invoke hook runs a script that compares the RUNNING kernel against the
+# newest INSTALLED one and touches the sentinel when they differ. Deliberately kernel-only, which
+# matches the design: unattended-upgrades is security-only with `^linux-` blacklisted, so it will
+# essentially never set this — the sentinel appears when the OPERATOR deliberately runs the
+# node_baseline kernel play, and kured then automates only the mechanical reboot.
+#
+# It uses `linux-version` (from `linux-base`, a dependency of every linux-image package, so
+# guaranteed present — measured `linux-base 4.12.1` installed) rather than parsing `dpkg -l`
+# output, because kernel version ordering is not string ordering.
+#
+# /run is a tmpfs (measured), so the sentinel self-clears on boot: after a reboot the running
+# kernel IS the newest and the script writes nothing. No cleanup task is needed, and none exists.
+unattended_upgrades_sentinel_script: /usr/local/sbin/k3s-notify-reboot-required
+unattended_upgrades_sentinel_hook: /etc/apt/apt.conf.d/99-k3s-reboot-required

infra/ansible/roles/unattended_upgrades/handlers/main.yml

@@ -0,0 +1,29 @@
+---
+# Re-arm apt's periodic timers after a config or drop-in change.
+#
+# WHY A daemon_reload IS REQUIRED AND NOT OPTIONAL. The timer windows are set by drop-ins under
+# /etc/systemd/system/apt-daily{,-upgrade}.timer.d/. systemd does not re-read unit files on its own,
+# so until it is told to, the RUNNING manager keeps the shipped schedule and the new window is
+# ignored — the files look right in git and on disk, `systemctl cat` shows them, and the upgrade
+# still fires at the old time. That is a silent config no-op, which is the failure mode this whole
+# role is written against.
+#
+# `daemon_reload` re-parses unit files and re-runs generators. It starts and stops nothing, so it
+# cannot disturb k3s, containerd or Longhorn — the same reasoning node_baseline's "Reload systemd
+# unit files" handler records.
+#
+# RESTARTING THE TIMERS IS ALSO SAFE, and is not the same as running the upgrade. `*.timer` units
+# only hold the schedule; restarting one recomputes its next elapse and does NOT trigger its
+# `*.service`. With `Persistent=true` set in the drop-in, a window missed while the node was down is
+# caught up on the next boot rather than skipped.
+#
+# Both fire only when a task actually changed, so a re-run of the role is silent.
+- name: Restart unattended-upgrades timers
+  ansible.builtin.systemd_service:
+    name: "{{ item }}"
+    state: restarted
+    enabled: true
+    daemon_reload: true
+  loop:
+    - apt-daily.timer
+    - apt-daily-upgrade.timer

infra/ansible/roles/unattended_upgrades/tasks/main.yml

@@ -0,0 +1,369 @@
+---
+# roles/unattended_upgrades — automatic Debian security patching, kernel excluded, reboots delegated
+# to kured. Every decision and its reasoning lives in defaults/main.yml; this file is the mechanism.
+#
+# TAGS — following roles/node_baseline, every task carries one of `packages`, `config` or `verify`.
+# An untagged run does all three in that order. The split exists because `verify` is the half an
+# operator will want to re-run on its own to answer "is automatic patching actually working on this
+# box", without re-writing any configuration:
+#   ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify
+#
+# IT DOES NOT UPGRADE ANYTHING, AND IT DOES NOT REBOOT. This role installs and CONFIGURES the
+# upgrader; it never runs `apt upgrade` itself and never takes a reboot. Both of those are deliberate
+# and both follow roles/node_baseline/tasks/main.yml:61-67 ("IT DOES NOT REBOOT", after k3s-prod-2
+# hung on a warm reboot on 2026-09-03 and needed a physical power button). Applying this role is
+# therefore a SAFE operation on a live cluster at any time of day: the first unattended upgrade
+# happens at the next timer firing, and any reboot it implies is kured's to schedule inside the
+# 03:00-06:00 window.
+- name: Automatic security upgrades (unattended-upgrades; kernel excluded, reboots via kured)
+  when: unattended_upgrades_enabled | bool
+  tags: [packages, config, verify]
+  block:
+
+    # ── PREFLIGHT ───────────────────────────────────────────────────────────────────────────────
+    - name: Assert this is a Debian-family host
+      # The config file layout, the origin expressions and the package names below are all
+      # Debian-specific. On anything else this role would write files that are silently ignored,
+      # which is the failure mode this whole PR exists to avoid.
+      ansible.builtin.assert:
+        that:
+          - ansible_facts['os_family'] == 'Debian'
+        fail_msg: >-
+          REFUSING TO CONFIGURE unattended-upgrades on {{ inventory_hostname }}: os_family is
+          '{{ ansible_facts["os_family"] }}', not 'Debian'. The allowed-origin expressions and
+          config paths in this role are Debian-specific.
+      tags: [packages, config, verify]
+
+    # ── PACKAGES ────────────────────────────────────────────────────────────────────────────────
+    - name: Install unattended-upgrades and needrestart
+      # The reboot-sentinel writer is NOT a package here. On Debian nothing ships one -- see
+      # defaults/main.yml -- so this role installs its own script and dpkg hook, two tasks below.
+      ansible.builtin.apt:
+        name: "{{ unattended_upgrades_packages }}"
+        state: present
+        update_cache: true
+        cache_valid_time: 3600
+      tags: [packages]
+
+    # ── THE REBOOT SENTINEL WRITER (ours, because Debian ships none) ────────────────────────────
+    # Full rationale and the four measurements behind it: defaults/main.yml. Short version: kured
+    # watches /var/run/reboot-required and nothing on Debian creates it, so without these two files
+    # the entire reboot half is an invisible no-op.
+    - name: Install the reboot-sentinel writer
+      ansible.builtin.copy:
+        dest: "{{ unattended_upgrades_sentinel_script }}"
+        owner: root
+        group: root
+        mode: "0755"
+        content: |
+          #!/bin/sh
+          # MANAGED BY ANSIBLE -- infra/ansible/roles/unattended_upgrades. Local edits are reverted.
+          #
+          # Touch /var/run/reboot-required when the RUNNING kernel is not the newest INSTALLED one.
+          # kured watches that path; see the role's defaults/main.yml for why this is ours to ship.
+          #
+          # `linux-version` comes from linux-base (a dependency of every linux-image package), and
+          # is used rather than sorting `dpkg -l` output because kernel version ordering is not
+          # string ordering. If it is somehow missing, exit 0 and write nothing: a dpkg hook must
+          # never fail an apt transaction, and a missing sentinel is caught by the verify assert.
+          set -eu
+          command -v linux-version >/dev/null 2>&1 || exit 0
+
+          running="$(uname -r)"
+          newest="$(linux-version list 2>/dev/null | linux-version sort --reverse 2>/dev/null | head -n1)"
+
+          # No newest (nothing installed we can read) -> nothing to say.
+          [ -n "$newest" ] || exit 0
+
+          if [ "$running" != "$newest" ]; then
+            touch /var/run/reboot-required 2>/dev/null || exit 0
+            # kured only needs the file, but the .pkgs companion is what an operator reads to find
+            # out WHY a node is waiting, so write a reason rather than an empty marker.
+            printf 'linux-image-%s\n' "$newest" > /var/run/reboot-required.pkgs 2>/dev/null || true
+          fi
+          exit 0
+      tags: [packages, config]
+
+    - name: Install the dpkg hook that runs the sentinel writer
+      # Post-Invoke, so it runs after dpkg has finished and a newly installed kernel is already
+      # visible to `linux-version list`. `|| true` so a failure here can never abort an apt
+      # transaction -- an unwritten sentinel is a missed reboot, but a broken apt is a broken node.
+      ansible.builtin.copy:
+        dest: "{{ unattended_upgrades_sentinel_hook }}"
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          // MANAGED BY ANSIBLE -- infra/ansible/roles/unattended_upgrades. Local edits reverted.
+          // Writes /var/run/reboot-required when the running kernel is not the newest installed.
+          DPkg::Post-Invoke { "{{ unattended_upgrades_sentinel_script }} || true"; };
+      tags: [packages, config]
+
+    # ── CONFIG ──────────────────────────────────────────────────────────────────────────────────
+    # Written as a whole file rather than patched in place, for the reason node_baseline gives for
+    # its own apt source: an idempotent full-file write with a managed-by header is reviewable and
+    # re-runnable, whereas line-editing a conffile fights dpkg on every package upgrade.
+    #
+    # 52- RATHER THAN 50-: Debian ships /etc/apt/apt.conf.d/50unattended-upgrades as a CONFFILE of
+    # the unattended-upgrades package. Overwriting it would make every future apt upgrade of that
+    # package prompt about a locally-modified conffile (or, under the non-interactive defaults a
+    # scripted run uses, silently keep one side and discard the other). apt reads the directory in
+    # lexical order and later files win, so a separate 52- file overrides the shipped defaults
+    # without ever touching them, and `apt-config dump` in the verify block below proves which value
+    # actually won rather than assuming this ordering holds.
+    - name: Configure unattended-upgrades (52-k3s-unattended-upgrades)
+      ansible.builtin.copy:
+        dest: /etc/apt/apt.conf.d/52-k3s-unattended-upgrades
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          // MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades. Local edits are reverted
+          // on the next run. Rationale for every value: that role's defaults/main.yml.
+          //
+          // Overrides the shipped 50unattended-upgrades by lexical ordering (52 > 50). The shipped
+          // file is a dpkg conffile and is deliberately left untouched.
+
+          // SECURITY ONLY. Deliberately excludes trixie-updates (non-security fixes) and
+          // trixie-backports (o=Debian Backports) -- and excluding backports is what makes it
+          // structurally impossible for this upgrader to move the EFI-memattr-critical kernel,
+          // which is installed from there. See defaults/main.yml DECISION 2.
+          Unattended-Upgrade::Origins-Pattern {
+          {% for origin in unattended_upgrades_allowed_origins %}
+                  "{{ origin }}";
+          {% endfor %}
+          {% if unattended_upgrades_include_updates | bool %}
+                  "origin=Debian,codename=${distro_codename}-updates,label=Debian";
+          {% endif %}
+          };
+
+          // NEVER AUTOMATICALLY UPGRADED, whatever the origin list later says. `^linux-` is the
+          // EFI-memattr guard: these HP minis need kernel >= 6.13 or they can boot with a fraction
+          // of their RAM, and 6.12 kernels are still installed and bootable on all three nodes.
+          // Do not remove without re-reading infra/PROD-PROVISIONING.md §0b.
+          Unattended-Upgrade::Package-Blacklist {
+          {% for pkg in unattended_upgrades_blacklist %}
+                  "{{ pkg }}";
+          {% endfor %}
+          };
+
+          // REBOOTS BELONG TO kured (platform/kured), NOT TO apt. All three nodes are
+          // control-plane + etcd with quorum 2 of 3, so three hosts independently deciding to
+          // reboot at their own Automatic-Reboot-Time is how the cluster is lost. u-u leaves
+          // /var/run/reboot-required behind; kured takes a cluster-wide lock and reboots exactly
+          // one node at a time, after cordon + drain, inside 03:00-06:00, gated on Longhorn volume
+          // health.
+          Unattended-Upgrade::Automatic-Reboot "{{ unattended_upgrades_automatic_reboot | bool | lower }}";
+          Unattended-Upgrade::Automatic-Reboot-WithUsers "false";
+
+          // Keep the node's disk from filling with superseded .debs, and keep a log an operator can
+          // read after the fact (/var/log/unattended-upgrades/). MinAge guards against removing a
+          // package still needed by a rollback.
+          Unattended-Upgrade::Remove-Unused-Kernel-Packages "false";
+          Unattended-Upgrade::Remove-New-Unused-Dependencies "true";
+          Unattended-Upgrade::Remove-Unused-Dependencies "false";
+          Unattended-Upgrade::AutoFixInterruptedDpkg "true";
+          Unattended-Upgrade::MinimalSteps "true";
+          Unattended-Upgrade::SyslogEnable "true";
+
+          // Remove-Unused-Kernel-Packages is FALSE on purpose, against the Debian default. These
+          // nodes deliberately carry more than one kernel: the backports kernel they boot and the
+          // distro kernels that are the documented rollback path (node_baseline writes the rollback
+          // procedure onto the node itself). An unattended process pruning kernels could remove the
+          // fallback an operator needs at 3am in front of a box that will not boot.
+      notify: Restart unattended-upgrades timers
+      tags: [config]
+
+    - name: Enable the apt periodic timers that actually run unattended-upgrades
+      # Installing the package is NOT sufficient to make it run on a schedule: without these
+      # counters apt's daily units do nothing, and the whole mechanism is inert while looking
+      # installed. `1` means "every day".
+      ansible.builtin.copy:
+        dest: /etc/apt/apt.conf.d/20auto-upgrades
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          // MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades.
+          // Without these four counters the apt-daily timers fire and do nothing, so
+          // unattended-upgrades would be installed, configured, and never once invoked.
+          APT::Periodic::Update-Package-Lists "1";
+          APT::Periodic::Download-Upgradeable-Packages "1";
+          APT::Periodic::Unattended-Upgrade "1";
+          APT::Periodic::AutocleanInterval "7";
+      tags: [config]
+
+    - name: Ensure the systemd timer drop-in directories exist
+      # `copy` does not create parent directories, and these two .timer.d/ dirs do not exist on a
+      # stock Debian node — without this task the next task fails with "Destination directory does
+      # not exist" on a first run.
+      ansible.builtin.file:
+        path: "/etc/systemd/system/{{ item }}.timer.d"
+        state: directory
+        owner: root
+        group: root
+        mode: "0755"
+      loop:
+        - apt-daily
+        - apt-daily-upgrade
+      tags: [config]
+
+    - name: Pin the apt timer windows into kured's reboot window
+      # Overriding the shipped timers rather than editing them: a drop-in survives package upgrades.
+      # OnCalendar is cleared first (`OnCalendar=` with no value) because systemd ACCUMULATES
+      # OnCalendar entries across drop-ins — omit the reset and the unit keeps its shipped schedule
+      # IN ADDITION to ours, which would run the upgrade at both times and look like it worked.
+      ansible.builtin.copy:
+        dest: "/etc/systemd/system/{{ item.unit }}.timer.d/override.conf"
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          # MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades.
+          # Lands the upgrade inside kured's 03:00-06:00 reboot window so a sentinel written by an
+          # upgrade can be acted on the same night rather than ~24h later.
+          [Timer]
+          OnCalendar=
+          OnCalendar={{ item.time }}
+          RandomizedDelaySec={{ unattended_upgrades_randomised_delay }}
+          Persistent=true
+        force: true
+      loop:
+        - { unit: apt-daily, time: "{{ unattended_upgrades_download_time }}" }
+        - { unit: apt-daily-upgrade, time: "{{ unattended_upgrades_upgrade_time }}" }
+      loop_control:
+        label: "{{ item.unit }} at {{ item.time }}"
+      notify: Restart unattended-upgrades timers
+      tags: [config]
+
+    - name: Configure needrestart (restart services, but never k3s)
+      # See defaults/main.yml DECISION 3. Auto-restart is what makes a security-only policy real --
+      # a patched library on disk does nothing for a process still mapping the deleted one -- but
+      # k3s.service is excluded because restarting the control plane unattended is worse than a
+      # delayed library fix. `$nrconf{restart}` is Perl config read by needrestart itself.
+      ansible.builtin.copy:
+        dest: /etc/needrestart/conf.d/50-k3s.conf
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          # MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades.
+          # Restart mode: '{{ unattended_upgrades_needrestart_mode }}'  (a = automatically, l = list only, i = interactive)
+          $nrconf{restart} = '{{ unattended_upgrades_needrestart_mode }}';
+          # Units matching these patterns are NEVER restarted automatically. k3s bundles containerd
+          # and the kubelet, so this one entry covers the entire control plane and runtime on these
+          # nodes (measured 2026-10-17: k3s.service is the only matching unit). A library fix that
+          # needs k3s restarted therefore waits for the reboot kured will take.
+          $nrconf{blacklist_rc} = [
+          {% for pat in unattended_upgrades_needrestart_blacklist %}
+              qr({{ pat }}),
+          {% endfor %}
+          ];
+      tags: [config]
+
+    # ── VERIFY ──────────────────────────────────────────────────────────────────────────────────
+    # These tasks exist because every component of this mechanism fails SILENTLY. An installed
+    # package, a written config file and a green play are all compatible with nothing ever being
+    # upgraded and no reboot ever being signalled.
+
+    - name: Ask apt-config what the effective settings actually are
+      # `apt-config dump` is apt's OWN parse of the whole /etc/apt/apt.conf.d/ directory, so it
+      # proves what WON after lexical ordering rather than what this role wrote. If the shipped
+      # 50unattended-upgrades were ever renamed above 52-, this is the task that would catch it.
+      ansible.builtin.command: apt-config dump
+      register: unattended_upgrades_aptconfig
+      changed_when: false
+      check_mode: false
+      tags: [verify]
+
+    - name: Assert the kernel is blacklisted and automatic reboots are off
+      # The two properties that must never silently regress. Asserted against apt's own view.
+      ansible.builtin.assert:
+        that:
+          # apt-config renders each list entry as `Unattended-Upgrade::Package-Blacklist:: "^linux-";`
+          # so the anchored pattern itself is the thing to look for. Checking for the bare string
+          # `linux-` would pass on almost any dump and prove nothing.
+          - "'\"^linux-\"' in unattended_upgrades_aptconfig.stdout"
+          - "'Unattended-Upgrade::Automatic-Reboot \"false\"' in unattended_upgrades_aptconfig.stdout"
+          # The origin list must be non-empty, or u-u runs nightly and upgrades nothing — installed,
+          # scheduled, green, and inert.
+          - "'Debian-Security' in unattended_upgrades_aptconfig.stdout"
+        fail_msg: >-
+          EFFECTIVE apt CONFIG ON {{ inventory_hostname }} IS NOT WHAT THIS ROLE INTENDED.
+          Either the `^linux-` kernel blacklist or `Automatic-Reboot "false"` is not winning.
+          Both are safety properties, not preferences: without the blacklist an unattended process
+          can move a kernel these HP minis must keep >= 6.13 (PROD-PROVISIONING.md §0b), and
+          without Automatic-Reboot=false three control-plane nodes can each reboot themselves
+          independently and take etcd below quorum. Inspect with `apt-config dump | grep -i
+          unattended` and check for a file sorting after 52- in /etc/apt/apt.conf.d/.
+        success_msg: >-
+          Effective apt config confirmed: kernel blacklisted, automatic reboot disabled.
+      tags: [verify]
+
+    # THE SENTINEL IS THE SINGLE POINT OF FAILURE FOR THE ENTIRE REBOOT HALF OF THIS DESIGN.
+    # This was originally a `find` over /etc/kernel/postinst.d and /etc/apt/apt.conf.d, on the
+    # theory that the writer came from a package and the role could only discover whatever that
+    # package happened to ship. That theory was wrong: no such package exists on Debian (see
+    # defaults/main.yml), so the writer is now OURS and lives at two known paths. Checking two
+    # exact paths is strictly stronger than a glob -- a glob would have matched Debian's own
+    # unattended-upgrades apt conffile and passed while no sentinel writer existed at all.
+    - name: Stat the sentinel writer script
+      ansible.builtin.stat:
+        path: "{{ unattended_upgrades_sentinel_script }}"
+      register: unattended_upgrades_sentinel_script_stat
+      changed_when: false
+      tags: [verify]
+
+    - name: Stat the sentinel dpkg hook
+      ansible.builtin.stat:
+        path: "{{ unattended_upgrades_sentinel_hook }}"
+      register: unattended_upgrades_sentinel_hook_stat
+      changed_when: false
+      tags: [verify]
+
+    - name: Assert something on this node can write the reboot sentinel
+      # If this fails, kured is a no-op: it watches /var/run/reboot-required forever, the file is
+      # never created, no reboot is ever taken, and NOTHING GOES RED -- the DaemonSet stays 1/1
+      # Running and `kured_reboot_required` reports a steady 0 on every node. That is the exact
+      # shape of failure (a check whose failure mode is a green tick) this repo has been burned by
+      # before, so it is an assert and not a debug message.
+      ansible.builtin.assert:
+        that:
+          - unattended_upgrades_sentinel_script_stat.stat.exists
+          - unattended_upgrades_sentinel_script_stat.stat.executable | default(false)
+          - unattended_upgrades_sentinel_hook_stat.stat.exists
+        fail_msg: >-
+          NOTHING ON {{ inventory_hostname }} CAN CREATE /var/run/reboot-required.
+          {{ unattended_upgrades_sentinel_script }} (executable) and
+          {{ unattended_upgrades_sentinel_hook }} must both exist -- this role ships both, because
+          Debian provides no sentinel writer of its own (update-notifier-common is an UBUNTU
+          package and is not installable here). kured watches that path and reboots only when it
+          exists, so with no writer the reboot automation is permanently and INVISIBLY idle --
+          kured stays 1/1 Running and reports kured_reboot_required=0 on every node forever.
+          Re-run this role without `--tags verify` to install them.
+        success_msg: >-
+          Reboot sentinel writer present ({{ unattended_upgrades_sentinel_script }} + dpkg hook)
+          -- a kernel change will signal kured on the next apt transaction.
+      tags: [verify]
+
+    - name: Report the pending-reboot state of this node
+      # Read-only. Tells the operator whether this node is currently waiting on kured for a reboot,
+      # which is the question `--tags verify` is usually being run to answer.
+      ansible.builtin.stat:
+        path: /var/run/reboot-required
+      register: unattended_upgrades_reboot_required
+      changed_when: false
+      tags: [verify]
+
+    - name: Summary
+      ansible.builtin.debug:
+        msg: >-
+          {{ inventory_hostname }}: security-only unattended upgrades ACTIVE
+          (upgrade timer {{ unattended_upgrades_upgrade_time }} +/-{{ unattended_upgrades_randomised_delay }}),
+          kernel and microcode blacklisted, automatic reboot DISABLED.
+          Reboot currently pending: {{ unattended_upgrades_reboot_required.stat.exists }}.
+          {% if unattended_upgrades_reboot_required.stat.exists %}
+          This node is waiting for kured to reboot it in the next 03:00-06:00 window; it will be
+          cordoned and drained first, and no other node will be rebooted at the same time.
+          {% endif %}
+      tags: [verify]

infra/vpn-ingress/new-peer.sh

@@ -3,7 +3,7 @@
 # new-peer.sh — enrol one device on the remote-access VPN (platform/vpn-ingress).
 #
 # Generates a WireGuard keypair, renders the device's client profile, prints a QR code for a phone,
-# and prints the one `[Peer]` stanza to paste into platform/vpn-ingress/peers.yaml. The device's
+# and prints the one `[Peer]` stanza to paste into platform/vpn-ingress/peers.conf. The device's
 # private half is written to ONE 0600 file and nowhere else: not sent to the cluster, not committed,
 # never logged.
 #
@@ -62,7 +62,7 @@ set -euo pipefail
 die() { printf 'new-peer.sh: %s\n' "$*" >&2; exit 1; }
 
 REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
-PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.yaml"
+PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.conf"
 
 # --- defaults, each mirroring a value that is DECLARED elsewhere in this repo -------------------
 # These are duplicated here rather than parsed out of the YAML on purpose: this script is meant to
@@ -134,16 +134,21 @@ CLIENT_ADDR="${CLIENT_BASE}.${OCTET}"
 # Two devices sharing an address does not error anywhere: the server routes that address to
 # whichever peer handshook most recently, so BOTH devices appear to work intermittently.
 # COMMENT LINES ARE NOT ALLOCATIONS. This used to grep the whole file and so matched the
-# COMMENTED-OUT EXAMPLE STANZA in peers.yaml, which made `.2` — the first address anyone would
+# COMMENTED-OUT EXAMPLE STANZA in the register, which made `.2` — the first address anyone would
 # ask for — permanently un-allocatable with a message claiming it was already taken. Strip
 # comments first: a real stanza's `AllowedIPs` line is never commented, while both the template
 # and every peer's `# <name> — <owner>` label are.
-if [ -r "$PEERS_FILE" ] \
-   && grep -vE '^[[:space:]]*#' "$PEERS_FILE" | grep -q "AllowedIPs *= *${CLIENT_ADDR}/32"; then
-  die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.yaml. Pick another octet."
+# FAIL if the register is missing rather than skipping the check. The path was silently wrong once
+# (it still said peers.yaml after the file became peers.conf), and because the old guard was
+# `[ -r "$PEERS_FILE" ] && ...` the only symptom would have been duplicate detection quietly
+# never firing again — the exact silent-skip shape this repo treats as a defect.
+[ -r "$PEERS_FILE" ] || die "cannot read the peer register '$PEERS_FILE'.
+  Duplicate-address detection cannot run, so this script refuses to issue a profile."
+if grep -vE '^[[:space:]]*#' "$PEERS_FILE" | grep -q "AllowedIPs *= *${CLIENT_ADDR}/32"; then
+  die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.conf. Pick another octet."
 fi
-if [ -r "$PEERS_FILE" ] && grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then
-  printf 'new-peer.sh: WARNING a peer comment already mentions "%s" in peers.yaml.\n' "$NAME" >&2
+if grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then
+  printf 'new-peer.sh: WARNING a peer comment already mentions "%s" in peers.conf.\n' "$NAME" >&2
 fi
 
 # --- output location: 0700, and never inside the repo -------------------------------------------
@@ -210,7 +215,7 @@ cat <<EOF
   Profile     : ${PROFILE}   (0600; contains the private key)
 
   ------------------------------------------------------------------------------------------
-  STEP 1 — add this to platform/vpn-ingress/peers.yaml, under data.peers.conf, then PR it:
+  STEP 1 — append this to platform/vpn-ingress/peers.conf, then PR it:
 
     # ${NAME} — <owner>, enrolled $(date -u +%Y-%m-%d)
     [Peer]
@@ -229,7 +234,7 @@ cat <<EOF
   STEP 3 — delete the profile from this machine once the device has it:
     shred -u ${PROFILE} 2>/dev/null || rm -f ${PROFILE}
 
-  The tunnel will not work until the peers.yaml PR is MERGED and Flux has rolled the DaemonSet
+  The tunnel will not work until the peers.conf PR is MERGED and Flux has rolled the DaemonSet
   (~1 min). Until then the device handshakes and gets silence, exactly like an unknown key.
   ------------------------------------------------------------------------------------------
 EOF

platform/kured/README.md

@@ -0,0 +1,127 @@
+# platform/kured
+
+**kured (Kubernetes Reboot Daemon)** — coordinated, one-node-at-a-time OS reboots of the three prod
+control-plane nodes.
+
+- **Flux tier:** `clusters/base/kured.yaml` (its own Kustomization, `dependsOn: [platform]`,
+  `wait: true`)
+- **Chart:** `kured` 6.1.0 → appVersion 1.23.0, from `https://kubereboot.github.io/charts`
+- **The other half:** `infra/ansible/roles/unattended_upgrades` — the package upgrades. **Neither
+  half is useful alone.**
+- **Runbook:** `/runbooks/node-reboot-automation`
+- **Every tuned value and its reasoning:** `helmrelease.yaml`. That file is the argument; this one is
+  the orientation.
+- **Design record:** `decisions/2026-10-17-automated-upgrades.md`
+
+## What this component owns
+
+The *mechanics* of a reboot: take a cluster-wide lock, cordon, drain, reboot, uncordon, hold the
+lock a while longer. Nothing else.
+
+**It does not decide whether to reboot.** kured reboots a node if and only if
+`/var/run/reboot-required` exists there. On these nodes that file is written by a deliberate
+operator kernel upgrade (`playbooks/node-baseline.yml --tags kernel`) — unattended-upgrades is
+security-only with the kernel blacklisted, so it essentially never produces one. **The human gate is
+the sentinel, not the daemon.**
+
+## The problem this fixes
+
+Rebooting these three nodes was a manual, serialized, supervised procedure, and
+`roles/node_baseline` has to end its kernel run by shouting the rules at the operator: *"Reboot prod
+nodes ONE AT A TIME, verify each returns before touching the next, and NEVER node 2 first"*. That is
+a correct procedure and an error-prone one, and nothing enforced it.
+
+What it has to get right, measured on prod 2026-10-17:
+
+- **etcd quorum is 2 of 3** — all three nodes are control-plane+etcd. Two down is a lost cluster.
+- **Every drain degrades all 25 Longhorn volumes to 2/3**, because 3 nodes × 3 replicas ×
+  `replica-soft-anti-affinity: false` leaves nowhere to place a replacement, and rebuild is
+  serialized behind a 600s wait. Reboot the next node too early and volumes hit 1/3, where
+  `block-if-contains-last-replica` can block a drain forever.
+- **Six PodDisruptionBudgets allow zero voluntary disruptions** — 3 Longhorn `instance-manager-*`
+  (structurally 0 by design; this is how Longhorn gates the drain) and 3 CNPG `*-primary`.
+
+## Placement — why its own Kustomization
+
+`platform` is `wait: true` over twelve components and gates `platform-config` → every apps wave. A
+reboot daemon that failed to go healthy there would stall **every application in the cluster** on
+"dependency not ready". A component added to make maintenance safer must not be able to cause an
+outage. `wait: true` here is a health **signal** for something nothing depends on — and nothing
+should ever `dependsOn: [kured]`.
+
+The `dependsOn: [platform]` edge is load-bearing, not conventional: it supplies the
+`monitoring.coreos.com` CRDs this component's `PrometheusRule` needs (F-MON1 — a pass containing an
+unknown kind aborts), plus the Prometheus that kured **queries** before every reboot.
+
+## How concurrent reboots are made impossible
+
+| mechanism | value | why |
+|---|---|---|
+| cluster-wide lock | one annotation on kured's own DaemonSet | a node reboots only while holding it; enforced by API-server optimistic concurrency, not timing |
+| `concurrency` | `1` | one lock holder at a time |
+| **`lockTtl`** | **`0` (disabled)** | a TTL lets reboots continue after the holder dies — on 3 etcd nodes that is the catastrophe. A hung node holds the lock **forever** and nothing cascades. |
+| `lockReleaseDelay` | `30m` | the next node waits out Longhorn's 600s replenishment even if no alert latched |
+| alert gate | `alertFilterMatchOnly: true` + anchored allow-list | refuses to proceed while a volume is degraded/faulted or a node is down |
+| window | 03:00–06:00 daily, America/New_York | matches `renovate.json`'s window; daily (not weekly) is what makes `KuredRebootPending`'s `for: 36h` meaningful |
+
+## Deliberate non-choices
+
+- **No `--ignore-daemonsets` / `--delete-emptydir-data` / `--force`.** Those flags **do not exist**
+  in kured 1.23.0 — it hardcodes all three in the drain helper (`cmd/kured/main.go:438-449`).
+  Passing them would CrashLoop the DaemonSet. All three are genuinely required here (12 DaemonSets,
+  135 emptyDir pods).
+- **`rebootMethod: command`, not `signal`.** `signal` would allow `privileged: false` + `CAP_KILL`,
+  but needs a host-side `kured-reboot.service` that does not exist on these nodes. A real future
+  hardening step; explicitly not taken here.
+- **No tolerations.** No prod node is tainted (measured). A future tainted node would be silently
+  uncovered — hence `KuredNodeCoverageGap`.
+- **`NodeEtcdRequestsSlow` / `NodeIOPressure*` are NOT in the alert gate**, despite being exactly
+  what you would want to pause for. prod-1 has a documented etcd WAL I/O starvation history and
+  including them risks a permanent wedge that looks like safety.
+- **No Slack/shoutrrr notifications.** Would require a SOPS Secret and a decryption patch; the three
+  alerts cover the same ground through Alertmanager.
+
+## Versioning
+
+Chart pinned `>=6.1.0 <6.2.0`. The minor ceiling is **deliberate here**, unlike elsewhere in this
+repo where it is a main cause of pin rot: a kured minor may change drain or lock semantics, and this
+is the one component where "a human read the changelog" should be a precondition. `renovate.json`'s
+`blast-radius` rule additionally forbids auto-merging it, so the ceiling is a review prompt, not a
+wall.
+
+## What breaks silently, and what catches it
+
+Every failure mode of this component leaves a cluster that looks healthy — kured stays 1/1 Running
+while rebooting nothing. `alerts.yaml` ships three rules over kured's single metric, with a
+companion promtool suite:
+
+| alert | catches |
+|---|---|
+| `KuredNotReporting` | the scrape pipeline is dead (chart defaults `service.create`/`metrics.create` to **false**; a NetworkPolicy can time the scrape out) — and the other two are inert while this is true |
+| `KuredNodeCoverageGap` | kured missing on *some* node, which `absent()` cannot see |
+| `KuredRebootPending` | a reboot was offered and blocked for >36h |
+
+## Post-merge verification
+
+```bash
+infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds,pod,svc,servicemonitor -o wide'
+infra/devcontainer/dev.sh run bash -c 'flux --context k3s-prod get kustomization kured'
+infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured logs ds/kured --tail=40'
+```
+
+Expect `3/3` ready, the Kustomization `Ready=True`, and kured logging a reboot-check decision each
+period. Then confirm the scrape is real — this is the step that proves the alerts are not inert:
+
+```bash
+infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n monitoring exec \
+  sts/prometheus-kube-prometheus-stack-prometheus -c prometheus -- \
+  wget -qO- "http://localhost:9090/api/v1/query?query=count(kured_reboot_required)"'
+```
+
+**Expect `3`.** Anything else (and especially an empty result) means the Service, ServiceMonitor or
+NetworkPolicy is wrong and all three alerts are silently dead.
+
+**Before trusting the timer with a real kernel upgrade**, do one watched drain to settle the open
+question the design record flags: whether CNPG actually switches its primary away in response to a
+drain-induced eviction. If it does not, draining `k3s-prod-1` blocks on three 0-disruption PDBs at
+once.

platform/kured/alerts.promtool-test.yaml

@@ -0,0 +1,282 @@
+---
+# promtool unit tests for the three rules in platform/kured/alerts.yaml.
+#
+# WHY THIS FILE EXISTS. The rules it covers are the ONLY thing that can tell an operator that the
+# reboot automation has stopped working, and every failure mode they describe is silent — kured stays
+# 1/1 Running while rebooting nothing. An alert that cannot fire is therefore not a cosmetic defect
+# here, it is the removal of the only signal. This repo has shipped that bug in both directions
+# already (a permanently-firing VPNTunnelDown; a structurally inert VeleroNoRecentBackup over an
+# absent series), which is why the harness exists at all. Each rule below is pinned BOTH ways: quiet
+# on the healthy state, firing on the constructed bad one, with the `for:` boundary asserted rather
+# than assumed.
+#
+# HOW TO RUN. `platform/monitoring/tests/run.sh` — the one repo-wide rule-unit harness. It finds this
+# file by the COMPANION convention (`<manifest>.promtool-test.yaml` beside `<manifest>.yaml`), so the
+# manifest under test is exactly platform/kured/alerts.yaml and nothing else. Run just this file with
+#   platform/monitoring/tests/run.sh platform/kured/alerts.promtool-test.yaml
+# The harness flattens the PrometheusRule CR's `.spec.groups` into the bare rules file promtool wants,
+# `promtool check rules` it first so a non-parsing file cannot pass vacuously, and runs it all in a
+# temp dir. It leaves Flux's `${BASE_DOMAIN}` in the runbook_url annotations verbatim, which is safe
+# because the assertions below query the `ALERTS` series and never compare annotations.
+#
+# WHY THE ASSERTIONS QUERY `ALERTS` RATHER THAN USING `exp_alerts`. promtool's alert_rule_test
+# compares annotations VERBATIM, which would mean pasting each rule's description prose in here and
+# re-pasting it on every wording change. Querying the synthetic `ALERTS` series proves what actually
+# matters — the alert reaches `pending` and then `firing` across its `for:` boundary, with the right
+# labels — and stays decoupled from annotation text. Same choice as
+# platform/cnpg/alerts.promtool-test.yaml.
+#
+# THE SERIES SHAPES ARE REAL. `kured_reboot_required{node=...}` is a GaugeVec with exactly one label,
+# `node` — read from kured's cmd/kured/main.go:92-96 (the only registered metric in the project).
+# `kube_node_info{node=...}` was measured on prod Prometheus on 2026-10-17 returning
+# `count(kube_node_info) == 3`, matching the three nodes. The extra kube_node_info labels prod
+# carries are omitted deliberately: every rule aggregates with `count()`, so they cannot affect the
+# result, and inventing plausible-but-wrong values would be worse than leaving them out.
+#
+# evaluation_interval is 5m, not the usual 1m: KuredRebootPending has `for: 36h`, and at a 1m
+# interval promtool would evaluate that test over 2000+ steps for no added coverage.
+rule_files:
+  - alerts.rules.yaml
+evaluation_interval: 5m
+
+tests:
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  # 1. THE HEALTHY STEADY STATE. Three nodes, kured running on all three, nothing pending.
+  #    ALL THREE RULES MUST BE SILENT. This is the case that a badly-written `absent()` or a
+  #    `count() <` comparison with an off-by-one gets wrong, and it is also the state the cluster
+  #    is in ~100% of the time, so a false positive here would train the operator to ignore the
+  #    whole group.
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  - interval: 5m
+    name: healthy cluster - no reboots pending, kured on every node, all three rules quiet
+    input_series:
+      - series: 'kured_reboot_required{node="k3s-prod-1"}'
+        values: '0x60'
+      - series: 'kured_reboot_required{node="k3s-prod-2"}'
+        values: '0x60'
+      - series: 'kured_reboot_required{node="k3s-prod-3"}'
+        values: '0x60'
+      - series: 'kube_node_info{node="k3s-prod-1"}'
+        values: '1x60'
+      - series: 'kube_node_info{node="k3s-prod-2"}'
+        values: '1x60'
+      - series: 'kube_node_info{node="k3s-prod-3"}'
+        values: '1x60'
+    promql_expr_test:
+      # Nothing from this group may be pending or firing, at any point in the window.
+      - expr: 'count(ALERTS{alertname=~"Kured.*"})'
+        eval_time: 4h
+        exp_samples: []
+      # And the underlying expressions must each be genuinely false, not merely unevaluated:
+      # absent() yields no series because the vector is populated...
+      - expr: 'absent(kured_reboot_required)'
+        eval_time: 4h
+        exp_samples: []
+      # ...coverage is exact (3 == 3), so the `<` comparison drops the sample...
+      - expr: 'count(kured_reboot_required) < count(kube_node_info)'
+        eval_time: 4h
+        exp_samples: []
+      # ...and no node reports a pending reboot.
+      - expr: 'max by (node) (kured_reboot_required) == 1'
+        eval_time: 4h
+        exp_samples: []
+
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  # 2. KuredNotReporting — THE WHOLE PIPELINE IS DEAD. No kured series at all, which is what the
+  #    cluster looks like if the chart's `service.create`/`metrics.create` regress to their FALSE
+  #    defaults, if the ServiceMonitor is dropped, or if a NetworkPolicy silently times out the
+  #    scrape (the Longhorn precedent). The nodes still exist, so kube_node_info is present —
+  #    proving the rule keys on kured's absence and not on a dead cluster.
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  - interval: 5m
+    name: no kured series anywhere - KuredNotReporting fires after 30m
+    input_series:
+      - series: 'kube_node_info{node="k3s-prod-1"}'
+        values: '1x60'
+      - series: 'kube_node_info{node="k3s-prod-2"}'
+        values: '1x60'
+      - series: 'kube_node_info{node="k3s-prod-3"}'
+        values: '1x60'
+    promql_expr_test:
+      # The expression itself yields 1 immediately.
+      - expr: 'absent(kured_reboot_required)'
+        eval_time: 10m
+        exp_samples:
+          - labels: '{}'
+            value: 1
+      # `for: 30m` is honoured — still only pending at 25m.
+      - expr: 'ALERTS{alertname="KuredNotReporting"}'
+        eval_time: 25m
+        exp_samples:
+          - labels: 'ALERTS{alertname="KuredNotReporting", alertstate="pending", severity="warning"}'
+            value: 1
+      # ...and firing once past it.
+      - expr: 'ALERTS{alertname="KuredNotReporting", alertstate="firing"}'
+        eval_time: 35m
+        exp_samples:
+          - labels: 'ALERTS{alertname="KuredNotReporting", alertstate="firing", severity="warning"}'
+            value: 1
+      # THE GUARD CLAUSE: with kured absent, the other two rules are inert — exactly the reason
+      # this rule has to exist. KuredRebootPending cannot fire (no series to be 1) and
+      # KuredNodeCoverageGap cannot either, because `count()` over an empty vector returns NO
+      # SAMPLE rather than 0, so the `<` comparison has nothing to compare. Asserting that here
+      # pins the dependency: if a future edit made coverage-gap fire in this state, the overlap
+      # would be deliberate rather than accidental.
+      - expr: 'ALERTS{alertname="KuredRebootPending"}'
+        eval_time: 35m
+        exp_samples: []
+      - expr: 'ALERTS{alertname="KuredNodeCoverageGap"}'
+        eval_time: 35m
+        exp_samples: []
+      - expr: 'count(kured_reboot_required) < count(kube_node_info)'
+        eval_time: 35m
+        exp_samples: []
+
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  # 3. KuredNodeCoverageGap — kured COVERS ONLY TWO OF THREE NODES. k3s-prod-3's series stops at
+  #    1h and never returns (a node taint this DaemonSet does not tolerate, a node that never came
+  #    back from a reboot, a DaemonSet nodeSelector edit). This also pins the `for: 2h` choice,
+  #    which exists SO THAT AN ORDINARY REBOOT DOES NOT TRIP IT: the gap is real from ~1h05 but
+  #    the alert is only pending an hour later, and a healthy reboot is minutes.
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  - interval: 5m
+    name: kured missing on one node - KuredNodeCoverageGap fires after 2h, not before
+    input_series:
+      - series: 'kured_reboot_required{node="k3s-prod-1"}'
+        values: '0x60'
+      - series: 'kured_reboot_required{node="k3s-prod-2"}'
+        values: '0x60'
+      # Ends after 1h (13 samples at 5m) and is stale thereafter.
+      - series: 'kured_reboot_required{node="k3s-prod-3"}'
+        values: '0x12'
+      - series: 'kube_node_info{node="k3s-prod-1"}'
+        values: '1x60'
+      - series: 'kube_node_info{node="k3s-prod-2"}'
+        values: '1x60'
+      - series: 'kube_node_info{node="k3s-prod-3"}'
+        values: '1x60'
+    promql_expr_test:
+      # Before the gap opens, coverage is exact and nothing is pending.
+      - expr: 'ALERTS{alertname="KuredNodeCoverageGap"}'
+        eval_time: 30m
+        exp_samples: []
+      # The shortfall is real well before the alert fires — 2 < 3.
+      - expr: 'count(kured_reboot_required)'
+        eval_time: 2h
+        exp_samples:
+          - labels: '{}'
+            value: 2
+      # A SHORT OUTAGE DOES NOT ALERT: still only pending at 2h, an hour after the gap opened.
+      - expr: 'ALERTS{alertname="KuredNodeCoverageGap"}'
+        eval_time: 2h
+        exp_samples:
+          - labels: 'ALERTS{alertname="KuredNodeCoverageGap", alertstate="pending", severity="warning"}'
+            value: 1
+      # Firing once the gap has persisted past `for: 2h`.
+      - expr: 'ALERTS{alertname="KuredNodeCoverageGap", alertstate="firing"}'
+        eval_time: 3h15m
+        exp_samples:
+          - labels: 'ALERTS{alertname="KuredNodeCoverageGap", alertstate="firing", severity="warning"}'
+            value: 1
+      # KuredNotReporting must stay QUIET: two nodes are still publishing, so the vector is not
+      # absent. This is the distinction between "kured is gone" and "kured is incomplete", and
+      # getting it wrong in either direction would collapse two different diagnoses into one.
+      - expr: 'ALERTS{alertname="KuredNotReporting"}'
+        eval_time: 3h15m
+        exp_samples: []
+
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  # 4. KuredRebootPending — A REBOOT IS BLOCKED. k3s-prod-1 reports a pending reboot continuously:
+  #    the sentinel was written (an operator kernel upgrade) and the reboot has not happened across
+  #    more than one 03:00-06:00 window. Asserts the 36h boundary in both directions, which is the
+  #    assertion that makes the DAILY-window decision load-bearing rather than cosmetic: on a
+  #    weekly window this threshold could not distinguish "blocked" from "waiting for Sunday".
+  #    Series run to 38h20m (456 samples at 5m) so the post-boundary evaluation has real data.
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  - interval: 5m
+    name: reboot pending on one node for over 36h - KuredRebootPending fires, per-node label kept
+    input_series:
+      - series: 'kured_reboot_required{node="k3s-prod-1"}'
+        values: '1x456'
+      - series: 'kured_reboot_required{node="k3s-prod-2"}'
+        values: '0x456'
+      - series: 'kured_reboot_required{node="k3s-prod-3"}'
+        values: '0x456'
+      - series: 'kube_node_info{node="k3s-prod-1"}'
+        values: '1x456'
+      - series: 'kube_node_info{node="k3s-prod-2"}'
+        values: '1x456'
+      - series: 'kube_node_info{node="k3s-prod-3"}'
+        values: '1x456'
+    promql_expr_test:
+      # The expression selects ONLY the pending node — the operator needs to know which box.
+      - expr: 'max by (node) (kured_reboot_required) == 1'
+        eval_time: 1h
+        exp_samples:
+          - labels: '{node="k3s-prod-1"}'
+            value: 1
+      # A NORMAL WAIT IS NOT AN ALERT. At 24h the node is still inside its legitimate
+      # once-per-day window wait, so the alert may be pending but must NOT be firing.
+      - expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}'
+        eval_time: 24h
+        exp_samples: []
+      # Still not firing just short of the boundary.
+      - expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}'
+        eval_time: 35h30m
+        exp_samples: []
+      # Firing past it, carrying the node label through to the notification.
+      - expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}'
+        eval_time: 36h30m
+        exp_samples:
+          - labels: 'ALERTS{alertname="KuredRebootPending", alertstate="firing", node="k3s-prod-1", severity="warning"}'
+            value: 1
+      # The two healthy nodes must NOT be alerted on. A rule written with `kured_reboot_required`
+      # unaggregated, or compared with `>= 0`, would raise all three and bury the real one.
+      - expr: 'count(ALERTS{alertname="KuredRebootPending", alertstate="firing"})'
+        eval_time: 36h30m
+        exp_samples:
+          - labels: '{}'
+            value: 1
+      # Liveness rules stay quiet: kured is present and covering all three nodes. A pending reboot
+      # is NOT a wiring fault and must not be reported as one.
+      - expr: 'ALERTS{alertname="KuredNotReporting"}'
+        eval_time: 36h30m
+        exp_samples: []
+      - expr: 'ALERTS{alertname="KuredNodeCoverageGap"}'
+        eval_time: 36h30m
+        exp_samples: []
+
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  # 5. THE RULE MUST STOP. A reboot that SUCCEEDS clears the sentinel, and the alert must clear
+  #    with it — the other half of the failure space this harness exists for (a permanently-firing
+  #    rule is how VPNTunnelDown taught this repo the lesson). k3s-prod-1 is pending for ~40h,
+  #    long enough to fire, then the reboot happens and the gauge returns to 0.
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  - interval: 5m
+    name: reboot completes after firing - KuredRebootPending resolves
+    input_series:
+      # 1 for 480 samples (40h), then 0 for the remainder.
+      - series: 'kured_reboot_required{node="k3s-prod-1"}'
+        values: '1x480 0x120'
+      - series: 'kured_reboot_required{node="k3s-prod-2"}'
+        values: '0x600'
+      - series: 'kured_reboot_required{node="k3s-prod-3"}'
+        values: '0x600'
+      - series: 'kube_node_info{node="k3s-prod-1"}'
+        values: '1x600'
+      - series: 'kube_node_info{node="k3s-prod-2"}'
+        values: '1x600'
+      - series: 'kube_node_info{node="k3s-prod-3"}'
+        values: '1x600'
+    promql_expr_test:
+      # Firing while the reboot is still outstanding.
+      - expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}'
+        eval_time: 39h
+        exp_samples:
+          - labels: 'ALERTS{alertname="KuredRebootPending", alertstate="firing", node="k3s-prod-1", severity="warning"}'
+            value: 1
+      # Cleared once the sentinel is gone. If this assertion ever fails, the rule has become
+      # self-sustaining and the operator will start ignoring it.
+      - expr: 'ALERTS{alertname=~"Kured.*"}'
+        eval_time: 45h
+        exp_samples: []

platform/kured/alerts.yaml

@@ -0,0 +1,124 @@
+---
+# Alert rules for the kured reboot automation.
+#
+# WHY THESE THREE, AND WHY THEY ARE NOT OPTIONAL. Reboot automation's characteristic failure is
+# SILENCE, not noise. Every way this component can break leaves a cluster that looks perfectly
+# healthy: a sentinel nothing writes, a Service the chart did not create, a NetworkPolicy that drops
+# the scrape, a taint that excludes a node, a drain wedged on a PodDisruptionBudget, a cluster lock
+# held forever by a node that hung on warm reboot. In none of those cases does anything turn red on
+# its own. kured itself stays 1/1 Running throughout.
+#
+# kured EXPOSES EXACTLY ONE METRIC. Read from cmd/kured/main.go:92-96 and :113 on 2026-10-17 — the
+# only `prometheus.MustRegister` in the tree is a GaugeVec `Subsystem: "kured"`, `Name:
+# "reboot_required"`, labelled `{node}`. So the full vocabulary available is
+# `kured_reboot_required{node="..."}`, 1 when the sentinel exists and 0 when it does not. Each rule
+# below is built from that one series plus `kube_node_info` (kube-state-metrics), and nothing else.
+# There is deliberately no rule over an invented metric name: this repo has shipped a
+# structurally-inert alert before (VeleroNoRecentBackup over an absent series, which is why
+# platform/monitoring/tests/run.sh exists), and an alert that cannot fire is worse than no alert
+# because it also removes the suspicion that would have caught it.
+#
+# THESE RULES LIVE HERE, NOT IN platform/monitoring/alerts.yaml, for two reasons. (1) The unit-test
+# harness treats a `<manifest>.promtool-test.yaml` COMPANION as covering just its sibling, whereas a
+# `tests/*.test.yaml` SUITE covers every PrometheusRule in its directory — so shipping them beside
+# the component keeps the test obligation local and legible (the platform/cnpg precedent). (2)
+# platform/monitoring/alerts.yaml is concurrently edited by other open branches; adding 3 rules to
+# its tail would create a merge conflict for no benefit.
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: kured
+  namespace: kured
+  labels:
+    app.kubernetes.io/name: kured
+spec:
+  groups:
+    - name: platform-health.node-reboots
+      rules:
+        # ── 1. IS THE MECHANISM ALIVE AT ALL? ────────────────────────────────────────────────
+        - alert: KuredNotReporting
+          # `absent()` yields a series only when the ENTIRE vector is empty, so this fires when no
+          # node at all is publishing — kured gone, or its Service/ServiceMonitor/NetworkPolicy
+          # regressed so nothing is scraped. It stays quiet while a single node is mid-reboot
+          # (the other two still report), which is what makes it specific to a wiring failure
+          # rather than to a reboot in progress.
+          #
+          # THIS IS THE RULE THAT GUARDS THE OTHER TWO. Both of them read `kured_reboot_required`,
+          # so if that series stops existing they go quiet and look like good news. This one turns
+          # the absence itself into the signal. The specific regression it is aimed at: the kured
+          # chart defaults `service.create` and `metrics.create` to FALSE, and Longhorn has already
+          # demonstrated on this cluster (see the companion note in
+          # conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh) that a
+          # NetworkPolicy can let targets be DISCOVERED while every scrape times out and the TSDB
+          # stays empty.
+          expr: absent(kured_reboot_required)
+          # 30m absorbs a HelmRelease upgrade rolling the DaemonSet and a Prometheus restart,
+          # without tolerating a real wiring break for more than one quiet night's worth of time.
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: kured is publishing no reboot_required series on any node
+            description: >-
+              No node is reporting kured_reboot_required, so the reboot automation is either not
+              running or not being scraped. While this is true, OS reboots are not happening and
+              the two alerts that would say so (KuredRebootPending, KuredNodeCoverageGap) are
+              themselves inert. Nodes will silently accumulate pending kernel upgrades.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-reboot-automation"
+
+        # ── 2. IS IT ALIVE ON *EVERY* NODE? ──────────────────────────────────────────────────
+        - alert: KuredNodeCoverageGap
+          # A per-node agent that silently covers only some nodes is the subtler half of the
+          # liveness question, and `absent()` cannot see it. kured publishes one series per node it
+          # runs on; kube-state-metrics publishes one `kube_node_info` per node that exists
+          # (measured 2026-10-17: 3, matching the 3 nodes). A shortfall means at least one node has
+          # no reboot automation — most likely a new or re-provisioned node carrying a taint this
+          # DaemonSet does not tolerate (it declares none, because no prod node is tainted today).
+          # That node would then never reboot, with nothing else to indicate it.
+          expr: count(kured_reboot_required) < count(kube_node_info)
+          # 2h, chosen to sit comfortably ABOVE the duration of a healthy reboot. A node being
+          # rebooted stops publishing for a few minutes and legitimately creates a shortfall; a
+          # node that is still missing after two hours is either genuinely uncovered or has hung on
+          # reboot, and both of those want a human.
+          for: 2h
+          labels:
+            severity: warning
+          annotations:
+            summary: kured is not running on every node
+            description: >-
+              Fewer nodes are reporting kured_reboot_required than exist in the cluster, so at
+              least one node has no reboot automation and will never be rebooted by it. Most
+              likely a node carries a taint this DaemonSet does not tolerate, or has hung during
+              a reboot.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-reboot-automation"
+
+        # ── 3. IS A REBOOT STUCK? ────────────────────────────────────────────────────────────
+        - alert: KuredRebootPending
+          # A node has wanted a reboot for longer than a full maintenance window plus margin, so
+          # the reboot was OFFERED and did not happen. The causes are all real and all invisible
+          # otherwise: the Prometheus alert gate is blocking on a degraded Longhorn volume that is
+          # not recovering; the drain hit `drainTimeout` against a 0-disruption PDB; another node
+          # holds the cluster lock and never released it (`lockTtl: 0` means a lock held by a hung
+          # node is held FOREVER — that is deliberate, see helmrelease.yaml, and this is the alert
+          # that makes the resulting stall visible).
+          expr: max by (node) (kured_reboot_required) == 1
+          # 36h, and the window being DAILY is what makes this number meaningful — see the window
+          # block in helmrelease.yaml. With a 03:00-06:00 window every day, a pending reboot waits
+          # at most ~24h legitimately, so 36h means at least one full window elapsed without the
+          # reboot completing. On a weekly window this threshold would have to exceed 7 days, which
+          # is longer than this cluster's 7d Prometheus retention and is reset by any Prometheus
+          # restart past the 1h for-outage-tolerance — i.e. unreliable in the one direction that
+          # matters. Note this alert can still be LATE for that reason, but it cannot be spurious.
+          for: 36h
+          labels:
+            severity: warning
+          annotations:
+            summary: Node {{ $labels.node }} has needed a reboot for over 36h
+            description: >-
+              kured reports a pending reboot on {{ $labels.node }} that has survived at least one
+              full 03:00-06:00 maintenance window. The reboot is being blocked, not merely
+              waiting: likely a degraded Longhorn volume holding the Prometheus alert gate, a
+              drain timing out against a PodDisruptionBudget, or a stale cluster lock held by a
+              node that did not come back. The kernel or security update that needed the reboot is
+              installed but NOT running.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-reboot-automation"

platform/kured/helmrelease.yaml

@@ -0,0 +1,289 @@
+---
+# kured (Kubernetes Reboot Daemon) — coordinated, one-node-at-a-time reboots of the three prod
+# control-plane nodes.
+#
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# READ THIS FIRST: kured DOES NOT DECIDE *WHETHER* TO REBOOT. IT ONLY DECIDES *WHEN* AND *HOW*.
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# kured reboots a node if and only if the file `/var/run/reboot-required` exists on it. On these
+# nodes exactly one thing creates that file: a deliberate operator run of
+# `playbooks/node-baseline.yml --tags kernel` (which advances the backports kernel). The
+# unattended-upgrades role this PR also adds installs SECURITY updates only and BLACKLISTS
+# `linux-image-*`/`linux-headers-*`, so it can never produce a reboot-required sentinel on its own —
+# see infra/ansible/roles/unattended_upgrades/defaults/main.yml for that argument in full.
+#
+# That is the whole answer to this repo's two standing objections to reboot automation:
+#
+#   * infra/ansible/roles/node_baseline/tasks/main.yml:61-67 — "IT DOES NOT REBOOT." k3s-prod-2 hung
+#     on a warm reboot on 2026-09-03 (powered, off the LAN ~38 minutes, physical power button
+#     required), so rebooting was made "a SERIALIZED, SUPERVISED operator step ... and a role cannot
+#     enforce that." STILL TRUE, and nothing here changes it: node_baseline continues not to reboot.
+#   * docs/plans/cluster-upgrade-plan.md:291-299 — system-upgrade-controller was rejected because it
+#     "automates away the human gate between nodes", and the operator's hard requirement is an
+#     explicit PASS/STOP verification BETWEEN every node.
+#
+# THE HUMAN GATE IS PRESERVED, AND IT IS THE SENTINEL. The operator decides whether a kernel moves
+# and on which hosts (`--limit k3s-prod-1`), which is what writes the sentinel on those hosts and
+# only those hosts. kured then performs the cordon/drain/reboot/uncordon that the operator would
+# otherwise have typed by hand — the error-prone mechanical half, including the ordering rule
+# node_baseline's own message has to shout ("NEVER node 2 first"). Run the kernel play one host at a
+# time and the operator keeps per-node PASS/STOP control exactly as before; run it on all three and
+# kured still does them strictly one at a time, just in an order it chooses. Both are safe for
+# quorum; only the first preserves the ordering preference. See runbooks/node-reboot-automation.md.
+#
+# cluster-upgrade-plan.md:329-330 and :545-548 name the condition under which its own rejection
+# reverses — "SUC becomes viable once the cluster is on fast disks ... where drain-driven rebuild is
+# cheap" — and prod is that cluster. This component takes that clause up, but NOT its drain
+# assumption: see the Longhorn section below, where drain-driven rebuild is measurably NOT free even
+# on SSD, and is gated on explicitly rather than waved through.
+#
+# WHAT IS STILL UNMITIGATED, STATED PLAINLY: kured cannot press a power button. If a node hangs on
+# warm reboot the way k3s-prod-2 did in September, kured will not recover it. The blast radius is
+# bounded by design rather than by luck — `concurrency: 1` plus `lockTtl: 0` means the hung node
+# keeps the cluster lock forever, so NO second node is ever rebooted behind it; etcd keeps 2/3 quorum
+# and the cluster stays up, degraded, until a human attends. The KuredRebootPending alert in
+# alerts.yaml is what tells them. That is strictly better than the previous automation story (none)
+# but it is NOT the same as "safe to ignore at 3am on a Sunday".
+apiVersion: helm.toolkit.fluxcd.io/v2
+kind: HelmRelease
+metadata:
+  name: kured
+  namespace: kured
+spec:
+  interval: 30m
+  chart:
+    spec:
+      chart: kured
+      # Chart 6.1.0 -> appVersion 1.23.0 (published 2026-06-30; newest of the 80 entries in
+      # https://kubereboot.github.io/charts/index.yaml, read 2026-10-17).
+      #
+      # THE `<6.2.0` CEILING IS A DELIBERATE CHOICE HERE, NOT THE REFLEX IT IS ELSEWHERE. This PR's
+      # own audit found that 9 of 12 HelmReleases in this repo pin a minor-locking range, and that
+      # this is a MAIN CAUSE of pin rot: Renovate can then only ever offer patches inside a line it
+      # has been told not to leave (kube-prometheus-stack is stuck 4 chart majors back that way).
+      # For a component whose job is to reboot control-plane nodes, that ceiling is worth paying on
+      # purpose: a kured MINOR may change drain or lock semantics, and this is the one component
+      # where "a human looked at the changelog" should be a precondition, not a nice-to-have.
+      # Renovate will raise the minor as a non-automerged PR (see the kured packageRule in
+      # renovate.json), so the ceiling is a review prompt, not a wall.
+      version: ">=6.1.0 <6.2.0"
+      sourceRef:
+        kind: HelmRepository
+        name: kured
+        namespace: kured
+  install:
+    remediation:
+      retries: 3
+  upgrade:
+    remediation:
+      retries: 3
+  values:
+    # ── IDENTITY / PLACEMENT ───────────────────────────────────────────────────────────────────
+    # No tolerations. Measured 2026-10-17: all three prod nodes carry `spec.taints` EMPTY (k3s does
+    # not taint its servers), so a DaemonSet reaches every node untainted. If a future node IS
+    # tainted, kured silently stops covering it and that node then never reboots — invisible,
+    # because the other two keep reporting. KuredNodeCoverageGap in alerts.yaml is the detector: it
+    # compares kured's per-node series count against the node count rather than merely asking
+    # whether kured exists.
+    priorityClassName: system-node-critical
+    # The chart's defaults are kept: `hostPID: true` and `containerSecurityContext.privileged: true`
+    # with `readOnlyRootFilesystem: true`. Both are load-bearing and neither is reducible:
+    # `rebootMethod: command` runs the reboot by nsenter'ing /proc/1/ns/mnt, which needs the host PID
+    # namespace to see PID 1 and the privilege to enter its mount namespace. The documented
+    # lower-privilege alternative (privileged:false + CAP_KILL) applies ONLY to
+    # `rebootMethod: signal`, which signals a host-side kured-reboot.service that does not exist on
+    # these nodes. Choosing `signal` would mean shipping and maintaining that unit via Ansible for a
+    # capability reduction on a DaemonSet that is already cluster-admin-adjacent by function; it is a
+    # reasonable future hardening step and is explicitly NOT taken here.
+    resources:
+      requests:
+        cpu: 10m
+        memory: 32Mi
+      limits:
+        # No CPU limit on purpose (the repo's node-pressure work treats CPU limits on tiny system
+        # agents as a throttling hazard). Memory is capped: kured holds no state beyond one
+        # annotation and a gauge, and 128Mi is ~4x its idle RSS upstream reports.
+        memory: 128Mi
+
+    configuration:
+      # ── THE QUORUM CONSTRAINT: TWO NODES MUST NEVER BE DOWN AT ONCE ──────────────────────────
+      # All three prod nodes are control-plane + etcd (measured 2026-10-17: every node reports
+      # `control-plane,etcd`). etcd quorum is 2 of 3, so one node down is degraded-but-up and two
+      # nodes down is a LOST CLUSTER that does not self-heal. Everything in this block exists to
+      # make the second case structurally unreachable rather than merely unlikely.
+      #
+      # concurrency: 1 — kured's cluster-wide lock is a single annotation
+      # (`weave.works/kured-node-lock`) on kured's OWN DaemonSet object, and a node reboots only
+      # while it holds that lock. One lock + concurrency 1 = one reboot, enforced by the API server's
+      # optimistic concurrency rather than by timing. This is the chart default; it is written out
+      # explicitly because it is the single most important value in this file.
+      concurrency: 1
+      # lockTtl: 0 (DISABLED) — THIS IS A SAFETY DECISION, NOT A DEFAULT WE FORGOT TO TUNE.
+      # A TTL makes the lock EXPIRE, which exists for the case "the lock holder died and we want
+      # reboots to continue". On a 3-node etcd cluster that is precisely the catastrophe: if node A
+      # has hung on a warm reboot (which has happened here — see the header) and its lock expires,
+      # node B becomes eligible and takes the cluster from 2/3 to 1/3. We would rather the reboot
+      # cycle STOP FOREVER and page a human than have it step over a corpse. The cost is that a
+      # genuinely stale lock needs manual clearing; runbooks/node-reboot-automation.md has the
+      # procedure, and KuredRebootPending is the alert that surfaces it.
+      lockTtl: 0
+      # lockReleaseDelay: 30m — hold the lock for 30 minutes AFTER this node is back and uncordoned,
+      # before any other node may take it. See the Longhorn section: this is the STRUCTURAL half of
+      # the inter-node gate, deliberately not dependent on an alert's `for:` timer having latched.
+      lockReleaseDelay: 30m
+
+      # ── THE MAINTENANCE WINDOW: 03:00-06:00 EVERY DAY, America/New_York ──────────────────────
+      # The timezone and the 03:00-06:00 band match renovate.json's existing
+      # `"schedule": ["after 3am and before 6am"]` / `"timezone": "America/New_York"`, so every
+      # unattended change in this repo lands in the same quiet 3-hour window.
+      #
+      # DAILY, NOT WEEKLY, AND THE REASON IS THE ALERT — not impatience. A weekly (`- sun`) window
+      # was the first draft and it breaks the only alert that can tell you this mechanism has
+      # stalled. With a Sunday-only window, a kernel upgraded on a Monday legitimately waits six
+      # days, so "a reboot has been pending too long" cannot be distinguished from "it is Tuesday"
+      # until the threshold exceeds a full 7-day cycle. That forces `for: 8d`+, which is longer than
+      # this cluster's Prometheus retention (MONITORING_PROMETHEUS_RETENTION is 7d) and is reset by
+      # any Prometheus restart beyond the 1h default for-outage-tolerance — i.e. the alert would be
+      # unreliable in exactly the direction that matters. A daily window caps the legitimate wait at
+      # under 24h, which makes KuredRebootPending's `for: 36h` both robust and meaningful: 36 hours
+      # with a sentinel still set means at least one full window passed and SOMETHING BLOCKED IT.
+      #
+      # Daily does NOT weaken the quorum guarantee — that is `concurrency: 1` plus the lock, not the
+      # calendar. It does mean a 3-node kernel rollout can finish in one to three nights instead of
+      # three weeks, with each node still gated on the previous one's Longhorn rebuilds (below).
+      rebootDays:
+        - su
+        - mo
+        - tu
+        - we
+        - th
+        - fr
+        - sa
+      startTime: "03:00"
+      endTime: "06:00"
+      timeZone: America/New_York
+      # How often to re-evaluate (sentinel + window + alerts + lock). 1h is the chart default; at a
+      # 3h window that is three evaluation opportunities per node per night.
+      period: 1h
+
+      # ── THE SENTINEL ─────────────────────────────────────────────────────────────────────────
+      # Written ONLY by a deliberate operator kernel upgrade (see the header). `update-notifier-common`
+      # — the package that owns /etc/kernel/postinst.d and actually creates this file on Debian — was
+      # measured ABSENT on all three nodes on 2026-10-17, which is why the unattended_upgrades role
+      # installs it. Without that package nothing ever writes this path, kured correctly never
+      # reboots, every probe looks healthy, and the entire mechanism is a silent no-op. That failure
+      # mode is the reason KuredNotReporting exists.
+      #
+      # THE PATH kured LOGS IS NOT THIS PATH, and that trips people debugging it. The chart
+      # defaults `useRebootSentinelHostPath: true` (kept), which bind-mounts the DIRNAME of this
+      # value onto /sentinel read-only and renders the flag as
+      # `--reboot-sentinel=/sentinel/reboot-required` (daemonset.yaml:134). So the container watches
+      # /sentinel/reboot-required and the host file is /var/run/reboot-required. Same file, two
+      # names. Set this value as the HOST path, as below.
+      rebootSentinel: /var/run/reboot-required
+
+      # ── THE LONGHORN GATE: WHY POD READINESS IS NOT A SUFFICIENT INTER-NODE SIGNAL ────────────
+      # Measured on prod 2026-10-17, and this is the finding that shaped this file:
+      #   * 25 Longhorn volumes, ALL `numberOfReplicas: 3`, all healthy, with exactly 25 replicas on
+      #     each of the 3 nodes. So there is no single-replica volume to lose today.
+      #   * BUT `replica-soft-anti-affinity: false` (replicas may not share a node) and
+      #     `disable-scheduling-on-cordoned-node: true`. With 3 nodes and 3 replicas, cordoning ONE
+      #     node leaves nowhere to place a replacement — so every drain takes ALL 25 volumes to
+      #     degraded 2/3 for the whole reboot.
+      #   * Recovery is slow and serialized: `replica-replenishment-wait-interval: 600` (10 minutes
+      #     before a rebuild even starts) and `concurrent-replica-rebuild-per-node-limit: 1`.
+      # Therefore: if the next node is rebooted before rebuilds finish, those volumes go to 1/3, and
+      # the drain after that genuinely CAN hit Longhorn's `node-drain-policy:
+      # block-if-contains-last-replica` and block forever. The inter-node gate must be VOLUME
+      # ROBUSTNESS, not "are the pods Ready again".
+      #
+      # It is gated twice, on purpose, because each half covers the other's blind spot:
+      #   1. lockReleaseDelay: 30m (above) — unconditional. Covers the 600s replenishment wait with
+      #      margin and does not depend on any alert having fired.
+      #   2. The Prometheus alert gate below — semantic. Covers a rebuild that takes LONGER than 30
+      #      minutes (25 serialized rebuilds on one node can), which a fixed delay cannot.
+      prometheusUrl: http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090
+      # Only consider alerts that are FIRING, not merely pending. A `pending` alert is a 10-minute
+      # guess; blocking a weekly reboot on one would make the gate fire on noise.
+      alertFiringOnly: true
+      # ── alertFilterMatchOnly: true IS LOAD-BEARING. DO NOT DROP IT. ──────────────────────────
+      # kured's two alert modes are near-opposites and the flag name does not advertise it:
+      #   * false (the chart default): alertFilterRegexp is an IGNORE list — kured blocks on ANY
+      #     firing alert except those matching.
+      #   * true: alertFilterRegexp is an ALLOW list — kured blocks ONLY on alerts that match.
+      # We need `true`, because this cluster always has something firing. Measured via the Prometheus
+      # API on 2026-10-17, these five were firing with no reboot pending and no end in sight:
+      # CNPGPrimariesCoLocated, KubeDeploymentReplicasMismatch, KubeDeploymentRolloutStuck,
+      # KubePodNotReady, OAuth2ProxyGateUnobserved (the last three all trace to the pm-agent pod,
+      # ImagePullBackOff on a placeholder digest for 12 days). Under the default `false`, kured would
+      # have been blocked by those on day one and would never have rebooted anything — a silent
+      # no-op indistinguishable, from outside, from "no reboots were needed".
+      alertFilterMatchOnly: true
+      # The ALLOW list: the conditions under which rebooting ANOTHER node risks data or quorum.
+      # Anchored and explicit — a prefix match here would silently re-admit the chronic five above.
+      #   LonghornVolumeDegraded  — the inter-node gate (see above). `longhorn_volume_robustness
+      #                             {state="degraded"} == 1`, for: 10m, in platform/monitoring/alerts.yaml.
+      #   LonghornVolumeFaulted   — data already at risk; never reboot into this.
+      #   KubeNodeNotReady        — a node is already down. Rebooting a second one loses quorum. This
+      #                             is the single most important entry for the etcd constraint.
+      #   NodeExporterDown        — a node we cannot see the health of. Treated as "assume the worst".
+      # DELIBERATELY NOT INCLUDED: NodeEtcdRequestsSlow and the NodeIOPressure* family. They are
+      # exactly the conditions you would want to pause for, but prod-1 has a documented history of
+      # etcd WAL I/O starvation, so including them risks a permanent wedge that looks like safety.
+      # Revisit once those are quiet for a sustained period; KuredRebootPending makes a wedge visible
+      # either way.
+      alertFilterRegexp: ^(LonghornVolumeDegraded|LonghornVolumeFaulted|KubeNodeNotReady|NodeExporterDown)$
+
+      # ── THE DRAIN ────────────────────────────────────────────────────────────────────────────
+      # NOTE ON FLAGS THAT DO NOT EXIST: there is no `--ignore-daemonsets`, no
+      # `--delete-emptydir-data` and no `--force` in kured 1.23.0. Read from
+      # cmd/kured/main.go:438-449 on 2026-10-17, kured HARDCODES
+      # `IgnoreAllDaemonSets: true`, `DeleteEmptyDirData: true` and `Force: true` in the drain
+      # helper. Passing them as extraArgs would CrashLoop the DaemonSet on an unknown flag. This
+      # matters because all three are genuinely REQUIRED here and it is easy to assume they must be
+      # configured: 12 DaemonSets and 135 emptyDir-carrying pods were measured on 2026-10-17, and a
+      # drain without those three settings refuses to start.
+      #
+      # drainTimeout: 15m — the chart/kured default is 0, meaning INFINITE, and infinite is the wrong
+      # answer here. Measured 2026-10-17, six PodDisruptionBudgets currently allow ZERO voluntary
+      # disruptions, and a plain drain loops on them indefinitely:
+      #   * 3x longhorn-system/instance-manager-* — one per node, `minAvailable: 1` against a
+      #     selector pinned to a single node's single pod, so disruptionsAllowed is STRUCTURALLY 0
+      #     forever. This is the mechanism by which Longhorn's node-drain-policy blocks: the
+      #     longhorn-manager deletes the node's instance-manager PDB only once it judges the drain
+      #     safe. The drain is SUPPOSED to wait on this one.
+      #   * 3x CNPG `<cluster>-primary` (keycloak-db, nextcloud-db, semaphore-db) — `minAvailable: 1`
+      #     against a selector matching exactly one pod (the primary), so again structurally 0 until
+      #     the CNPG operator switches the primary away. All three primaries were on k3s-prod-1.
+      # With a timeout, a drain that is genuinely stuck ABORTS, the lock is released, and kured
+      # retries next period instead of holding the cluster lock forever on a wedged node. 15m is
+      # chosen to be longer than a CNPG switchover and longer than the `unifi` VM's eviction (below)
+      # usually needs, while still being well inside the 3h window.
+      drainTimeout: 15m
+      # Taint a node PreferNoSchedule while its reboot is pending, so the scheduler stops placing new
+      # pods on a node that is about to go away. With concurrency 1 this is a small win (there is no
+      # second rebooting node to shed pods onto) but it is free and it shortens the drain.
+      preferNoScheduleTaint: weave.works/kured-node-reboot
+      # Record reboot state on the Node objects (`weave.works/kured-reboot-in-progress`,
+      # `weave.works/kured-most-recent-reboot-needed`). Off by default. Turned on because the only
+      # other observability kured offers is ONE gauge, and during an incident `kubectl get node -o
+      # yaml` is the thing an operator can reach without Prometheus.
+      annotateNodes: true
+      logFormat: json
+
+    # ── METRICS: BOTH OF THESE MUST BE true OR THE ALERTS IN alerts.yaml CANNOT FIRE ───────────
+    # The chart ships `service.create: false` AND `metrics.create: false`. Leaving either at its
+    # default means no Service, or a ServiceMonitor with nothing to select, so `kured_reboot_required`
+    # is never scraped — and a PrometheusRule over a series that no target produces is INERT: quiet
+    # forever, indistinguishable from healthy. This repo has shipped that exact bug twice
+    # (VeleroNoRecentBackup over an absent series; the inert gotk_* rules in
+    # runbooks/image-automation.md), which is why platform/monitoring/tests/run.sh exists at all.
+    # KuredNotReporting (alerts.yaml) is the standing detector for this pair regressing.
+    # Prometheus selects ServiceMonitors cluster-wide: measured 2026-10-17, `serviceMonitorSelector`
+    # and `serviceMonitorNamespaceSelector` are both `{}`, so no label plumbing is needed here.
+    service:
+      create: true
+      port: 8080
+    metrics:
+      create: true
+      interval: 60s

platform/kured/helmrepository.yaml

@@ -0,0 +1,12 @@
+---
+# HelmRepository for the official kured chart (github.com/kubereboot/charts). Renovate maintains the
+# pinned chart version in helmrelease.yaml via its `flux` manager, whose managerFilePatterns already
+# cover `platform/**` (renovate.json). Same Flux source pattern as platform/headlamp.
+apiVersion: source.toolkit.fluxcd.io/v1
+kind: HelmRepository
+metadata:
+  name: kured
+  namespace: kured
+spec:
+  interval: 1h
+  url: https://kubereboot.github.io/charts

platform/kured/kustomization.yaml

@@ -0,0 +1,36 @@
+---
+# platform/kured — the Kubernetes Reboot Daemon.
+#
+# Rendered by its OWN Flux Kustomization (clusters/base/kured.yaml, dependsOn platform), NOT by a
+# line in platform/kustomization.yaml. The reasoning is the same one the Cilium split-out wrote down
+# and intel-gpu-plugin reused: inside the shared `wait: true` platform tier, this component's health
+# would be indistinguishable from eleven others, the only emergency valve would be suspending the
+# whole tier, and that tier's 5m timeout is wrong for a 3-node DaemonSet rollout. It matters more
+# than usual here, because `platform` gates platform-config which gates every apps wave — so a wedged
+# reboot daemon placed there could stall every application in the cluster. A component whose entire
+# purpose is to make maintenance safer must not be able to cause an outage.
+#
+# `namespace: kured` applies to every resource below, so none of them repeats it. Keep it: the
+# CiliumNetworkPolicy in networkpolicy.yaml is a CR whose namespace kustomize would otherwise leave
+# to chance.
+#
+# NO SECRET OF ANY KIND. kured authenticates to the API server with its own ServiceAccount token and
+# talks to nothing off-cluster — no registry credential, no webhook URL, no notification token. So
+# neither clusters/dev nor clusters/prod adds a SOPS decryption patch for this tier (same as
+# homepage/runbooks/intel-gpu-plugin). If Slack or shoutrrr notifications are ever wanted, the chart
+# takes them via `configuration.notifyUrl`, and THAT would need a SOPS-encrypted Secret plus a
+# decryption patch — do not inline a webhook URL here.
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: kured
+resources:
+  - namespace.yaml
+  - helmrepository.yaml
+  - helmrelease.yaml     # chart installs the DaemonSet / RBAC / SA / Service / ServiceMonitor
+  # default-deny baseline + the Prometheus scrape ingress + the apiserver/Prometheus egress.
+  # The scrape ingress is NOT optional decoration: without it the ServiceMonitor's targets are
+  # discovered and every scrape times out, leaving the TSDB empty and all three rules in alerts.yaml
+  # permanently quiet. See the header in that file for the measured Longhorn precedent.
+  - networkpolicy.yaml
+  # Three rules over kured's single metric, with a companion promtool unit test beside them.
+  - alerts.yaml

platform/kured/namespace.yaml

@@ -0,0 +1,34 @@
+---
+# kured — namespace for the Kubernetes Reboot Daemon DaemonSet.
+#
+# PLATFORM INFRA, NOT AN APPLICATION NAMESPACE, and here that distinction is the difference between
+# a working component and an admission rejection. kured runs `hostPID: true` with a `privileged: true`
+# container (it `nsenter`s /proc/1/ns/mnt to run `/bin/systemctl reboot` on the host). Both are
+# forbidden by platform/kyverno/policies/restrict-host-namespaces.yaml — but that policy, like every
+# other policy in this repo, is scoped by
+#   `namespaceSelector: matchLabels: homelab.keithjackson.dev/app-namespace: "true"`
+# and its own header states the carve-out: "platform components that do [need these] live in platform
+# namespaces that do NOT carry the app-namespace label, so they are out of scope."
+# Verified by reading the policies on this branch, not assumed:
+#   * restrict-host-namespaces.yaml rule `disallow-host-namespaces`     (hostPID)
+#   * restrict-host-namespaces.yaml rule `disallow-privileged-containers` (privileged)
+#
+# We set the label EXPLICITLY to "false" rather than leaving it unset, following
+# platform/intel-gpu-plugin/namespace.yaml and platform/vpn-egress: an unset label and a deliberate
+# "false" are identical to the admission engine but not to a reviewer, and this component's
+# correctness depends entirely on the answer.
+#
+# NO PodSecurity labels. There is no cluster-wide PSA baseline in this repo — `pod-security.*`
+# appears only where a namespace opts IN (conformance/incluster, platform/kyverno/admission,
+# platform/pm-agent), all at `enforce: restricted`. Adding them here would reject this DaemonSet for
+# the same reasons Kyverno would. Leaving them off is the convention for a privileged platform
+# namespace (platform/vpn-ingress/namespace.yaml sets none either).
+#
+# NO `gateway-access` label. kured serves no HTTP to users and attaches no HTTPRoute; its only
+# listener is the :8080 metrics endpoint, reached in-cluster by Prometheus. See networkpolicy.yaml.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: kured
+  labels:
+    homelab.keithjackson.dev/app-namespace: "false"

platform/kured/networkpolicy.yaml

@@ -0,0 +1,125 @@
+---
+# Default-deny baseline (R-NET-01): deny all ingress + egress, allow only DNS. `kured` is a PLATFORM
+# namespace, so Kyverno's generator — scoped to `app-namespace=true` — does not produce this; we ship
+# it explicitly, as crafty/headlamp/homepage/runbooks all do.
+#
+# THESE POLICIES DO APPLY TO kured, UNLIKE platform/vpn-ingress'. That component's identical-looking
+# baseline is honest-but-inert because its DaemonSet runs `hostNetwork: true` and therefore has no
+# Cilium endpoint for any policy to select. kured's chart defaults to `hostNetwork: false` and we
+# keep that default (it needs the host PID namespace to reboot, not the host network stack), so it
+# has a normal pod identity and every rule here is enforced.
+#
+# Cilium enforces the UNION of these policies (allow if ANY policy allows), so the CiliumNetworkPolicy
+# below adds the kube-apiserver path on top of this deny-all floor.
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: default-deny
+  namespace: kured
+  labels:
+    homelab.keithjackson.dev/baseline: default-deny
+spec:
+  podSelector: {}
+  policyTypes:
+    - Ingress
+    - Egress
+  ingress: []
+  egress:
+    - to:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: kube-system
+      ports:
+        - protocol: UDP
+          port: 53
+        - protocol: TCP
+          port: 53
+---
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# PROMETHEUS MUST REACH :8080, OR EVERY RULE IN alerts.yaml IS INERT.
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# This is not a theoretical hazard in this repo, it is a measured one. From the Longhorn entry in
+# conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh: with the ServiceMonitor
+# applied and no NetworkPolicy allowance, all three targets were DISCOVERED and every scrape failed
+# with "context deadline exceeded" after 10.0s — "a drop, not a refusal" — and the TSDB stayed
+# empty. Target discovery succeeding is therefore NOT evidence the scrape works, and the alerts
+# would have been quiet rather than broken. KuredNotReporting is the standing detector for exactly
+# this regression, and it can only work if this policy is correct.
+#
+# THE WHOLE `monitoring` NAMESPACE, NOT THE PROMETHEUS POD LABELS — following the same choice
+# longhorn/networkpolicy-metrics.yaml made and for the same reason: the prometheus-operator owns
+# those pod labels and rewrites them on upgrade (measured 2026-10-17, the pod carries
+# `app.kubernetes.io/name: prometheus` plus five operator-managed labels including a
+# controller-revision-hash), so selecting on them makes the scrape silently fail on a chart bump.
+# The namespace holds only the monitoring stack.
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: kured-allow-prometheus-scrape
+  namespace: kured
+  labels:
+    app.kubernetes.io/name: kured
+spec:
+  podSelector:
+    matchLabels:
+      # Set by the chart's `kured.matchLabels` helper (_helpers.tpl:70-71) with release name `kured`.
+      app.kubernetes.io/name: kured
+  policyTypes:
+    - Ingress
+  ingress:
+    - from:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: monitoring
+      ports:
+        - protocol: TCP
+          # The container port the chart names `metrics` (daemonset.yaml:211-213), from
+          # `configuration.metricsPort`, which we leave at the chart default of 8080 and mirror in
+          # the HelmRelease's `service.port`.
+          port: 8080
+---
+# Egress allows. kured talks to exactly two things and nothing else; both are required and the
+# component is useless without either.
+#
+#   1. THE KUBE-APISERVER — the lock annotation on its own DaemonSet (`weave.works/kured-node-lock`),
+#      cordon/uncordon (`nodes` patch), the node annotations, listing pods, and creating
+#      `pods/eviction` objects during the drain. Reached via the `kube-apiserver` Cilium entity
+#      rather than an IP: on this cluster the API is fronted by a kube-vip VIP that moves between
+#      nodes on every reboot, so any CIDR-based rule here would break during the exact operation
+#      this component exists to perform. Same construct as platform/headlamp/networkpolicy.yaml.
+#
+#   2. PROMETHEUS, on 9090 — the alert gate. The HelmRelease sets
+#      `prometheusUrl: http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090`
+#      and `alertFilterMatchOnly: true`, so before taking the cluster lock kured asks Prometheus
+#      whether any of LonghornVolumeDegraded / LonghornVolumeFaulted / KubeNodeNotReady /
+#      NodeExporterDown is firing. IF THIS EGRESS IS BLOCKED, kured cannot evaluate the gate.
+#      It treats the query error as "do not proceed" and skips the reboot — which is the correct
+#      fail-safe direction, but it fails SILENTLY and indefinitely: reboots simply stop happening
+#      and nothing goes red except, eventually, KuredRebootPending. Do not remove this rule
+#      thinking it is only for metrics; metrics ingress is the policy above, this is the gate.
+#
+# NOTHING ELSE. No egress to the internet (kured pulls no content at runtime), no DNS beyond the
+# baseline above, no Longhorn or etcd access — kured reads Longhorn's health THROUGH Prometheus
+# alerts, never from Longhorn's own API, which is why that allow is absent here.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: kured-allow-egress
+  namespace: kured
+  labels:
+    app.kubernetes.io/name: kured
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: kured
+  egress:
+    - toEntities:
+        - kube-apiserver
+    - toEndpoints:
+        - matchLabels:
+            io.kubernetes.pod.namespace: monitoring
+            app.kubernetes.io/name: prometheus
+      toPorts:
+        - ports:
+            - port: "9090"
+              protocol: TCP

platform/monitoring/node-pressure-dashboard.yaml

@@ -199,7 +199,7 @@ data:
           "id": 15,
           "type": "timeseries",
           "title": "CAUSE 4 — NFS RPC retransmission ratio, per node",
-          "description": "node_nfs_rpc_retransmissions_total / node_nfs_rpcs_total, from /proc/net/rpc/nfs. Alert line at 1% (NodeNFSRetransmissionRateHigh). MEASURED BASELINE IS EXACTLY ZERO on all three nodes over 48h, including throughout the 2026-09-14 spike — so any visible line here means the export at 192.168.2.102, or the path to it, is dropping requests. A flat zero does NOT clear NFS: it only clears the TRANSPORT. A merely SLOW export needs per-mount RPC round-trip time, which arrives with the mountstats collector (see kube-prometheus-stack.yaml) and is the known gap recorded in runbooks/node-io.md.",
+          "description": "node_nfs_rpc_retransmissions_total / node_nfs_rpcs_total, from /proc/net/rpc/nfs. Alert line at 1% (NodeNFSRetransmissionRateHigh). MEASURED BASELINE IS EXACTLY ZERO on all three nodes over 48h, including throughout the 2026-09-14 spike — so any visible line here means the export at 192.168.2.30, or the path to it, is dropping requests. A flat zero does NOT clear NFS: it only clears the TRANSPORT. A merely SLOW export needs per-mount RPC round-trip time, which arrives with the mountstats collector (see kube-prometheus-stack.yaml) and is the known gap recorded in runbooks/node-io.md.",
           "gridPos": {"h": 8, "w": 12, "x": 12, "y": 39},
           "fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [
             {"color": "green", "value": null}, {"color": "red", "value": 0.01}

platform/oauth2-proxy/helmrelease.yaml

@@ -63,6 +63,75 @@ spec:
       # gets oauth2-proxy's HTML sign-in page (HTTP 403) instead of an immediate redirect (docs/15 UX).
       skip-provider-button: "true"
       scope: "openid email profile"
+    # ── Cold-start gate: do not start the proxy until Keycloak's discovery doc answers ──────────
+    # MEASURED on prod 2026-10-17, during the UPS move: oauth2-proxy performs OIDC discovery ONCE at
+    # startup and EXITS 1 if it fails. On a whole-cluster cold boot Keycloak is not serving yet, so
+    # the Gateway answers 503 and the proxy dies:
+    #
+    #   [provider.go:55] Performing OIDC Discovery...
+    #   [main.go:73] ERROR: Failed to initialise OAuth2 Proxy: ... failed to discover OIDC
+    #                configuration: unexpected status "503": no healthy upstream
+    #
+    # It then sits in exponential CrashLoopBackOff long after Keycloak is healthy -- one replica was
+    # still down minutes later and only a manual pod delete cleared it. There is no retry flag; the
+    # process simply does not survive a cold start. A readinessProbe cannot help (the container is
+    # already dead), and CrashLoopBackOff backoff is not tunable per-pod.
+    #
+    # `extraInitContainers` is the chart's OWN documented hook for this -- its commented example in
+    # values.yaml is literally `- name: wait-for-idp` -- and the chart already uses an alpine init
+    # container for its built-in waitForRedis gate, so this adds no new base image to the platform.
+    # A pod in Init does not accrue CrashLoopBackOff backoff and retries promptly, which is the whole
+    # point.
+    #
+    # BOUNDED, AND FAILS OPEN. After ~5 minutes it gives up and starts the proxy anyway: the worst
+    # case is then exactly today's behaviour, never something new. An unbounded wait would trade a
+    # CrashLoop for a pod wedged in Init forever, which is not an improvement.
+    #
+    # SHELL VARIABLES ARE WRITTEN BARE ($i, $url, $(( ))), WITH NO BRACED FORM ANYWHERE. This file is
+    # rendered through Flux envsubst (platform declares postBuild.substituteFrom cluster-vars), which
+    # CONSUMES braced expansions -- so a braced shell variable renders to the empty string and
+    # silently breaks the loop. BASE_DOMAIN below is braced ON PURPOSE: it is a real cluster-vars
+    # knob and must be substituted. Same hazard, and the same convention, as the gitea and nextcloud
+    # OIDC bootstrap hooks. (ci: flux-postbuild-substitution enforces it.)
+    extraInitContainers:
+      - name: wait-for-idp
+        image: docker.io/library/alpine:3.22
+        imagePullPolicy: IfNotPresent
+        command: ["/bin/sh", "-c"]
+        args:
+          - |
+            set -u
+            url=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
+            i=0
+            while [ "$i" -lt 60 ]; do
+              # busybox wget validates against alpine's ca-certificates-bundle, so a real TLS
+              # failure is a non-zero exit here rather than a false pass.
+              if wget -q -O /dev/null --timeout=5 "$url" 2>/dev/null; then
+                echo "[wait-for-idp] Keycloak discovery reachable; starting oauth2-proxy"
+                exit 0
+              fi
+              i=$(( i + 1 ))
+              sleep 5
+            done
+            echo "[wait-for-idp] discovery still unreachable after ~5m; starting anyway so this" >&2
+            echo "[wait-for-idp] gate can never be worse than the un-gated behaviour it replaces" >&2
+            exit 0
+        securityContext:
+          allowPrivilegeEscalation: false
+          runAsNonRoot: true
+          runAsUser: 65534
+          runAsGroup: 65534
+          readOnlyRootFilesystem: true
+          capabilities:
+            drop:
+              - ALL
+        resources:
+          requests:
+            cpu: 10m
+            memory: 16Mi
+          limits:
+            memory: 32Mi
+
     # Metrics for the ServiceMonitor (../oauth2-proxy servicemonitor in platform-config).
     metrics:
       enabled: true

platform/vpn-ingress/README.md

@@ -13,7 +13,7 @@ egress out the house WAN.
 | Transport | WireGuard, **udp/51820**, kernel implementation |
 | Where it runs | `hostNetwork` **DaemonSet** — one warm listener on every node |
 | Client subnet | `${WG_CLIENT_CIDR}` = `10.90.0.0/24`; `.1` is the server, `.2`–`.254` devices |
-| Authorisation | a device's **public key** is in [`peers.yaml`](./peers.yaml), or it cannot connect |
+| Authorisation | a device's **public key** is in [`peers.conf`](./peers.conf), or it cannot connect |
 | The one secret | the server's private key — **SOPS-encrypted in this tree**, `wg-server-key.sops.yaml` |
 | Exposure | one hand-made WAN port-forward on the SOHO router, off-repo |
 | Enrolling a device | [`infra/vpn-ingress/new-peer.sh`](../../infra/vpn-ingress/new-peer.sh) |
@@ -129,7 +129,7 @@ translations win here.
 
 | half | where | why |
 |---|---|---|
-| peer **public** keys + addresses | [`peers.yaml`](./peers.yaml), **plaintext in git** | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
+| peer **public** keys + addresses | [`peers.conf`](./peers.conf), **plaintext in git**, as a content-hashed `configMapGenerator` so an edit rolls the pods | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
 | the server's **private** key | `wg-server-key.sops.yaml`, **SOPS-encrypted in git** | the only real secret (R-SEC-01, spec docs/07). Ships with a `REPLACE-ME` placeholder: an agent must not mint a long-lived private key (CLAUDE.md §6), so the operator `sops set`s it in |
 
 The two are joined only at pod start, by the init container, into a **`medium: Memory` emptyDir** —
@@ -166,8 +166,9 @@ NetworkPolicy applies.
 ## Operating it
 
 * **Enrol a device:** `./infra/vpn-ingress/new-peer.sh <name> <octet>` → paste the printed `[Peer]`
-  stanza into `peers.yaml` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
-* **Revoke a device:** delete its stanza → PR → merge. No other step; the key stops working.
+  stanza into `peers.conf` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
+* **Revoke a device:** delete its stanza from `peers.conf` → PR → merge. No other step; the key
+  stops working as soon as the pods roll (the generator's content hash guarantees they do).
 * **Is it up?** `kubectl --context k3s-prod -n vpn-ingress get ds wireguard` — three Ready.
 * **Who is connected?** `kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show`
   (`latest handshake` under ~3 min means live). Never dump `wg showconf` — it prints the private key.
@@ -211,7 +212,7 @@ right way; one was a defect in my own manifest.
   covered generically by `KubeDeploymentReplicasMismatch`/`KubePodNotReady` from
   kube-prometheus-stack. **This is the top follow-up.**
 * **No preshared keys.** They would add a symmetric post-quantum hedge on top of the Noise
-  handshake — and would be a real secret, forcing `peers.yaml` into SOPS and costing the
+  handshake — and would be a real secret, forcing `peers.conf` into SOPS and costing the
   reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the
   `wg-server-key` Secret.
 * **No IPv6.** The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6

platform/vpn-ingress/daemonset.yaml

@@ -108,7 +108,7 @@ spec:
       dnsPolicy: ClusterFirstWithHostNet
       automountServiceAccountToken: false
       # ── Assemble the running config: PUBLIC peer register (git) + PRIVATE server key (Secret) ──
-      # The two halves are deliberately stored apart (peers.yaml explains why) and are only ever
+      # The two halves are deliberately stored apart (peers.conf explains why) and are only ever
       # joined here, at pod start, into a tmpfs that is destroyed with the pod. The server's private
       # key is therefore never written to any node's disk, never committed, and never logged.
       initContainers:

platform/vpn-ingress/kustomization.yaml

@@ -11,10 +11,16 @@
 # NOT in platform/kustomization.yaml — see clusters/base/vpn-ingress.yaml.
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
+# REQUIRED for the configMapGenerator below to be wired up at all, not merely tidy. A generated
+# ConfigMap inherits NO namespace, every other object here declares `vpn-ingress`, and kustomize's
+# nameReference transformer will not rewrite a reference across a namespace boundary — so without
+# this the DaemonSet keeps mounting the unhashed name `wg-peers`, which does not exist, and the
+# pods wedge on a missing volume. Verified by building it both ways (kustomize v5.8.1).
+# Same placement as runbooks/kustomization.yaml, which uses `namespace: runbooks` for its own
+# content generator.
+namespace: vpn-ingress
 resources:
   - namespace.yaml
-  # The peer register: which devices may dial in. PLAINTEXT and reviewable on purpose (see the file).
-  - peers.yaml
   # The listener itself: hostNetwork DaemonSet, udp/51820 on every node.
   - daemonset.yaml
   # Namespace default-deny baseline. NOTE: does NOT constrain the hostNetwork DaemonSet — the file
@@ -38,3 +44,26 @@ resources:
 # hands Flux the raw ciphertext, the Secret APPLIES CLEANLY full of ENC[...], and nothing in any
 # Flux condition looks wrong. The init container's `wg pubkey` guard exists partly to turn exactly
 # that into a named error rather than a silently non-working tunnel.
+
+# ── THE PEER REGISTER, as a CONTENT-HASHED ConfigMap ─────────────────────────────────────────
+# peers.conf (plaintext, reviewable — see the file) becomes the `wg-peers` ConfigMap. It is a
+# GENERATOR, not a plain resource, and that distinction is load-bearing rather than stylistic:
+# kustomize appends a hash of the content to the generated name and rewrites the DaemonSet's
+# volume reference to match, so adding or revoking a peer changes the pod template and ROLLS THE
+# PODS automatically.
+#
+# A fixed-name ConfigMap (which is what this component shipped first) does not do that. Flux
+# updates it in place, the pod template is byte-identical, no rollout is triggered — and because
+# the init container assembles wg0.conf ONCE at pod start into a tmpfs, the running tunnel keeps
+# the old peer list indefinitely. Measured on prod 2026-10-06: a merged peer reached the
+# ConfigMap while `wg show` listed no peers and the DaemonSet still read generation=1, having
+# never rolled. The runbook's "PR -> merge -> Flux rolls the DaemonSet" was false until this.
+#
+# DO NOT set `disableNameSuffixHash` here. The hash IS the mechanism.
+configMapGenerator:
+  - name: wg-peers
+    files:
+      - peers.conf
+    options:
+      labels:
+        app.kubernetes.io/name: vpn-ingress

platform/vpn-ingress/networkpolicy.yaml

@@ -19,7 +19,7 @@
 # protects this port, and it would be enforcing nothing. So there isn't one.
 #
 # WHAT ACTUALLY PROTECTS THE EXPOSED PORT, then, since the CNI does not:
-#   1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.yaml gets
+#   1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.conf gets
 #      NO REPLY — see README "Why this is safe to expose". This is the real control.
 #   2. The SOHO router's forward: only udp/51820 is forwarded inbound, to one node.
 #   3. The blast radius once authenticated is deliberately the whole LAN — that IS the requirement

platform/vpn-ingress/peers.conf

@@ -1,4 +1,16 @@
----
+# THIS FILE IS THE MOUNTED CONTENT. It is turned into the `wg-peers` ConfigMap by
+# `configMapGenerator` in kustomization.yaml, whose generated name carries a CONTENT HASH — so
+# editing this file changes the ConfigMap name, which changes the DaemonSet's pod template, which
+# ROLLS THE PODS. That is the whole reason it is a generator and not an inline ConfigMap manifest:
+# a fixed-name ConfigMap updates in place, the pod template never changes, no rollout happens, and
+# the init container's wg0.conf (built once at pod start, in tmpfs) keeps the OLD peer list
+# forever. Measured 2026-10-06: a merged peer sat in the ConfigMap while `wg show` listed none and
+# the DaemonSet reported generation=1 — it had never rolled. Same mechanism and same fix as
+# runbooks/kustomization.yaml.
+#
+# Every line here is WireGuard config. `#` is a legal comment in it, so the documentation below
+# travels inside the file that is actually mounted, rather than in a wrapper that gets stripped.
+#
 # ============================================================================================
 # THE PEER REGISTER — every device allowed to dial in, in PLAINTEXT, in git, on purpose.
 # ============================================================================================
@@ -41,43 +53,40 @@
 # the Noise handshake and would be a genuine secret — which would force this register into SOPS
 # and cost the reviewability above. Revisit only if the threat model changes; if it does, the PSK
 # belongs in the `wg-server-key` Secret, not here. See README "Hardening not taken".
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: wg-peers
-  namespace: vpn-ingress
-  labels:
-    app.kubernetes.io/name: vpn-ingress
-data:
-  # Appended VERBATIM to the generated [Interface] block, in this order, at pod start.
-  #
-  # *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. ***
-  # An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of
-  # authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the
-  # DaemonSet will run and listen, and every handshake will be refused because no key is trusted.
-  # The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible
-  # (`wg show` reports an interface with no peers).
-  #
-  # COPY THIS TEMPLATE, uncomment, one block per device:
-  #
-  #   # <device-name> — <owner>, enrolled <YYYY-MM-DD>
-  #   [Peer]
-  #   PublicKey = <the public key new-peer.sh printed>
-  #   AllowedIPs = 10.90.0.<octet>/32
-  #
-  # The placeholders above are deliberately NOT valid values. An earlier version of this template
-  # used a real-looking `10.90.0.2/32`, and new-peer.sh's duplicate-address check — which grepped
-  # the whole file — matched it, so `.2` could never be allocated and the script insisted it was
-  # already taken. The check now ignores comments; keeping the example unmatchable too means a
-  # future grep cannot be fooled either.
-  #
-  # Keep the `# <name> — <owner>, enrolled <date>` comment line. It is the only place a key is
-  # tied to a human-recognisable device; `wg show` can only ever print the key itself, so without
-  # it an incident responder cannot tell which stanza to delete.
-  peers.conf: |
-    # No peers enrolled yet. See the template in this file's comments and
-    # ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device.
-    # keith-phone — <owner>, enrolled 2026-10-06
-    [Peer]
-    PublicKey = pBRSHrRyG2hnsuZ6FvHfEzF8hOVTF5stHUb/7fW2Km8=
-    AllowedIPs = 10.90.0.2/32
+#
+# name: wg-peers
+# namespace: vpn-ingress
+# labels:
+# app.kubernetes.io/name: vpn-ingress
+# Appended VERBATIM to the generated [Interface] block, in this order, at pod start.
+#
+# *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. ***
+# An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of
+# authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the
+# DaemonSet will run and listen, and every handshake will be refused because no key is trusted.
+# The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible
+# (`wg show` reports an interface with no peers).
+#
+# COPY THIS TEMPLATE, uncomment, one block per device:
+#
+#   # <device-name> — <owner>, enrolled <YYYY-MM-DD>
+#   [Peer]
+#   PublicKey = <the public key new-peer.sh printed>
+#   AllowedIPs = 10.90.0.<octet>/32
+#
+# The placeholders above are deliberately NOT valid values. An earlier version of this template
+# used a real-looking `10.90.0.2/32`, and new-peer.sh's duplicate-address check — which grepped
+# the whole file — matched it, so `.2` could never be allocated and the script insisted it was
+# already taken. The check now ignores comments; keeping the example unmatchable too means a
+# future grep cannot be fooled either.
+#
+# Keep the `# <name> — <owner>, enrolled <date>` comment line. It is the only place a key is
+# tied to a human-recognisable device; `wg show` can only ever print the key itself, so without
+# it an incident responder cannot tell which stanza to delete.
+#
+# No peers enrolled yet. See the template in this file's comments and
+# ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device.
+# keith-phone — <owner>, enrolled 2026-10-06
+[Peer]
+PublicKey = pBRSHrRyG2hnsuZ6FvHfEzF8hOVTF5stHUb/7fW2Km8=
+AllowedIPs = 10.90.0.2/32

platform/vpn-ingress/wg-server-key.sops.yaml.example

@@ -5,7 +5,7 @@
 # WHAT THIS IS: the WireGuard SERVER's long-term private key for the remote-access VPN
 # (platform/vpn-ingress). It is the ONE secret in the whole component. The init container in
 # daemonset.yaml reads it at `/run/wg-server-key/privatekey` and joins it to the PLAINTEXT peer
-# register (peers.yaml) in a tmpfs, so the assembled wg0.conf never touches a node's disk.
+# register (peers.conf) in a tmpfs, so the assembled wg0.conf never touches a node's disk.
 #
 # WHY IT IS IN GIT, SOPS-ENCRYPTED, AND NOT HAND-CREATED. R-SEC-01 / spec docs/07: secrets live in
 # source, encrypted, and reach the cluster only through Flux. An out-of-band `kubectl create secret`

renovate.json

@@ -1,6 +1,6 @@
 {
   "$schema": "https://docs.renovatebot.com/renovate-schema.json",
-  "_comment": "Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). Renovate owns HELM CHART + DEPENDENCY bumps (opens PRs; allowed classes auto-merge on green CI). Flux image-automation owns per-service container IMAGE TAGS (the $imagepolicy setters). The two do NOT overlap: this config DISABLES image-tag updates so Renovate never fights Flux over a running version. Self-hosted against the bootstrap Gitea; platform: gitea set at runtime by the operator's renovate runner.",
+  "_comment": "Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). AS OF 2026-10-17 RENOVATE OWNS BOTH HALVES: Helm chart/dependency bumps AND per-service container image tags. It opens PRs for everything; allowed low-risk classes auto-merge on green CI. The ADR-0009 split that gave image tags to Flux image-automation is formally closed — see .gitea/workflows/renovate.yaml for the three measured reasons (it pushes commits rather than opening PRs, so it bypasses the CLAUDE.md §1 gate; `main` is protected with user_can_push:false and the flux-system GitRepository has no secretRef, so it cannot write anyway; and enabling it would need an in-cluster git WRITE credential for the repo that defines the cluster). image-automation has been commented out at platform/kustomization.yaml:52 since it was authored and the cluster holds zero ImagePolicy/ImageRepository/ImageUpdateAutomation objects, so nothing is being torn down. CONSEQUENCE: the `docker` and `kubernetes` managers, previously disabled ONLY to avoid fighting that controller, are now ENABLED — without them every plain `image: repo/name:tag` line in this repo was maintained by nothing at all, which is how the oauth2-proxy auth sidecars reached 7 minors behind the oauth2-proxy running in the same cluster. RUNNER: .gitea/workflows/renovate.yaml (scheduled daily); platform/endpoint are set there, not here.",
   "extends": [
     "config:recommended",
     ":dependencyDashboard",
@@ -15,13 +15,24 @@
   "labels": ["dependencies", "automated-update"],
   "ignoreUnstable": true,
   "respectLatest": true,
-  "enabledManagers": ["flux", "kustomize", "helmv3", "regex", "github-actions"],
+  "enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "regex", "github-actions"],
   "flux": {
     "managerFilePatterns": ["/(^|/)platform/.+\\.ya?ml$/", "/(^|/)apps/.+\\.ya?ml$/"]
   },
   "kustomize": {
     "managerFilePatterns": ["/(^|/)kustomization\\.ya?ml$/"]
   },
+  "_kubernetes_comment": "ADDED 2026-10-17. The `kubernetes` manager reads plain `image:` fields out of Kubernetes manifests, and it is NOT auto-detected — it only looks at files matched here, which is why this block is as broad as it is. Scope covers every place this repo writes a container image: platform components, the rendered app waves, the ServiceDefinition catalog that generates them, the runbooks web server and the pm-agent job template. `catalog/` matters as much as `apps/`: apps/wave-*/*.yaml are GENERATED by catalog/render.py, so a bump applied only to the rendered output is reverted by the next render and would also fail ci/render-golden.sh. Renovate will raise BOTH, and both must land together.",
+  "kubernetes": {
+    "managerFilePatterns": [
+      "/(^|/)platform/.+\\.ya?ml$/",
+      "/(^|/)platform-config/.+\\.ya?ml$/",
+      "/(^|/)apps/.+\\.ya?ml$/",
+      "/(^|/)catalog/.+\\.ya?ml$/",
+      "/(^|/)runbooks/serving/.+\\.ya?ml$/",
+      "/(^|/)infra/agent/.+\\.ya?ml$/"
+    ]
+  },
   "customManagers": [
     {
       "customType": "regex",
@@ -60,14 +71,29 @@
       "matchFileNames": ["ci/versions.env", "infra/devcontainer/versions.env"],
       "automerge": false,
       "addLabels": ["ci-tooling"]
+    },
+    {
+      "description": "ADDED 2026-10-17. NEVER AUTO-MERGE THE BLAST-RADIUS COMPONENTS, not even a patch. The default rule above auto-merges minor/patch/digest on a green gate, which is right for an app image: a bad Sonarr bump inconveniences one pod. It is wrong for these five, because the gate is a STATIC check — it renders manifests and lints them, it does not boot a node, drain a volume or form an etcd quorum (vars.CLUSTER_AVAILABLE is unset, so L2/L3 conformance does not run in CI at all). So for anything whose failure mode is the CLUSTER rather than a service, a green gate is not evidence and a human must read the changelog. kured is here because it reboots control-plane nodes; longhorn because a storage-engine bump migrates live volume data; cilium because it is the CNI and a bad bump is a total partition (and its version is DUAL-PINNED in infra/ansible/roles/cilium/defaults/main.yml, which Renovate does not see, so the two would silently desync); cloudnative-pg because it drives Postgres switchovers; kube-prometheus-stack because ci/versions.env asserts an exact promtool/Prometheus lockstep and a chart bump that moves Prometheus fails the gate until that file moves with it.",
+      "matchPackageNames": [
+        "kured",
+        "longhorn",
+        "cilium",
+        "cloudnative-pg",
+        "kube-prometheus-stack"
+      ],
+      "automerge": false,
+      "addLabels": ["needs-review", "blast-radius"]
     }
   ],
+  "_unmanageable_comment": "WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A FLOATING tag has no version to compare, so Renovate skips it silently — it does not pin it and does not warn. As of 2026-10-17 that is: plexinc/pms-docker:latest, lscr.io/linuxserver/jackett:latest, pihole/pihole (NO TAG AT ALL), quay.io/keycloak/keycloak (no tag), nextcloud:31-apache and docker:27-dind. Those must be pinned BY HAND to a concrete version before any automation can maintain them, and until then they drift on every pull with nothing recording what is running. Separately, the HelmRelease minor-locking ranges (9 of 12, e.g. `>=88.0.0 <89.0.0` on kube-prometheus-stack) cap Renovate at patches inside a line it has been told not to leave; raising those ceilings is deliberate per-component work and is NOT part of enabling the bot. Both lists are in decisions/2026-10-17-automated-upgrades.md.",
   "ignorePaths": [
     "**/*.example",
     "**/*.sops.yaml",
     "conformance/**",
-    "platform/image-automation/per-service/**"
+    "platform/image-automation/per-service/**",
+    "platform/kyverno/tests/**"
   ],
-  "docker": { "enabled": false },
-  "kubernetes": { "enabled": false }
+  "_ignorePaths_comment": "platform/kyverno/tests/** ADDED 2026-10-17: those Kyverno policy fixtures pin `alpine:latest` and a `conformance-runner:latest`, which are test inputs whose whole point is to be a fixed shape, not software to keep current. `conformance/**` was already ignored for the same reason; this closes the matching hole that enabling the `kubernetes` manager would otherwise have opened.",
+  "_docker_comment": "The `docker` manager (Dockerfiles, docker-compose) and the `kubernetes` manager above were BOTH previously `enabled: false`, solely so Renovate would not fight Flux image-automation over a running tag. That controller was never enabled and never will be here (see _comment), so the only thing those two flags were achieving was leaving every plain `image:` tag in the repo unmanaged. `docker` is left out of enabledManagers because this repo contains no Dockerfile that ships to the cluster; `kubernetes` is now ON and is the one that matters.",
+  "docker": { "enabled": true }
 }

runbooks/kustomization.yaml

@@ -85,6 +85,9 @@ configMapGenerator:
       - node-io=node-io.md
       - node-memory=node-memory.md
       - node-memory-swap=node-memory-swap.md
+      # Covers all three platform-health.node-reboots rules (KuredRebootPending,
+      # KuredNodeCoverageGap, KuredNotReporting) from platform/kured/alerts.yaml.
+      - node-reboot-automation=node-reboot-automation.md
       # Linked from OAuth2ProxyGateUnobserved, and the diagnostic procedure for any gated app
       # that "is up but does not work" — the gate answers an unauthenticated XHR with a 302,
       # not a 401, so the failure arrives as an HTTP 200 carrying the wrong body.

runbooks/node-reboot-automation.md

@@ -0,0 +1,254 @@
+# Node reboot automation (kured)
+
+**Alerts:** `KuredRebootPending`, `KuredNodeCoverageGap`, `KuredNotReporting`
+**File:** `platform/kured/alerts.yaml`
+**Expressions:** `max by (node) (kured_reboot_required) == 1` / `count(kured_reboot_required) < count(kube_node_info)` / `absent(kured_reboot_required)`
+
+## What this means here
+
+These three alerts cover the OS reboot half of the automated upgrade process. The design is two
+pieces that only work together:
+
+- **`roles/unattended_upgrades`** (Ansible, on the host) installs Debian **security** updates
+  nightly. It never upgrades the kernel, never restarts k3s, and never reboots. When something it
+  installed needs a reboot, it leaves `/var/run/reboot-required` behind.
+- **kured** (`platform/kured`, a DaemonSet) watches that file. When it exists, kured takes a
+  **cluster-wide lock**, cordons the node, drains it, reboots it, and uncordons it — then holds the
+  lock for a further 30 minutes before any other node may go.
+
+**kured does not decide whether to reboot.** On these nodes the sentinel is written almost
+exclusively by a deliberate operator kernel upgrade
+(`playbooks/node-baseline.yml --tags kernel`), because the kernel is blacklisted from unattended
+upgrades. So a pending reboot normally means *a human already asked for this*.
+
+Which alert you have tells you which half is wrong:
+
+| alert | meaning |
+|---|---|
+| `KuredRebootPending` | The mechanism is alive and a reboot was offered, but it did not happen for >36h. **Something is blocking it.** |
+| `KuredNodeCoverageGap` | kured is not running on every node. The uncovered node will never be rebooted, silently. |
+| `KuredNotReporting` | kured is publishing nothing at all. Reboots are not happening, **and the other two alerts are inert.** Treat this as the most serious of the three. |
+
+**All three nodes are control-plane + etcd, so quorum is 2 of 3.** One node down is degraded but
+up; two is a lost cluster. Everything below is written to protect that. `concurrency: 1` plus
+`lockTtl: 0` means a node that hangs on reboot keeps the lock **forever** and no second node is ever
+rebooted behind it. That is deliberate: the cycle stops and pages a human rather than stepping over
+a corpse.
+
+**A reboot is not seamless.** Most workloads here are single-replica, so each reboot cycle takes a
+brief outage of whatever runs on that node — including `coredns` (1 replica), which means a
+cluster-wide DNS gap while it reschedules. See "Expected outage per cycle" below.
+
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
+picks a current-context per invocation, and that default has already changed once. Do not run two
+`dev.sh` invocations at once; they share one container and interfere.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `uncordon`, `rollout`) and `flux reconcile` are
+operator actions, not agent actions. Where this runbook gives a mutating command it is for an
+operator, and it says so.
+
+## First commands
+
+Start here, in order. The first question is always *which half is broken*.
+
+    # 1. Is kured running at all, and on how many nodes?
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds,pod -o wide'
+
+    # 2. Which node is asking for a reboot, and who holds the lock?
+    #    The lock is an annotation on kured's OWN DaemonSet. An empty/absent value means no holder.
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds kured \
+      -o jsonpath="{.metadata.annotations}"; echo'
+
+    # 3. Is a node cordoned, and is a reboot in progress?
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes \
+      -o custom-columns=NAME:.metadata.name,SCHEDULABLE:.spec.unschedulable,TAINTS:.spec.taints'
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes \
+      -o jsonpath="{range .items[*]}{.metadata.name}{\" \"}{.metadata.annotations.weave\.works/kured-reboot-in-progress}{\"\n\"}{end}"'
+
+    # 4. What is kured itself saying? It logs its decision every period (1h).
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured logs ds/kured --tail=80'
+
+    # 5. Is the gate blocking it? These are the ONLY alerts kured blocks on.
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n longhorn-system \
+      get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness' \
+      | grep -v healthy
+
+    # 6. Does the node actually want a reboot? (read-only; SSH as k3s, no sudo needed)
+    ssh -o BatchMode=yes 192.168.2.10 'ls -la /var/run/reboot-required; cat /var/run/reboot-required.pkgs 2>/dev/null'
+
+## Likely causes, most likely first
+
+### 1. The Longhorn gate is holding it (the common `KuredRebootPending` cause)
+
+kured queries Prometheus before taking the lock and refuses to proceed while
+`LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `KubeNodeNotReady` or `NodeExporterDown` is
+firing. This is working as designed and it is the **most likely** cause.
+
+Why it lingers: there are 3 nodes, every volume has 3 replicas, and
+`replica-soft-anti-affinity: false` means replicas may not share a node. So cordoning one node
+leaves nowhere to put a replacement and **every volume goes degraded 2/3** for the duration.
+Recovery is slow on purpose: `replica-replenishment-wait-interval` is 600s before a rebuild even
+starts, and `concurrent-replica-rebuild-per-node-limit` is 1, so ~25 volumes rebuild one at a time.
+
+Check progress, then wait:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n longhorn-system \
+      get volumes.longhorn.io -o json' | python3 -c 'import json,sys,collections; \
+      v=json.load(sys.stdin)["items"]; print(collections.Counter(x["status"]["robustness"] for x in v))'
+
+If volumes are rebuilding, **do nothing**. If a volume is stuck `degraded` or `faulted` with no
+rebuild progressing, that is a storage problem to fix first — see `/runbooks/longhorn-degraded`.
+Rebooting another node while a volume is at 1/3 is how you lose data.
+
+### 2. The drain timed out against a PodDisruptionBudget
+
+Six PDBs currently allow **zero** voluntary disruptions, and two kinds matter:
+
+- **`longhorn-system/instance-manager-*`** (one per node). `minAvailable: 1` against a selector
+  pinned to a single node's single pod, so `disruptionsAllowed` is structurally **always 0**.
+  Longhorn's `longhorn-manager` *deletes* the PDB when its `node-drain-policy`
+  (`block-if-contains-last-replica`) judges the drain safe. The drain is **supposed** to wait here.
+- **CNPG `<cluster>-primary`** (`keycloak-db`, `nextcloud-db`, `semaphore-db`). Also structurally 0
+  until the CNPG operator switches the primary to another instance. If all three primaries are on
+  the node being drained, three switchovers must happen.
+
+kured's `drainTimeout` is 15m: a genuinely stuck drain **aborts**, releases the lock, and is retried
+next period rather than wedging forever. Repeated timeouts show in kured's logs.
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pdb -A -o wide' | awk '$NF==0 || NR==1'
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
+      -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PRIMARY:.status.currentPrimary'
+
+If CNPG will not switch over, an **operator** can move the primary deliberately before the next
+window rather than fighting the drain.
+
+### 3. A node hung on reboot and still holds the lock
+
+`k3s-prod-2` has done this before (2026-09-03: powered, off the LAN ~38 minutes, physical power
+button required). `lockTtl: 0` means the lock does **not** expire, so the cycle stops here —
+correctly, since the cluster is at 2/3 and must not go to 1/3.
+
+Symptoms: one node `NotReady` or absent, `KuredNodeCoverageGap` firing alongside
+`KuredRebootPending`, and the lock annotation naming the missing node.
+
+**Operator recovery:** attend to the node physically. Once it is back and `Ready`, kured clears its
+own lock on the next period. If the node is gone for good and the lock must be cleared by hand,
+that is an operator action:
+
+    # OPERATOR ONLY. Only when you have confirmed the named node is NOT mid-reboot.
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured annotate ds kured \
+      weave.works/kured-node-lock- --overwrite'
+
+If the node was pruned mid-reboot and is left **cordoned** with no daemon to uncordon it:
+
+    # OPERATOR ONLY.
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod uncordon <node>'
+
+### 4. Nothing can write the sentinel (so nothing ever reboots)
+
+If `KuredRebootPending` never fires and reboots never happen even after a kernel upgrade, suspect
+the sentinel writer. On Debian, `/var/run/reboot-required` is created by a hook shipped in
+**`update-notifier-common`** — and it was absent on all three nodes before this automation landed.
+Without it kured is permanently, invisibly idle.
+
+    ssh -o BatchMode=yes 192.168.2.10 'dpkg -l update-notifier-common needrestart unattended-upgrades | tail -4'
+    ssh -o BatchMode=yes 192.168.2.10 'ls /etc/kernel/postinst.d/ /etc/apt/apt.conf.d/ | sort'
+
+Fix by re-running the role, which asserts this explicitly:
+
+    cd infra/ansible && ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify
+
+### 5. kured is not being scraped (`KuredNotReporting`)
+
+The chart defaults `service.create` and `metrics.create` to **false**, and the `kured` namespace
+ships a default-deny NetworkPolicy. If either enablement regressed, or the Prometheus ingress allow
+was dropped, targets are *discovered* and every scrape times out — the TSDB stays empty and all
+three rules go quiet. Longhorn hit exactly this failure on this cluster.
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get svc,servicemonitor,networkpolicy,ciliumnetworkpolicy'
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n monitoring exec \
+      sts/prometheus-kube-prometheus-stack-prometheus -c prometheus -- \
+      wget -qO- "http://localhost:9090/api/v1/targets?state=active" ' | grep -o '"job":"kured[^,]*' | head
+
+### 6. A node is uncovered by the DaemonSet (`KuredNodeCoverageGap`)
+
+kured declares **no tolerations**, because no prod node is tainted. A newly provisioned or
+re-provisioned node carrying a taint gets no kured pod and will never be rebooted.
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes \
+      -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints'
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds kured \
+      -o custom-columns=DESIRED:.status.desiredNumberScheduled,READY:.status.numberReady'
+
+The fix is a PR adding the needed toleration to `platform/kured/helmrelease.yaml` — not a `kubectl
+patch`, which Flux reverts.
+
+## Expected outage per reboot cycle
+
+Stated plainly, because this process is **not** seamless. Every workload at 1 replica on the
+draining node is down until it reschedules and becomes Ready elsewhere. Measured 2026-10-17: **45
+Deployments and 3 StatefulSets run a single replica.** The ones worth knowing about:
+
+- **`kube-system/coredns` — 1 replica.** A cluster-wide in-cluster DNS gap while it moves. This is
+  the single most impactful item and it is **not** fixed by this PR; raising coredns to 2–3 replicas
+  is recommended follow-up work.
+- **`monitoring/prometheus-…-0`, `monitoring/loki-0`, `monitoring/alertmanager-…-0`** — all
+  single-replica StatefulSets on Longhorn RWO volumes, so they additionally wait for volume
+  detach/reattach. **You lose monitoring and alerting for part of the cycle**, including these very
+  alerts.
+- **All three CNPG Postgres primaries** blip during switchover, which means Keycloak, Nextcloud and
+  Semaphore briefly lose their database.
+- **`gitea`, `nextcloud`, `plex`, `jackett`, `qbittorrent`, `radarr`, `sonarr`, `seerr`, `crafty`,
+  `homepage`, `headlamp`, `semaphore`** — each down for its reschedule.
+- **`unifi` (a KubeVirt VM)** has `evictionStrategy: LiveMigrate` but
+  `migrationMethod: BlockMigration`, so evicting it copies the whole root disk over the network.
+  Expect the drain of its node to be slow.
+- **`kube-vip`** fails the control-plane VIP over on every node reboot.
+
+Multi-replica and therefore fine: `cilium-operator`, `oauth2-proxy`, `kyverno-admission-controller`,
+the Longhorn CSI sidecars, `keycloak`, `pihole`, `runbooks`.
+
+## When to escalate or stop
+
+**STOP and do not intervene** if:
+
+- Longhorn volumes are `degraded` **and rebuilding**. This is the gate doing its job. Wait.
+- A single node is mid-reboot and under 15 minutes have passed.
+
+**STOP and get a human to the hardware** if:
+
+- Any node is `NotReady` while another reboot is pending. **Never clear the kured lock in this
+  state** — that is the one action that can take etcd from 2/3 to 1/3 and lose the cluster.
+- Two nodes are down. The cluster has no quorum; this runbook cannot help. Recover hardware first.
+
+**Escalate to a deliberate maintenance decision** if:
+
+- `KuredRebootPending` has been firing for over a week. The reboot is not going to happen on its
+  own; find the blocker above, or take the reboot manually during a watched window.
+- A volume is `faulted`. Fix storage before any further reboot.
+
+**Turn the automation off** if it is misbehaving and you need the cluster stable. The honest way is
+a PR, but for an emergency an operator may suspend the tier — this stops future reboots and leaves
+running nodes untouched:
+
+    # OPERATOR ONLY. Stops kured being reconciled; does NOT uncordon a node mid-cycle.
+    infra/devcontainer/dev.sh run bash -c 'flux --context k3s-prod suspend kustomization kured'
+
+Check afterwards whether a node was left cordoned (First commands, step 3) and uncordon it if so.
+Resume with `flux resume kustomization kured`. Nothing in the graph depends on `kured`, so
+suspending it stalls nothing else.
+
+---
+
+Source of truth: `runbooks/node-reboot-automation.md`. Index: `/runbooks/`
+
+Records: `decisions/2026-10-17-automated-upgrades.md` (design + what was not verified),
+`platform/kured/helmrelease.yaml` (every tuned value and why),
+`infra/ansible/roles/unattended_upgrades/README.md` (the package half, and the kernel exclusion).

runbooks/vpn-remote-access.md

@@ -21,7 +21,7 @@ behaves like a machine on the LAN: full reach to `192.168.2.0/24`, the Gateway's
 DNS, and internet egress out the house WAN.
 
 Authorisation is one thing only: the device's **public key is in
-`platform/vpn-ingress/peers.yaml`**. An unlisted key gets no reply at all.
+`platform/vpn-ingress/peers.conf`**. An unlisted key gets no reply at all.
 
 > **An enrolled device is as privileged as a machine sitting in the house** — including the three
 > k3s nodes (`192.168.2.10/.11/.12`: apiserver 6443, kubelet 10250, SSH 22). That is what "reach
@@ -135,6 +135,7 @@ specific failure in its log — so you never get a server that silently cannot c
 | `still SOPS CIPHERTEXT (starts with ENC[)` | the Secret reached the cluster undecrypted — this environment's overlay is missing its `vpn-ingress` decryption patch. **This is the dangerous one:** the Secret applies cleanly and every Flux condition stays green |
 | `wg pubkey rejected it` with none of the above | a mangled value, or a *public* key pasted by mistake |
 | `net.ipv4.ip_forward is 0 on this node` | the node_baseline role has not run here (step 1). The pod **cannot** set this itself — under `hostNetwork` the runtime mounts `/proc/sys` read-only, because it is the host's global setting |
+| `0 peer(s)` after you merged a peer | the DaemonSet did not roll. Check its `generation` (above). Pre-2026-10-06 this was guaranteed, because the register was a fixed-name ConfigMap; if it recurs, something has reverted `peers.conf` to a plain resource instead of a `configMapGenerator` |
 
 ### What a healthy start looks like
 
@@ -218,10 +219,23 @@ It generates the device's keypair, writes a `0600` client profile, prints a QR c
 
 Then:
 
-1. Paste the stanza into `platform/vpn-ingress/peers.yaml` under `data.peers.conf`, keeping the
+1. Append the stanza to `platform/vpn-ingress/peers.conf`, keeping the
    `# <name> — <owner>, enrolled <date>` comment. `wg show` can only ever print the key, so that
    comment is the only record of which stanza belongs to which device.
-2. PR → merge. Flux rolls the DaemonSet (~1 min).
+2. PR → merge. Flux rolls the DaemonSet (~1 min), because `peers.conf` is a
+   **content-hashed** `configMapGenerator`: editing it changes the ConfigMap's generated name,
+   which changes the pod template, which forces the roll. (It did **not** work this way at first —
+   a fixed-name ConfigMap updates in place, the pod template never changes, and the init container
+   keeps the old peer list forever. Measured on prod 2026-10-06.) Confirm it actually rolled:
+
+   ```bash
+   ./infra/devcontainer/dev.sh run bash -c \
+     'kubectl --context k3s-prod -n vpn-ingress get ds wireguard \
+        -o jsonpath="{.metadata.generation}{\"/\"}{.status.observedGeneration}{\"\n\"}"'
+   ```
+
+   The generation must have **increased** since before the merge. If it is still `1`, the roll did
+   not happen and `wg show` will list no peer no matter how long you wait.
 3. Get the profile onto the device (QR scan for phones; "Import tunnel from file" on
    Windows/macOS; `/etc/wireguard/<name>.conf` + `wg-quick@<name>` on Linux).
 4. **Delete the profile from your machine**: `shred -u <path>`. It is the only copy of that
@@ -261,7 +275,7 @@ if the router is not delivering the packets, and a `handshake` with no completed
 
 ## Revoking a device
 
-Delete its `[Peer]` stanza from `peers.yaml`, PR, merge. Nothing else — the key stops working on the
+Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the
 next reconcile. **Never re-use that address octet**: an old profile would silently start working
 again as a different device.