fix(renovate): the cron fired outside Renovate's own window, so it would never open a PR

.gitea/workflows/renovate.yaml shipped with `cron: "15 3 * * *"`. Gitea Actions
cron is UTC, and renovate.json carries schedule ["after 3am and before 6am"]
with timezone America/New_York -- and THAT inner schedule is authoritative:
Renovate re-checks it and declines to open PRs outside it.

  03:15 UTC  ->  23:15 EDT, the PREVIOUS EVENING in New York
  window     ->  03:00-06:00 local   =>  NOT INSIDE (3h45m early)

The original comment claimed the cron sat "slightly ahead" of the window. It
sat 3h45m behind it, on the previous day. Every nightly run would have started,
logged "not within schedule", opened ZERO PRs and exited SUCCESSFULLY -- a green
tick forever, indistinguishable from "nothing needed updating", hiding the ~30
pending updates PR #73 existed to surface.

Changed to `15 8 * * *`, chosen to survive DST rather than to look tidy:
  EDT (UTC-4): local 03:00-06:00 == 07:00-10:00 UTC
  EST (UTC-5): local 03:00-06:00 == 08:00-11:00 UTC
08:15 UTC is inside BOTH (04:15 EDT / 03:15 EST), verified by computation at
both extremes, so neither switchover needs an edit. A cron cannot follow DST.

Also confirmed while here: RENOVATE_TOKEN now exists as a repo Actions secret,
so the workflow's fail-hard preflight will pass.

STILL UNRESOLVED and the likelier remaining bug: renovate.json's
`enabledManagers` lists bare `regex`, where modern Renovate names the custom
manager `custom.regex`. If the bare name is ignored, every inline `# renovate:`
marker in the tree does nothing. Deliberately NOT changed -- guessing at a key
that could fail validation and take the whole run down is worse than a question
one dry run answers. See the handoff.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

.gitea/workflows/renovate.yaml

@@ -0,0 +1,187 @@
+# Renovate — the dependency/image update automation. SCHEDULED, self-hosted against this Gitea.
+#
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# WHY THIS FILE EXISTS: renovate.json HAS BEEN IN THIS REPO AND HAS NEVER ONCE RUN.
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# Measured 2026-10-17: `renovate.json` is present, complete and thoughtfully written — it declares
+# managers, schedules, automerge classes and grouping. Nothing executes it. There is no Renovate
+# workflow (this is the first), no Renovate CronJob or Deployment in the cluster
+# (`kubectl get cronjob,deploy,job -A | grep -i renovate` -> nothing), and no bot account doing it
+# from outside. Meanwhile SEVEN files carry comments asserting "Renovate maintains this pin"
+# (platform/homepage/deployment.yaml:40, runbooks/serving/deployment.yaml:87,
+# platform/crafty/deployment.yaml:63, platform/vpn-ingress/daemonset.yaml:259,
+# platform/keycloak/postgres-cluster.yaml:75, apps/wave-0/nextcloud-db.yaml:302, and
+# platform/headlamp/helmrepository.yaml:2). Those comments have been false for the life of the file.
+#
+# The cost is measurable. With nothing bumping anything: the six per-app oauth2-proxy auth sidecars
+# sit on v7.8.1 while the platform oauth2-proxy in the SAME cluster runs v7.15.5; Keycloak (the IdP)
+# runs 26.0.6, a November-2024 release, 8 minors behind; Velero runs a RELEASE CANDIDATE
+# (velero-plugin-for-aws v1.14.3-rc.2) with a stable available; the CNPG Postgres image is 16.4,
+# missing every 16.5+ security release. A pin nothing maintains is not a pin, it is a fossil.
+#
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# WHY RENOVATE AND NOT FLUX IMAGE-AUTOMATION (the decision, and it reverses ADR-0009's split)
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# ADR-0009 divided the work: Renovate owns Helm chart/dependency bumps, Flux image-automation owns
+# per-service container image TAGS, deliberately non-overlapping. The second half of that split
+# cannot be delivered on this repo's terms, for three measured reasons:
+#
+#   1. IT WOULD BYPASS THE MERGE GATE, WHICH CLAUDE.md §1 FORBIDS. ImageUpdateAutomation does not
+#      open pull requests; it COMMITS AND PUSHES. Its default target is the source's own branch —
+#      `main` — so it would write straight past the PR flow. It can be pointed at a side branch
+#      instead, but then nothing opens the PR and bumps silently accumulate on a branch nobody
+#      merges. Either way a human or a second bot has to close the loop.
+#   2. ON THIS CLUSTER IT CANNOT EVEN DO THAT. Measured 2026-10-17: `main` is protected with
+#      `user_can_push: false`, and the `flux-system` GitRepository has NO `spec.secretRef` at all —
+#      the source is anonymous and read-only. Enabling write-back means minting an in-cluster git
+#      WRITE credential for the repository that defines the cluster, which is a materially larger
+#      blast radius than a CI-scoped token.
+#   3. RENOVATE ALREADY OPENS PRs NATIVELY, so every change it proposes goes through the gate with
+#      no extra machinery.
+#
+# So: Renovate owns BOTH halves, and Flux image-automation stays disabled. Note this is a change of
+# plan, not a change of state — image-automation has been commented out at
+# platform/kustomization.yaml:52 since it was authored, and the cluster has ZERO
+# ImageRepository/ImagePolicy/ImageUpdateAutomation objects (measured). Its two controllers ARE
+# installed and have idled for 43 days. Nothing is being torn down here; a path that was never
+# taken is being formally closed, and renovate.json's `docker`/`kubernetes` managers — disabled
+# purely to avoid fighting a controller that never ran — are switched on to fill the gap.
+#
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# THIS WORKFLOW MUST NOT BE ABLE TO SUCCEED SILENTLY
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# A SKIPPED Gitea job reports `success` (CLAUDE.md §2), so "green" is not evidence this ran. Two
+# guards:
+#   * The preflight step FAILS HARD (exit 1) if RENOVATE_TOKEN is absent, rather than skipping.
+#     A missing credential on a dependency bot is indistinguishable from "nothing to update", and
+#     that is precisely how renovate.json stayed inert for months.
+#   * The summary step records what ran into the job summary unconditionally.
+# What this still does NOT give you is an alert when the SCHEDULE stops firing. See the handoff
+# entry's "What I did NOT verify" — that gap is stated, not papered over.
+#
+# OPERATOR PREREQUISITE — ONE SECRET, NOT COMMITTED HERE:
+#   `RENOVATE_TOKEN` — a Gitea access token for the bot account, scopes `write:repository` and
+#   `read:user`. Set it as an Actions secret on this repository. Nothing in this repo mints,
+#   stores or references its value; it is consumed by name only.
+#   Until it exists, this workflow FAILS LOUDLY on every scheduled run. That is intended: a
+#   visible red is the correct state for "automation configured but not credentialed".
+#
+# THE FIRST RUN WILL WANT TO OPEN A LOT OF PRs. ~30 dependencies are behind. renovate.json caps the
+# blast radius (`prConcurrentLimit: 5`, `prHourlyLimit: 2`), but run it manually with
+# `dry_run: true` FIRST and read the log before letting the schedule loose. That is what
+# workflow_dispatch's default is for.
+name: renovate
+
+on:
+  schedule:
+    # 08:15 UTC daily. renovate.json ALSO carries `"schedule": ["after 3am and before 6am"]` with
+    # `"timezone": "America/New_York"`, and THAT inner schedule is authoritative — Renovate re-checks
+    # it and DECLINES TO OPEN PRs outside it. So this cron must fire INSIDE the local window, not
+    # merely near it.
+    #
+    # WAS `15 3 * * *`, WHICH NEVER WOULD HAVE WORKED (fixed 2026-10-18, before the first scheduled
+    # run). Gitea Actions cron is UTC, so 03:15 UTC is 23:15 the PREVIOUS EVENING in New York --
+    # 3h45m BEFORE the window opens, not "slightly ahead" of it as the original comment claimed.
+    # Every nightly run would have started, logged "not within schedule", opened zero PRs and
+    # exited SUCCESSFULLY. A green tick forever, and no updates ever: the exact "failure mode is a
+    # green tick" shape this repo keeps getting burned by, and indistinguishable from "nothing
+    # needed updating".
+    #
+    # THE WINDOW IN UTC MOVES WITH US DST, and a cron cannot follow it:
+    #   EDT (UTC-4, Mar-Nov): local 03:00-06:00 == 07:00-10:00 UTC
+    #   EST (UTC-5, Nov-Mar): local 03:00-06:00 == 08:00-11:00 UTC
+    # 08:15 UTC is the only quarter-hour comfortably inside BOTH (04:15 EDT / 03:15 EST), so the
+    # schedule keeps working across both switchovers without an edit. Do not "tidy" this back to an
+    # early-morning-UTC value without redoing this arithmetic.
+    - cron: "15 8 * * *"
+  workflow_dispatch:
+    inputs:
+      dry_run:
+        description: "Dry run (no PRs, no branches). Leave true for the first run."
+        required: false
+        default: "true"
+      log_level:
+        description: "Renovate LOG_LEVEL (info | debug)"
+        required: false
+        default: "info"
+
+concurrency:
+  # Never two Renovate runs at once: concurrent runs race on the same branches and can open
+  # duplicate PRs. Not cancel-in-progress — a half-finished Renovate run can leave a branch pushed
+  # with no PR, so let the first one finish.
+  group: renovate
+  cancel-in-progress: false
+
+jobs:
+  renovate:
+    runs-on: ubuntu-latest
+    # Renovate scans every manager across the whole repo; 40m is generous for this repo's size and
+    # well under a runner's patience. Without a timeout a wedged run holds the `renovate`
+    # concurrency group and silently blocks every subsequent scheduled run.
+    timeout-minutes: 40
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Preflight - the token must exist, and its absence must be LOUD
+        env:
+          RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
+        run: |
+          set -euo pipefail
+          if [ -z "${RENOVATE_TOKEN:-}" ]; then
+            echo "::error::RENOVATE_TOKEN is not set on this repository." >&2
+            cat >&2 <<'MSG'
+
+          Renovate cannot run without a Gitea access token, and this step fails rather than
+          skipping ON PURPOSE: a skipped Gitea job reports `success`, so skipping here would make
+          "dependency automation is not running" look exactly like "everything is up to date".
+          That is the failure mode that let renovate.json sit inert in this repo for months while
+          seven files claimed "Renovate maintains this pin".
+
+          TO FIX: create a Gitea access token for the bot account with scopes
+          `write:repository` and `read:user`, then add it to this repository as the Actions
+          secret RENOVATE_TOKEN. Do not commit it anywhere.
+          MSG
+            exit 1
+          fi
+          echo "RENOVATE_TOKEN is present (value not printed)."
+
+      - name: Run Renovate
+        uses: renovatebot/[email protected]
+        with:
+          # Pinned, and Renovate maintains this pin itself: renovate.json's enabledManagers
+          # includes `github-actions`, and `helpers:pinGitHubActionDigests` is in its `extends`.
+          # This is the one dependency in the repo whose updater is its own updater.
+          configurationFile: renovate.json
+          token: ${{ secrets.RENOVATE_TOKEN }}
+        env:
+          # Self-hosted Gitea. `platform` + `endpoint` were the two values renovate.json's own
+          # header said were "set at runtime by the operator's renovate runner" — this is that
+          # runner, so they are set here rather than in the repo config, which keeps renovate.json
+          # portable and free of this instance's URL.
+          RENOVATE_PLATFORM: gitea
+          RENOVATE_ENDPOINT: https://gitea.home.keithjackson.dev/api/v1
+          RENOVATE_REPOSITORIES: Home-Network/home-network-k3s-impl
+          # Renovate must never push to a protected branch; it works on its own branches and opens
+          # PRs against this base. Stated explicitly rather than inherited, because this is the
+          # property that keeps the whole mechanism inside CLAUDE.md §1's branch -> PR -> gate path.
+          RENOVATE_BASE_BRANCHES: main
+          RENOVATE_ONBOARDING: "false"
+          RENOVATE_REQUIRE_CONFIG: "required"
+          RENOVATE_DRY_RUN: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run == 'true' && 'full' || '' }}
+          LOG_LEVEL: ${{ github.event_name == 'workflow_dispatch' && inputs.log_level || 'info' }}
+
+      - name: Summary
+        if: always()
+        run: |
+          {
+            echo "## Renovate"
+            echo
+            echo "- trigger: \`${{ github.event_name }}\`"
+            echo "- dry run: \`${{ github.event_name == 'workflow_dispatch' && inputs.dry_run || 'false' }}\`"
+            echo "- outcome: \`${{ job.status }}\`"
+            echo
+            echo "Renovate owns BOTH Helm chart bumps and container image tags here; Flux"
+            echo "image-automation is deliberately disabled (platform/kustomization.yaml)."
+            echo "Every change it proposes arrives as a pull request and must pass the merge gate."
+          } >> "$GITHUB_STEP_SUMMARY"

apps/wave-0/plex.yaml

@@ -77,7 +77,7 @@ spec:
   storageClassName: longhorn
   resources:
     requests:
-      storage: 8Gi
+      storage: 16Gi
 ---
 # PVC plex-media (R-STATE)
 apiVersion: v1

catalog/plex.yaml

@@ -351,7 +351,7 @@ exposure: public              # edge port-forward + public DNS; clients + Seerr
 egress: internet              # normal WAN egress (docs/03; R-NET-05)
 storage:
   - name: config
-    size: 8Gi
+    size: 16Gi
     class: longhorn           # Plex DB/metadata: replicated app state on the 3-replica tier (docs/04; R-STATE-01/02)
     mountPath: /config        # plexinc/pms-docker writes its identity/claim + "Library" DB under /config, NOT the
                               # renderer default /data/config. Without this the PVC sat unused and Plex wrote to the

catalog/qbittorrent.yaml

@@ -359,7 +359,7 @@ storage:
                                 # The live /config contents were copied onto each PVC before this landed,
                                 # so API keys and credentials other apps reference were preserved.
   # THE DOWNLOADS AREA — ONE mount of the SAME export sonarr/radarr/plex already consume
-  # (192.168.2.102:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
+  # (192.168.2.30:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
   # /downloads, which is precisely where this app is ALREADY configured to write.
   #
   # WHY subPath: downloads AND NOT THE EXPORT ROOT AT /media (the shape sonarr/radarr use):
@@ -434,7 +434,7 @@ dependencies: []                # default-deny: reaches only DNS; arr reach it n
                                 # for weeks and neither has any CNP naming the NFS server or port 2049
                                 # (grep apps/wave-1/sonarr.yaml: none). The SAME fact is what keeps the VPN
                                 # egress gateway off this traffic (its selector is a podSelector), and
-                                # 192.168.2.102 is inside the 192.168.0.0/16 excludedCIDR besides — see
+                                # 192.168.2.30 is inside the 192.168.0.0/16 excludedCIDR besides — see
                                 # platform/vpn-egress/egress-steering.yaml, "BULK NFS".
 ingressRoutes:
   - host: qbittorrent.${BASE_DOMAIN}

clusters/base/cluster-vars.yaml

@@ -234,26 +234,28 @@ data:
   WG_CLIENT_CIDR: "10.90.0.0/24"             # remote VPN clients; .1 = server. NOT a LAN range
 
   # --- Bulk (external NFS) storage host (docs/04 bulk tier) ---------------------------------------
-  # THE address of the Proxmox host whose ZFS datasets back every `class: bulk` volume: the media
-  # library (Storage/plex, 8.33 T — plex, sonarr, radarr) and the legacy Nextcloud archive
+  # THE address of the NAS whose ZFS datasets back every `class: bulk` volume: the media library
+  # (Storage/plex, 8.33 T — plex, sonarr, radarr, qbittorrent) and the legacy Nextcloud archive
   # (Storage/nextcloud, 270 G). The renderer emits it into each static NFS PersistentVolume's
   # `spec.nfs.server` (catalog/render.py, BULK_NFS_SERVER); the export PATH is per-dataset and is
   # declared in the ServiceDefinition, because the path is identical in every environment and only
   # the ADDRESS is not.
   #
-  # WHY THIS IS A KNOB AND NOT A LITERAL — the two clusters are on OPPOSITE SIDES of pfSense.
-  # The storage host has exactly one interface, vmbr0 192.168.1.9/24, on pfSense's LAN.
-  #   dev  nodes are 192.168.1.40/.41/.42 — the SAME L2 segment. They reach it directly, and one
-  #        NFS export to them already exists on that host (/mnt/Storage/k8s-volumes in
-  #        /etc/exports), so this path is proven, not theoretical. Base default = the dev value,
-  #        per the convention of the LAN block above.
-  #   prod nodes are 192.168.2.10/.11/.12 — pfSense's WAN. tcp/2049 does NOT traverse that today;
-  #        22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. Opening
-  #        the storage path is a PENDING OPERATOR DECISION with two candidate outcomes that give
-  #        DIFFERENT values here: 192.168.2.102 (another port-forward) or 192.168.1.9 (a route).
-  # So prod holds a loud TBD-PROD-* placeholder — see clusters/prod/kustomization.yaml for the
-  # blast radius of shipping it unset, which is NOT the usual "fails to apply".
-  BULK_NFS_SERVER: "192.168.1.9"
+  # IT IS NO LONGER A CROSS-NETWORK KNOB (2026-10-07). This used to read "the two clusters are on
+  # OPPOSITE SIDES of pfSense": the datasets lived on a Proxmox host whose only interface was
+  # vmbr0 192.168.1.9/24 on pfSense's LAN, which prod (192.168.2.0/24, pfSense's WAN) could reach
+  # only through a DNAT at 192.168.2.102. Both pools were physically moved onto a standalone
+  # TrueNAS at 192.168.2.30 — prod's OWN /24 — so there is no crossing to arrange and no candidate
+  # values to choose between. Measured 2026-10-07: 192.168.2.102:2049 no longer answers;
+  # 192.168.2.30:2049 and :111 do. There is now exactly ONE storage host and one correct value;
+  # this stays a knob only because the renderer reads it per-environment.
+  #
+  # *** CHANGING THIS VALUE IS NOT A PLAIN RECONCILE. *** A bound PersistentVolume's spec.nfs is
+  # immutable — measured by server dry-run 2026-10-07:
+  #     spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation
+  # so Flux's apply FAILS rather than updating, and the pods keep their old mount. The PVs must be
+  # deleted and recreated; the operator sequence is docs/BULK-NFS-STORAGE.md §10.
+  BULK_NFS_SERVER: "192.168.2.30"
 
   # Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style.
   # Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db,

clusters/base/kured.yaml

@@ -0,0 +1,98 @@
+---
+# Flux Kustomization: kured (Kubernetes Reboot Daemon) — reconciles ./platform/kured.
+# Performs the cordon / drain / reboot / uncordon of a node that has a pending OS reboot, strictly
+# one node at a time, inside a nightly window, gated on Longhorn volume health. Design rationale,
+# every tuned value, and the honest list of what is NOT mitigated: platform/kured/helmrelease.yaml.
+#
+# WHY ITS OWN KUSTOMIZATION, AND NOT ONE LINE IN platform/kustomization.yaml
+#
+# The same argument the Cilium split-out recorded in platform/kustomization.yaml and intel-gpu-plugin
+# reused, except the consequence is worse here than for either of them:
+#
+#   * `platform` is `wait: true` over twelve-plus components, so its Ready condition cannot tell you
+#     "the reboot daemon is wedged" from "longhorn is slow".
+#   * The only emergency valve would be `flux suspend kustomization platform` — freezing storage,
+#     TLS, DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a DaemonSet.
+#   * That tier's 10m interval / 5m timeout are called out there as WRONG for a 3-node DaemonSet
+#     rollout. This is a 3-node DaemonSet rollout.
+#   * And `platform` gates platform-config, which gates apps-wave-0 (wait:true), which gates
+#     apps-wave-1. A reboot daemon that failed to go healthy inside that tier would stall EVERY
+#     APPLICATION in the cluster on "dependency not ready". A component added to make maintenance
+#     safer must not acquire the ability to cause an outage; that trade is the whole argument.
+#
+# dependsOn: [platform] — AND THIS ONE IS A REAL EDGE, NOT CONVENTION
+#
+# Not `cilium` (what intel-gpu-plugin uses) and not `platform-config` (what the leaf UIs use). It is
+# `platform`, for two independent reasons:
+#
+#   1. CRD ORDERING (F-MON1). platform/kured ships a PrometheusRule, and the chart generates a
+#      ServiceMonitor. Both are CRs whose CRDs (monitoring.coreos.com) are installed by
+#      kube-prometheus-stack INSIDE the `platform` tier. Flux dry-runs every object in a pass before
+#      applying any of them, so a pass containing an unknown kind ABORTS — the failure that made the
+#      CDI StorageProfiles a true deadlock on prod on 2026-09-05 (see
+#      clusters/base/platform-config-cdi.yaml). Depending on `platform` guarantees the CRDs exist
+#      before this pass is attempted.
+#   2. FUNCTIONAL PREREQUISITE. kured's reboot gate QUERIES Prometheus
+#      (`prometheusUrl` in the HelmRelease). Prometheus lives in platform/monitoring. If kured ran
+#      first it would get connection errors, treat the gate as unevaluable and skip every reboot —
+#      the correct fail-safe direction, but silent. Longhorn, whose volume health IS the gate, is
+#      also in that tier.
+#
+# It does NOT need platform-config: no Keycloak, no TLS, no split-horizon DNS, no Gateway, no
+# HTTPRoute. kured serves no HTTP to users.
+#
+# wait: true — A HEALTH SIGNAL, NOT A GATE. Following intel-gpu-plugin rather than the leaf-UI
+# template. A leaf UI is legitimately not-Ready for long stretches of a cold start while it waits on
+# Keycloak and DNS, so its health is uninformative; kured has no such dependency — it is a DaemonSet
+# that opens a socket and watches a file, so on healthy nodes it is Ready in seconds and not-Ready
+# genuinely means BROKEN. This turns `flux get kustomization kured` into a one-line answer to "is
+# reboot automation live on all three nodes", which is the question the operator must be able to
+# answer before trusting a kernel upgrade to it.
+#
+# NOTHING IN THE GRAPH dependsOn THIS, so wait:true gates nothing — the blast radius of an unhealthy
+# kured is this Kustomization's own Ready condition. Do not add `dependsOn: [kured]` anywhere: that
+# would hand a node-rebooting DaemonSet the power to stall whatever depends on it, which is precisely
+# what the placement above exists to prevent.
+#
+# prune: true is safe and deliberate. If this Kustomization is deleted, kured's DaemonSet goes with
+# it and reboots simply stop happening — nodes are left uncordoned and running. The one caveat is
+# that pruning mid-reboot would leave a node CORDONED with no daemon to uncordon it; that is a
+# `kubectl uncordon` away and is covered in runbooks/node-reboot-automation.md.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: kured
+  namespace: flux-system
+spec:
+  # A HelmRelease plus five static manifests, nothing to re-render; kustomize-controller is
+  # source-triggered, so this is only the drift-correction floor.
+  interval: 10m
+  retryInterval: 2m
+  # 10m, not the 5m the `platform` tier uses: a 3-node DaemonSet rollout under `wait: true` can
+  # exceed 5m on a cold start (image pull on three nodes), which is one of the reasons the Cilium
+  # split-out gave for leaving that tier.
+  timeout: 10m
+  prune: true
+  # See the header: a health SIGNAL for a component nothing depends on, not a gate.
+  wait: true
+  dependsOn:
+    - name: platform   # monitoring CRDs + Prometheus + Longhorn; see the header, this is load-bearing
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/kured
+  # REQUIRED, not boilerplate. platform/kured/alerts.yaml renders `${BASE_DOMAIN}` into the
+  # `runbook_url` annotation of all three rules. Without this block that token becomes the EMPTY
+  # STRING and the Kustomization still reports Ready — Flux's substitution fails OPEN — so every
+  # alert would ship a dead link to `https:///runbooks/node-reboot-automation`. The gate stage
+  # `flux-postbuild-substitution` asserts the converse (a rendered token implies this block), and
+  # BASE_DOMAIN is declared in clusters/base/cluster-vars.yaml.
+  #
+  # Nothing else in this path contains `${...}` or a bare `$`: the HelmRelease's alertFilterRegexp is
+  # a plain anchored alternation, and there are no shell snippets anywhere in the component — which
+  # is deliberate, because envsubst would eat a braced shell variable and `$$` would trip the
+  # dollar-collapse assertion in the same check.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars

clusters/base/kustomization.yaml

@@ -85,4 +85,17 @@ resources:
   # pm-agent-credentials.sops.yaml, so both overlays add a SOPS decryption patch for it (by
   # name). See clusters/base/pm-agent.yaml and platform/pm-agent/README.md.
   - pm-agent.yaml        # Flux Kustomization -> ./platform/pm-agent (dependsOn apps-wave-0)
+  # kured (Kubernetes Reboot Daemon) — its OWN Kustomization (dependsOn platform, wait:true), NOT a
+  # line in platform/kustomization.yaml. Same reasoning as the Cilium and intel-gpu-plugin
+  # split-outs, with a sharper consequence: `platform` gates platform-config and every apps wave, so
+  # a wedged reboot DaemonSet placed in that wait:true tier could stall every application in the
+  # cluster — a component added to make maintenance SAFER must not be able to cause an outage.
+  # wait:true here is a HEALTH SIGNAL, not a gate: nothing dependsOn it, and nothing should.
+  # The `platform` edge is load-bearing rather than conventional — it supplies the
+  # monitoring.coreos.com CRDs this component's PrometheusRule/ServiceMonitor need (F-MON1), and
+  # Prometheus itself, which kured QUERIES before every reboot to gate on Longhorn volume health.
+  # It reboots a node only when /var/run/reboot-required exists, which on these nodes only a
+  # deliberate operator kernel upgrade creates — the human gate is the sentinel, not the daemon.
+  # See clusters/base/kured.yaml and platform/kured/helmrelease.yaml.
+  - kured.yaml           # Flux Kustomization -> ./platform/kured (dependsOn platform)
   - cluster-vars.yaml    # overlay knob surface (base defaults)

clusters/dev/kustomization.yaml

@@ -279,15 +279,15 @@ patches:
       - op: replace
         path: /data/CRAFTY_L4_VIP
         value: "192.168.1.55"
-      # Bulk (external NFS) storage host: the Proxmox host itself, on the SAME LAN as dev's nodes
-      # (192.168.1.40/.41/.42), which already hold an NFS export from it (/mnt/Storage/k8s-volumes).
-      # Restated verbatim rather than inherited, per this overlay's "supply every knob" convention.
-      # Dev is therefore the environment in which the bulk manifests can actually be VALIDATED
-      # end-to-end before prod's network path exists — provided the additive `zfs set sharenfs=`
-      # change described in docs/BULK-NFS-STORAGE.md is applied for the dev node addresses.
+      # Bulk (external NFS) storage host. THIS VALUE IS INERT — dev is decommissioned, its nodes
+      # (192.168.1.40/.41/.42) no longer exist, and the Proxmox host this used to name (192.168.1.9)
+      # went with the pools it held. It is restated per this overlay's "supply every knob"
+      # convention, not because anything consumes it. It carries the real address so that no file
+      # in the tree renders a host that is gone; it has NOT been proven from a dev node, because
+      # there is no dev node to prove it from.
       - op: replace
         path: /data/BULK_NFS_SERVER
-        value: "192.168.1.9"
+        value: "192.168.2.30"
       # Off-cluster backup target: the on-Proxmox MinIO (plain HTTP, path-style) that every CNPG
       # barmanObjectStore and Velero's backupStorageLocation write to.
       #

clusters/prod/kustomization.yaml

@@ -540,60 +540,48 @@ patches:
         path: /data/LAN_CIDR
         value: "192.168.2.0/24"
       # ------------------------------------------------------------------------------------------
-      # BULK (EXTERNAL NFS) STORAGE HOST — *** NOT YET DECIDED. THIS PLACEHOLDER IS LOAD-BEARING. ***
+      # BULK (EXTERNAL NFS) STORAGE HOST — the TrueNAS at 192.168.2.30.
       # ------------------------------------------------------------------------------------------
       # Backs every `class: bulk` volume: the 8.33 T media library (Storage/plex — plex, sonarr,
-      # radarr) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
+      # radarr, qbittorrent) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
       #
-      # THE STORAGE HOST IS NOT REACHABLE FROM PROD AT ALL RIGHT NOW. It has one interface,
-      # vmbr0 192.168.1.9/24, on pfSense's LAN; prod's nodes are on 192.168.2.0/24, pfSense's WAN.
-      # 22 and 8006 answer from prod only because pfSense port-forwards them via 192.168.2.102;
-      # 2049 and 111 do not. Opening that path is an OPERATOR decision, and the two candidate
-      # outcomes give DIFFERENT values here:
-      #     port-forward 192.168.2.102:2049 -> 192.168.1.9:2049   =>  "192.168.2.102"
-      #     a WAN->LAN route to 192.168.1.0/24                    =>  "192.168.1.9"
-      # Guessing either one authors a prod cluster that mounts nothing, or — worse, if the guess
-      # ever became reachable — mounts something unintended. So it stays a placeholder.
+      # THIS WAS THE REPO'S ONE LOAD-BEARING PLACEHOLDER, AND THE REASON IT COULD BE IS NOW GONE.
+      # The datasets lived on a Proxmox host with a single interface on pfSense's LAN
+      # (192.168.1.9), which prod could reach only through a DNAT at 192.168.2.102 — so the correct
+      # value here depended on an undecided operator choice, and guessing it would have authored a
+      # cluster that mounts nothing. Both pools were physically moved onto a standalone TrueNAS at
+      # 192.168.2.30, prod's OWN /24. No crossing to arrange, no candidates to choose between.
+      # Measured 2026-10-07: 192.168.2.102:2049 no longer answers; 192.168.2.30:2049 and :111 do.
       #
-      # WHICH PORTS THE DECISION MUST OPEN: exactly ONE, tcp/2049. The PVs mount nfsvers=4.1, and
-      # NFSv4 carries mount and lock in-protocol — no rpcbind/111, no mountd, no statd. Choosing v4
-      # over v3 was in part this decision's cost: v3 would have needed four ports and three of them
-      # on ephemeral numbers.
-      #
-      # WHAT THE EXPORT MUST ALLOW IS ALREADY SETTLED, and it is the same under BOTH outcomes.
-      # Measured 2026-09-07 from this workstation (192.168.2.132) through the existing port-forward:
-      # the storage host logged the connection as coming FROM 192.168.2.132, not from a pfSense LAN
-      # address (`ss -tn` peer 192.168.2.132, and the host's only route off-LAN is its default via
-      # 192.168.1.1). pfSense DNATs WITHOUT source NAT, so the NFS server will see prod's REAL node
-      # addresses either way. The additive `zfs set sharenfs=` values are in docs/BULK-NFS-STORAGE.md.
-      #
-      # *** BLAST RADIUS — READ BEFORE MERGING THIS TO A BRANCH PROD RECONCILES. ***
-      # A PersistentVolume's spec.nfs.server is an unvalidated string: the API server ACCEPTS this
-      # placeholder, the PV is created and the PVC BINDS to it (binding is pure control-plane). The
-      # failure surfaces only when a pod mounts — MountVolume.SetUp "Failed to resolve server
-      # TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr and nextcloud would sit in
-      # ContainerCreating on prod. Unlike the LAN knobs above, this placeholder does NOT fail at
-      # apply time and there is no CRD validation to lean on. Set this value BEFORE prod reconciles
-      # a revision containing the bulk volumes; the dev overlay renders a working value, which is
-      # where these manifests are meant to be proven first.
+      # ONE PORT IS STILL ALL IT NEEDS: tcp/2049. The PVs mount nfsvers=4.1 and NFSv4 carries mount
+      # and lock in-protocol — no rpcbind/111, no mountd, no statd. The NAS advertises
+      # `+3 +4 +4.1 +4.2` (/proc/fs/nfsd/versions, measured), so 4.1 negotiates.
       - op: replace
         path: /data/BULK_NFS_SERVER
-        # 192.168.2.102 is pfSense's WAN address, which DNATs tcp/2049 through to the Proxmox
-        # host's REAL address 192.168.1.9 on pfSense's LAN. Prod sits on the WAN side and cannot
-        # reach 192.168.1.9 directly, so the cluster mounts the NAT address. Dev is on the LAN and
-        # keeps the base value 192.168.1.9.
+        # VERIFIED ON THE NAS 2026-10-07, before any cluster touched it (`exportfs -v`):
+        #   /mnt/Storage/plex       192.168.2.10/.11/.12  rw,sec=sys,secure,root_squash,no_all_squash
+        #   /mnt/Storage/nextcloud  192.168.2.10/.11/.12  ro,sec=sys,secure,root_squash,no_all_squash
         #
-        # PROVEN from k3s-prod-1 on 2026-09-12, not inferred:
-        #   - mount of 192.168.2.102:/mnt/Storage/plex negotiated nfs4 vers=4.1 proto=tcp
-        #   - a write as uid 1000 succeeded. Tested as 1000 and NOT root deliberately: the export
-        #     carries no_root_squash, so a root write would have passed and proved nothing, while
-        #     LSIO images default to PUID 911 and would have gone Ready then failed every import
-        #     with EACCES. The tree is 0755 1000:3000, so uid 1000 is the only writer.
-        #   - ln() across downloads/ -> tv/ returned link count 2 (same inode), so the arrs get
-        #     real atomic moves instead of silently degrading to copy-then-delete on EXDEV.
-        # pfSense DNATs WITHOUT source NAT (the host logs prod's own node addresses as the peer),
-        # so the ZFS export ACL names 192.168.2.10/.11/.12 directly rather than one translated IP.
-        value: "192.168.2.102"
+        # no_all_squash is the flag the entire permission model rests on: client uids pass through
+        # untouched, so the arrs' PUID=1000 arrives as the owner of a 0755 1000:3000 tree. sec=sys
+        # keeps that numeric rather than principal-mapped. The export deliberately does NOT carry
+        # no_root_squash, which the Proxmox one did: nothing here needs it, because every write to
+        # the tree happens after the LSIO drop to uid 1000 and the images only chown their Longhorn
+        # /config. Both subPath directories (plex/downloads, nextcloud/nextcloud) already exist at
+        # 0755 1000:3000, so a squashed root can still traverse them for subPath resolution.
+        #
+        # The ro= posture on the archive is now SERVER-enforced, which the old export also managed.
+        # The media tree's read-only half (plex) stays a CLIENT-side guarantee — all four media pods
+        # can land on any node, so the server sees the same three addresses for reader and writers
+        # alike and cannot tell them apart. Unchanged by the move; see docs/BULK-NFS-STORAGE.md §5.
+        #
+        # *** CHANGING THIS VALUE REQUIRES DELETING AND RECREATING THE PVs. *** A bound PV's
+        # spec.nfs is immutable — measured by server dry-run 2026-10-07:
+        #     spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation
+        # so Flux's apply FAILS rather than updating and the pods keep their old, dead mount. The
+        # operator sequence is docs/BULK-NFS-STORAGE.md §10. Retain + the renderer's pre-set
+        # claimRef make it safe: no data is touched and each recreated PV rebinds its own PVC.
+        value: "192.168.2.30"
       # ==========================================================================================
       # OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05; bucket + token 2026-09-07).
       # ==========================================================================================

conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh

@@ -363,6 +363,54 @@ SCRAPE_OWNERS: list[dict] = [
         "why": "velero_* powers VeleroBackupFailure / PartialFailure / NoRecentBackup — the rules "
                "that failed to notice prod running with zero volume backups",
     },
+    # ── ADDED 2026-10-17 (feat/automated-upgrades). A NEW OWNER, NOT A RELAXED ASSERTION. ──────
+    # This entry is purely additive: it registers a new metric family and its enablement, and it
+    # changes nothing about how any existing owner is evaluated. It makes the check STRICTER for the
+    # component it covers, not looser — it adds two `require` patterns plus a `companion` obligation
+    # that platform/kured must keep satisfying, so a future edit that disables kured's Service,
+    # ServiceMonitor or NetworkPolicy FAILS the gate instead of silently re-inerting three alerts.
+    #
+    # WHY THE CHECK HAD TO BE TOUCHED AT ALL, rather than the manifest being wrong: assertion (3)
+    # fails any alert over a metric no owner claims, and the escape hatch (KNOWN_UNWIRED_METRICS) is
+    # SHRINK-ONLY, so declaring `kured_*` as accepted debt is not permitted and would be the wrong
+    # answer anyway — the scrape genuinely IS wired here. The only two honest options were to
+    # register the owner (this) or to ship the reboot automation with no alerting at all, which for a
+    # component whose every failure mode is silent would have been the worse engineering choice.
+    # `conformance/**` is a sensitive approval path precisely so that this reasoning gets read.
+    {
+        "owner": "kured: service.create + metrics.create in the HelmRelease",
+        "prefixes": ("kured_",),
+        "file": "platform/kured/helmrelease.yaml",
+        # BOTH keys are required and the chart defaults BOTH to false. `metrics.create` makes the
+        # ServiceMonitor; `service.create` makes the Service it selects. Enable only the monitor and
+        # it selects nothing, so the single kured metric is never scraped — and a PrometheusRule over
+        # a series no target produces is INERT, which for this component means its own liveness
+        # alarm is the thing that breaks. Matched as two separate anchored blocks rather than one
+        # pattern because they are separate top-level values keys, not nested.
+        "require": [
+            r"service:\s*\n\s*create:\s*true",
+            r"metrics:\s*\n\s*create:\s*true",
+        ],
+        "forbid": [],
+        "why": "kured_reboot_required is the ONLY metric kured exposes (one GaugeVec, "
+               "cmd/kured/main.go:92-96) and it powers all three platform-health.node-reboots "
+               "rules. Those rules are the only way to learn that OS reboot automation has stalled: "
+               "kured stays 1/1 Running while rebooting nothing, so a missed scrape here is "
+               "indistinguishable from a cluster that needed no reboots",
+        # The ServiceMonitor ALONE is not sufficient — the same NetworkPolicy trap Longhorn hit.
+        "companion": {
+            "file": "platform/kured/kustomization.yaml",
+            "require": [r"^\s*-\s*networkpolicy\.yaml\s*$"],
+            "why": "platform/kured/networkpolicy.yaml ships a default-deny baseline (the kured "
+                   "namespace is a platform namespace, so Kyverno's app-namespace-scoped generator "
+                   "does not cover it) AND the ingress allow that lets Prometheus reach :8080 "
+                   "through it. Drop the networkpolicy from this kustomization and the deny-all "
+                   "goes too, so this currently fails SAFE — but reinstating the baseline without "
+                   "the allow reproduces Longhorn's measured failure exactly: targets DISCOVERED, "
+                   "every scrape 'context deadline exceeded', TSDB empty, all three rules quiet. "
+                   "Keeping the two in one file makes that pair inseparable",
+        },
+    },
 ]
 
 # `up` is produced by EVERY scrape target, so no owner can meaningfully claim it and assertion (1)

conformance/tests/l0-static/ha-control-plane-as-code/check.sh

@@ -445,8 +445,21 @@ join = re.search(r"--server https://\{\{\s*k3s_api_host", K3S_TEXT)
 check(join is not None,
       "roles/k3s_server does not join secondaries with `--server https://{{ k3s_api_host }}` "
       "— without a join target the servers are separate single-node clusters")
-join_when = re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:\s*k3s_server_mode\s*==\s*'secondary'",
-                      K3S_TEXT, re.S)
+# LINE-BOUNDED ON PURPOSE -- DO NOT "SIMPLIFY" BACK TO `.*` + re.S.
+# This was `--server https://.*?\n(?:.*\n){0,10}?\s*when:...` with re.S, which made every `.`
+# match newlines too. That is nested unbounded repetition (a `{0,10}` of a quantifier that can each
+# swallow the whole document), so a FAILED or late match backtracks exponentially. It cost
+# **19m54s of CPU in CI** (measured 2026-10-18 via `docker top` on the live job container) and made
+# the whole merge gate take ~28 minutes instead of ~2 on any change touching `infra/**` -- because
+# `ci/touched-requirements.sh` maps `infra/*` to R-ENV-01, which pulls in this whole-repo check.
+# It was latent for as long as the regex existed and only crossed the threshold when
+# roles/k3s_server grew by ~100 lines (the kubelet graceful-shutdown config, PR #71): K3S_TEXT is
+# every *.yml in that role concatenated, so the role growing is the input growing.
+# `[^\n]*` cannot cross a line, so the `{0,10}` is a genuine ten-line window with nothing to
+# backtrack into. Measured on the same input: 0.2 ms, same verdict (match found).
+join_when = re.search(
+    r"--server https://[^\n]*\n(?:[^\n]*\n){0,10}?[ \t]*when:[ \t]*k3s_server_mode[ \t]*==[ \t]*'secondary'",
+    K3S_TEXT)
 check(join_when is not None,
       "the join install task is not gated on k3s_server_mode == 'secondary' — the founder "
       "would also try to join itself")

decisions/2026-10-06-new-peer-ignores-template-comments.md

@@ -0,0 +1,73 @@
+---
+date: 2026-10-06
+branch: fix/new-peer-ignores-template-comments
+title: new-peer.sh refused the first VPN address because its duplicate check matched its own commented-out example
+---
+
+## What changed
+
+The operator's first real enrolment failed:
+
+```
+$ ./infra/vpn-ingress/new-peer.sh keith-phone 2
+new-peer.sh: 10.90.0.2/32 is already allocated in platform/vpn-ingress/peers.yaml. Pick another octet.
+```
+
+**Nothing was allocated.** `peers.yaml` had zero peers — the pod's own init container reported
+`wg0.conf assembled: 0 peer(s)`. The duplicate-address check grepped the WHOLE file, and
+`peers.yaml` line 66 is a **commented-out example stanza** I wrote, containing literally
+`#   AllowedIPs = 10.90.0.2/32`. So `.2` — the first address anybody would ask for, and the one my
+own runbook uses in its example — was permanently un-allocatable, with a message asserting the
+opposite of the truth.
+
+Both halves fixed, because either alone leaves the trap half-set:
+
+- **`infra/vpn-ingress/new-peer.sh`**: the check strips comment lines before matching
+  (`grep -vE '^[[:space:]]*#' | grep -q …`). That is the correct predicate rather than a patch: a
+  real stanza's `AllowedIPs` line is never commented, while both the template and every peer's
+  `# <name> — <owner>` label always are.
+- **`platform/vpn-ingress/peers.yaml`**: the example now uses placeholders that cannot match
+  anything real — `# <device-name> — <owner>, enrolled <YYYY-MM-DD>` and
+  `AllowedIPs = 10.90.0.<octet>/32`. A future grep cannot be fooled by it either.
+
+The bug class is worth naming: **a documentation example that is byte-identical to real data will
+eventually be parsed as real data.** The same shape caused two earlier defects in this component —
+`flux-postbuild-substitution` flagged a `${...}` written inside a comment *explaining* that hazard,
+and `no-plaintext-secrets` flagged the `.sops.yaml.example` because the kustomization named it in a
+comment. Three instances now; the rule is to make examples structurally unmatchable, not merely
+commented out.
+
+No sensitive paths (`platform/vpn-ingress/**`, `infra/vpn-ingress/**`, `decisions/**`), so this
+merges on a green gate with no override.
+
+## What I verified
+
+Run in the pinned toolbox against the live tree, 2026-10-06.
+
+- **`.2` now works**, and it went through the REAL key path: `new-peer.sh keith-phone 2` rendered a
+  profile with `Address 10.90.0.2/32`, `AllowedIPs = 10.90.0.2/32` in the printed stanza, full
+  tunnel, at `.cache/wg-peers/keith-phone.conf` (0600). The throwaway keypair was deleted after.
+- **This incidentally closes a gap left open in #60**: the profile could only have been rendered by
+  deriving the server's public key via `sops -d --extract … | wg pubkey` from
+  `wg-server-key.sops.yaml` — so that pipeline is now proven **against the operator's real key**,
+  not just against a placeholder it was expected to reject.
+- **A real duplicate is still refused.** Appending an uncommented
+  `AllowedIPs = 10.90.0.5/32` stanza and asking for octet 5 produced the correct refusal; the tree
+  was restored afterwards and `git diff --stat` showed only the intended template edit.
+- **Zero active allocations remain**: `grep -vE '^[[:space:]]*#' peers.yaml | grep -c AllowedIPs`
+  → `0`.
+- **Toolbox gate**: rc 3, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`,
+  `l1` and `approval-policy` all pass; rc 3 is the two environmental skips (`gitea-settings`,
+  conformance L2/L3). **Not green. Reported as rc 3.**
+
+## What I did NOT verify
+
+- **No peer is enrolled by this branch.** The keypair I generated was a throwaway and was deleted;
+  the operator must run the script themselves, since the device's private key must exist only on
+  the device. So the end of the enrolment path — a stanza merged into `peers.yaml`, Flux rolling
+  the DaemonSet, and `wg show` listing a peer — is still unexercised.
+- **The duplicate-NAME warning** (`grep -qi "^ *# *${NAME} "`) deliberately still inspects
+  comments, because per-peer labels are comments. It is no longer tripped by the template, but I
+  did not test it against a real enrolled peer.
+- **Nothing about client traffic.** The WAN port-forward is still not configured and there is still
+  no off-LAN vantage point, so the tunnel has carried no packet.

decisions/2026-10-06-peers-configmap-content-hash.md

@@ -0,0 +1,103 @@
+---
+date: 2026-10-06
+branch: fix/peers-configmap-content-hash
+title: A merged VPN peer never reached the tunnel — the peer register was a fixed-name ConfigMap, so nothing rolled the pods
+---
+
+## What changed
+
+The operator enrolled a phone, merged the peer, and **`wg show` listed no peers.** Measured on prod
+2026-10-06, with the peer already on `main`:
+
+| observation | value |
+|---|---|
+| peer stanza on `main` | present (`AllowedIPs = 10.90.0.2/32`) |
+| GitRepository artifact | `main@sha1:e1bebb8` — Flux had fetched it |
+| `vpn-ingress` Kustomization `lastAppliedRevision` | `main@sha1:5928d81` — **behind**, 10m interval |
+| live `wg-peers` ConfigMap, active `AllowedIPs` lines | **0** |
+| DaemonSet volume reference | `configMap.name: wg-peers` — a **fixed** name |
+| DaemonSet `generation` / `observedGeneration` | **1 / 1** — it had **never rolled** |
+| init container, all three pods | `wg0.conf assembled: 0 peer(s)` |
+
+The lag was only the reconcile interval. **The defect is the fixed name.** When Flux does update a
+fixed-name ConfigMap it updates it *in place*: the DaemonSet's pod template is byte-identical, so
+no rollout is triggered — and because the init container assembles `wg0.conf` **once at pod start**
+into a tmpfs, the running tunnel keeps the old peer list indefinitely. Adding or revoking a peer
+would have had **no effect at all** until something unrelated happened to restart the pods. For a
+*revocation* that is a security defect, not an inconvenience: the operator would delete a stanza,
+see a green PR, and the device would keep working.
+
+My runbook and README both asserted "PR → merge → Flux rolls the DaemonSet (~1 min)". That was
+never true.
+
+**This repo already documents the fix, and I had read the file.** `runbooks/kustomization.yaml`:
+*"The generated name carries kustomize's content hash, so editing any runbook rolls the server
+automatically and a stale projection is impossible."*
+
+So the register becomes a **content-hashed generator**:
+
+- `platform/vpn-ingress/peers.yaml` (inline ConfigMap) → **`platform/vpn-ingress/peers.conf`**, a
+  plain file, plus a `configMapGenerator` in `kustomization.yaml`. The operator's enrolled peer was
+  carried across verbatim. Every line is WireGuard config and `#` is a legal comment there, so the
+  documentation now travels *inside the mounted file* instead of in a YAML wrapper that gets
+  stripped.
+- **`namespace: vpn-ingress` added to the kustomization — REQUIRED, not tidiness.** A generated
+  ConfigMap inherits no namespace; every other object here declares `vpn-ingress`; and kustomize's
+  `nameReference` transformer will not rewrite a reference across a namespace boundary. Without it
+  the build emitted `ConfigMap wg-peers-m9kcckcb5t` while the DaemonSet still mounted the unhashed
+  `wg-peers`, which would not exist — the pods would wedge on a missing volume. I caught this only
+  by building the output and reading the volumes block.
+- **`infra/vpn-ingress/new-peer.sh`**: `PEERS_FILE` repointed to `peers.conf`, and a missing
+  register is now **fatal** instead of skipped. The old guard was
+  `[ -r "$PEERS_FILE" ] && grep …`, so after the rename the only symptom would have been
+  duplicate-address detection quietly never firing again — the silent-skip shape this repo treats
+  as a defect in its own right.
+- Stale `peers.yaml` references swept from the README, runbook, `daemonset.yaml`,
+  `networkpolicy.yaml`, the SOPS example, and the script's own **user-facing output** (it told the
+  operator to paste into "peers.yaml, under data.peers.conf", which would no longer exist).
+- The runbook gained the verification that actually distinguishes this failure — the DaemonSet's
+  `generation` must *increase* after the merge — and a troubleshooting row for `0 peer(s)` after a
+  merge.
+
+Flux's `prune: true` removes the orphaned fixed-name `wg-peers` ConfigMap on the next reconcile.
+
+No sensitive paths (`platform/vpn-ingress/**`, `infra/vpn-ingress/**`, `runbooks/**`,
+`decisions/**`), so this merges on a green gate with no override.
+
+## What I verified
+
+Built and run in the pinned toolbox, 2026-10-06 (kustomize v5.8.1 via `kubectl kustomize`).
+
+- **The reference is now rewritten end to end**: generated `ConfigMap wg-peers-m9kcckcb5t` in
+  namespace `vpn-ingress`, and the DaemonSet mounts **`wg-peers-m9kcckcb5t`**. Before adding
+  `namespace:` the same build mounted the unhashed `wg-peers` — so the namespace line is load
+  bearing, measured both ways.
+- **The hash tracks content, which is the whole mechanism.** `wg-peers-m9kcckcb5t` →
+  append a second peer → `wg-peers-dgg7ht9c74` → restore → `wg-peers-m9kcckcb5t`. Deterministic
+  and reversible, so an edit necessarily changes the pod template.
+- **The operator's enrolled peer survived the file conversion byte-for-byte**: the active stanza in
+  `peers.conf` (`PublicKey = pBRSHrRy…`, `AllowedIPs = 10.90.0.2/32`) is identical to the one on
+  `main`, compared directly against `git show origin/main:platform/vpn-ingress/peers.yaml`.
+- **The build still emits all five objects**: Namespace, ConfigMap, Secret, DaemonSet,
+  NetworkPolicy — and the `Namespace` object's own name is still `vpn-ingress` (the namespace
+  transformer did not rewrite it).
+- **`new-peer.sh` works against the new register**: `spare-laptop 7` renders a profile and prints
+  "append this to platform/vpn-ingress/peers.conf"; `dupe 2` is **refused** because the operator's
+  real `.2` now occupies it — the first time the duplicate check has been exercised against real
+  data rather than against its own example.
+- **Toolbox gate**: rc 3, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`,
+  `l1` and `approval-policy` all pass; rc 3 is the two environmental skips. **Not green. Reported
+  as rc 3.**
+
+## What I did NOT verify
+
+- **That the roll actually happens on the cluster.** The whole point of the change is unproven
+  until Flux applies it: the success criterion is the DaemonSet's `generation` increasing and
+  `wg0.conf assembled: 1 peer(s)` in the init log, followed by `wg show` listing the peer.
+- **That the tunnel then carries traffic.** No handshake has ever been observed. The WAN
+  port-forward is reportedly configured but has never been exercised by a packet I can see, and I
+  have no off-LAN vantage point — runbook step 6 from cellular remains the only honest test.
+- **Revocation**, which is the case where this defect was a security problem. Deleting a stanza
+  should now roll the pods and drop the peer; untested.
+- **Whether the orphaned fixed-name `wg-peers` ConfigMap is actually pruned.** Reasoned from
+  `prune: true` on the Kustomization, not observed.

decisions/2026-10-06-vpn-ingress-ip-forward.md

@@ -0,0 +1,113 @@
+---
+date: 2026-10-06
+branch: fix/vpn-ingress-ip-forward
+title: The VPN tunnel never came up — a hostNetwork pod cannot write net.ipv4.ip_forward, and wg-quick treats a failed PostUp as fatal
+---
+
+## What changed
+
+The remote-access VPN reconciled for the first time with a real server key (#60 then #62) and
+**the tunnel did not come up**. One line in my own manifest was the whole cause.
+
+Measured in the `wireguard` container on prod, 2026-10-06:
+
+```
+[#] ip link add dev wg0 type wireguard
+[#] wg addconf wg0 /dev/fd/63
+[#] ip -4 address add 10.90.0.1/24 dev wg0
+[#] ip link set mtu 1420 up dev wg0
+[#] sysctl -q -w net.ipv4.ip_forward=1
+sysctl: permission denied on key "net.ipv4.ip_forward"
+[#] ip link delete dev wg0
+**** Tunnel /config/wg_confs/wg0.conf failed, will stop all others! ****
+```
+
+Everything up to that point worked. **The cause is not a missing capability.** With
+`hostNetwork: true` the runtime mounts `/proc/sys` **read-only** in the container — confirmed in the
+pod's own `/proc/mounts`: `proc /proc/sys proc ro,nosuid,nodev,noexec,relatime` — precisely because
+those sysctls are the *host's* globals and a container must not rewrite them. `CAP_NET_ADMIN` cannot
+lift a read-only mount, so this can never succeed from here. And `wg-quick` treats any failed
+`PostUp` as fatal, so it deleted the interface it had just finished building.
+
+**The line was never needed.** `node_baseline` writes `net.ipv4.ip_forward=1` to
+`/etc/sysctl.d/99-k3s-cilium.conf`, and reading the value from inside the pod returns `1`. I had
+written the header comment *correctly* — "ip_forward is ALREADY 1 on these nodes … so no pod sysctl
+is needed" — and then emitted the sysctl anyway, justified in its own comment as "re-asserted so
+this component does not silently depend on another role's sysctl file surviving". That defensive
+re-assertion is the entire outage. **The lesson is narrow and reusable: in a hostNetwork pod,
+verify host state, never write it.**
+
+Fix:
+
+- **`platform/vpn-ingress/daemonset.yaml`**: the `PostUp = sysctl …` line is gone, replaced by a
+  **read** in the init container — `grep -qx 1 /proc/sys/net/ipv4/ip_forward` — which fails closed
+  with a message naming the `node_baseline` role. Reading `/proc/sys` is permitted; only writing is
+  not. Fail-closed still matters: a tunnel that comes up but cannot forward would satisfy every
+  probe in this component and silently drop every client's traffic.
+- The header bullet that said "no pod sysctl is needed" now also says **do not attempt one**, with
+  the mechanism, so the next author does not re-add it for the same reason I did.
+- The `securityContext` comment claimed NET_ADMIN was for "create the wg0 link, set the host sysctl,
+  and write the netfilter rules". The middle clause was false; corrected.
+- The image's `chown: … Operation not permitted` startup run — plus its own "Permissions could not
+  be set … we will not provide support for it" — is now documented as **expected and harmless**, in
+  both the manifest and the runbook, so nobody chases it. Capabilities are dropped to `NET_ADMIN`
+  only, so its s6 init has no `CAP_CHOWN`; `/config` is a writable tmpfs, everything runs as root,
+  and `wg-quick` proceeds past it (as it did here — the failure was 30 lines later).
+- **`runbooks/vpn-remote-access.md`**: a "what a healthy start looks like" block with the real log
+  lines; `ip_forward is 0` added to the init-failure table; and the note that `0/1 Running` with
+  `Startup probe failed: Unable to access interface: No such device` is the signature of `wg-quick`
+  tearing the tunnel down — read the `[#]` line it died on.
+- **`platform/vpn-ingress/README.md`**: a table of what the first reconcile settled (below).
+
+No sensitive paths: `platform/vpn-ingress/**`, `runbooks/**` and `decisions/**` only. This merges on
+a green gate with no override.
+
+## What I verified
+
+Live prod, read-only, 2026-10-06. **Four questions this component's design rested on are now
+measured rather than assumed — three went the right way, one was the defect above.**
+
+- **The GitOps path is proven end to end.** `Kustomization vpn-ingress` `lastAppliedRevision:
+  main@sha1:cf7d5e4` equals the GitRepository artifact revision. The Secret carries
+  `kustomize.toolkit.fluxcd.io/name: vpn-ingress`, so Flux delivered it.
+- **SOPS decryption works, proven by digest rather than by eye.** The value in
+  `platform/vpn-ingress/wg-server-key.sops.yaml` on `main`, decrypted, is 44 bytes, is **not** the
+  placeholder, and **`wg pubkey` accepts it**. Its SHA-256 equals the SHA-256 of the live Secret's
+  decoded `data.privatekey` (`d7c8f8de6f252649…` both sides). No key material was printed at any
+  point — lengths, booleans and digests only.
+- **The `wg pubkey` guard fired correctly, and its `ENC[` branch was never needed.** During the
+  kubelet window before the updated Secret propagated, all three pods reported *"It is still the
+  committed REPLACE-ME PLACEHOLDER"* — the guard distinguishing a stale/unfilled value from
+  ciphertext, exactly as designed. It cleared on its own once the mount refreshed; **a transient
+  `REPLACE-ME` CrashLoop for a minute or two after the key PR merges is expected, not a fault.**
+- **The kernel module reaches the pod**: `ip link add dev wg0 type wireguard` succeeded and
+  `/proc/modules` lists `wireguard`. So the `node_baseline` addition works and `CAP_SYS_MODULE` is
+  correctly not required.
+- **`allowPrivilegeEscalation: false` does NOT break the image's s6 init** — the container reaches
+  `wg-quick` normally. The flag stays; the fear recorded in #55 is closed.
+- **The iptables backends MATCH, which was my single biggest documented doubt.** The image ships
+  `iptables v1.8.13 (nf_tables)` via `/usr/sbin/xtables-nft-multi`, and Debian 13 is nf_tables. The
+  nightmare case — rules accepted into a table the kernel never consults, so handshakes succeed and
+  every forwarded connection hangs — cannot occur on this pairing.
+- **`net.ipv4.ip_forward` is `1`** in the host netns the pod shares, and `/proc/sys` is `ro` there.
+  Both read directly from the pod.
+- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`,
+  `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; rc 3 is the two
+  environmental skips (`gitea-settings`, conformance L2/L3). **Not green. Reported as rc 3.**
+
+## What I did NOT verify
+
+- **That the tunnel now comes up.** This fix has not reconciled. The next reconcile is the test, and
+  its success criterion is specific: pods `1/1 Running`, and `wg show` reporting `interface: wg0`
+  with `listening port: 51820`.
+- **That the MASQUERADE and FORWARD rules actually apply.** They are the next statements after the
+  one that failed, so **they have never executed**. The backends matching (above) removes the worst
+  failure mode but proves nothing about the rules themselves. First check after the pods go Ready:
+  `iptables -t nat -S POSTROUTING | grep 10.90.0.0/24` on a node.
+- **Anything about client traffic.** Still zero peers enrolled, still no WAN port-forward, still no
+  off-LAN vantage point. The VPN has carried no packet.
+- **Whether the transient `REPLACE-ME` window has an upper bound.** It cleared within ~2 minutes
+  here; I did not determine the kubelet refresh behaviour, so a longer stall after a key change is
+  uncharacterised.
+- **The `sops set` hand-off command**, still. The operator filled the key in by their own route
+  (#62), so the exact quoting I documented inside `dev.sh run bash -c` remains unexercised.

decisions/2026-10-07-bulk-nfs-truenas.md

@@ -0,0 +1,85 @@
+---
+date: 2026-10-07
+branch: fix/bulk-nfs-truenas
+title: Repoint bulk NFS at the TrueNAS (192.168.2.30); the PVs must be recreated, not reconciled
+---
+
+## What changed
+
+Both ZFS pools were physically moved off the Proxmox host onto a standalone TrueNAS Community
+Edition at **192.168.2.30**, which sits on prod's own `192.168.2.0/24`. `BULK_NFS_SERVER` moves from
+`192.168.2.102` (the pfSense DNAT prod used to mount through) to `192.168.2.30`.
+
+- `clusters/prod/kustomization.yaml` — `BULK_NFS_SERVER` → `192.168.2.30`. The comment block that
+  justified the DNAT, and the "NOT YET DECIDED / THIS PLACEHOLDER IS LOAD-BEARING" banner, are
+  replaced with the measured export configuration and the recreate requirement below.
+- `clusters/base/cluster-vars.yaml` — base default `192.168.1.9` → `192.168.2.30`. That host no
+  longer exists.
+- `clusters/dev/kustomization.yaml` — same; marked explicitly **inert**, since dev is decommissioned
+  and its nodes are gone. It carries the real address only so nothing in the tree renders a dead one.
+- `docs/BULK-NFS-STORAGE.md` — §1 and §5 rewritten for TrueNAS (exports, service settings, the ZFS
+  `sharenfs` properties that had to be cleared, and the ACL hazard); §7 rewritten now that the
+  placeholder is gone; §9 retargeted from dev to prod; **new §10** documents the PV
+  delete-and-recreate. §2/§3/§4/§6 are unchanged by the move.
+- `catalog/qbittorrent.yaml`, `platform/monitoring/node-pressure-dashboard.yaml` — two comment/prose
+  references to `192.168.2.102` as the live export address.
+
+**The headline operational fact: repointing this knob is NOT a plain reconcile.** A bound PV's NFS
+source is immutable, so Flux's apply fails rather than updating and the pods keep their old, dead
+mount. The five PVs must be deleted and recreated. That sequence is §10 and is an operator action;
+it is **not** performed by this PR.
+
+## What I verified
+
+**On the NAS (192.168.2.30), measured 2026-10-07 by the operator at my request:**
+
+- `zfs list` — `Storage` → `/mnt/Storage`, `Storage/plex` → `/mnt/Storage/plex`,
+  `Storage/nextcloud` → `/mnt/Storage/nextcloud`. The PV paths resolve unchanged.
+- Ownership intact: `/mnt/Storage/plex`, `tv/`, `downloads/`, `movies/`,
+  `downloads/complete/`, `nextcloud/nextcloud/` all `drwxr-xr-x 1000:3000`. `movies/` 1171 entries
+  and `tv/` 107, against 1165 / 107 measured on Proxmox.
+- `acltype=nfsv4` (SCALE's built-in default, `SOURCE: default`), `aclinherit=passthrough`, and no
+  non-trivial ACLs — mode bits still govern.
+- Three stale `sharenfs` dataset properties (`plex`, `nextcloud`, `docker-volumes`) carried over from
+  Proxmox; cleared with `zfs inherit`, re-checked empty.
+- `exportfs -v`: both shares to `192.168.2.10/.11/.12` with `sec=sys,secure,root_squash,no_all_squash`,
+  `rw` on plex and `ro` on nextcloud.
+- `/proc/fs/nfsd/versions` → `+3 +4 +4.1 +4.2`, so the PVs' `nfsvers=4.1` negotiates.
+- **A defect was found and fixed during setup:** the plex share listed `192.168.2.13` — an address no
+  node holds — and so omitted `192.168.2.12` (k3s-prod-3). Corrected and re-verified.
+
+**From the cluster:**
+
+- Node InternalIPs are `192.168.2.10/.11/.12` (`kubectl get nodes`), confirming `.13` was wrong.
+- All five NFS PVs currently name `192.168.2.102`.
+- `192.168.2.30:2049` and `:111` accept connections; `192.168.2.102:2049` does not.
+- **`spec.nfs` immutability, by server-side dry-run** — this is the finding that shapes the change:
+  `spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation`.
+- The pods report `Ready` while their mounts are dead: a `stat /media` answers from attribute cache,
+  but a `readdir` blocks indefinitely (the PVs mount `hard`). Hence "scale to zero first" in §10.
+
+**Gate:** see the PR; run in the pinned toolbox from this worktree.
+
+## What I did NOT verify
+
+- **Nothing has mounted 192.168.2.30.** No pod has touched the new server — the PVs still point at
+  the old address and cannot be updated in place. Every §9 probe (numeric-id passthrough, the uid-1000
+  write, the hardlink/EXDEV check, Plex's read-only posture) is **outstanding**, not passing. The
+  equivalents passed against the old host on 2026-09-12; that is not evidence about this one.
+- **The §10 sequence is reasoned, not executed.** Retain + the renderer's pre-set `claimRef` are read
+  from the manifests and from `catalog/render.py`; I have not watched a PVC rebind after a PV delete
+  in this cluster.
+- **`root_squash` is a behaviour change from the Proxmox export**, which carried `no_root_squash`. The
+  argument that nothing needs root on the tree (LSIO drops to 1000; the images chown only their
+  Longhorn `/config`; both subPath directories already exist at 0755) is reasoned from the images and
+  the measured permissions. It has not been exercised. If a mount or subPath resolution fails with
+  EACCES, Maproot `root:root` reproduces the old export and is the first thing to try.
+- **Whether NFSv3 can be disabled.** Enabled Protocols is `NFSv3, NFSv4`; the PVs only need v4.1, so
+  narrowing would close the rpcbind/mountd/statd surface. Not done — it is a change to make after the
+  cutover is proven, not during it.
+- **Whether pfSense / the legacy stack is actually retired.** I measured only that
+  `192.168.2.102:2049` stopped answering. The other legacy consumers of these datasets
+  (`192.168.1.6`, `10.123.1.10`, `192.168.1.18`) were dropped from the new exports; if any is still
+  live, it has lost access. Not checked.
+- **L2/L3 conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline subset
+  only. Nothing in the gate exercises an NFS mount.

decisions/2026-10-17-automated-upgrades.md

@@ -0,0 +1,300 @@
+---
+date: 2026-10-17
+branch: feat/automated-upgrades
+title: Automated upgrades — kured + security-only unattended-upgrades, and Renovate finally gets a runner
+---
+
+## What changed
+
+Two halves of one process. Neither is useful alone and they must land together.
+
+### (a) OS upgrades and reboots
+
+**`platform/kured/`** — the Kubernetes Reboot Daemon, chart `6.1.0` / app `1.23.0`, in its own Flux
+Kustomization (`clusters/base/kured.yaml`, `dependsOn: [platform]`, `wait: true`), registered in
+`clusters/base/kustomization.yaml`. Not a line in `platform/kustomization.yaml`: that tier is
+`wait: true` and gates `platform-config` → every apps wave, so a wedged reboot daemon placed there
+could stall every application in the cluster. A component added to make maintenance safer must not
+be able to cause an outage.
+
+**`infra/ansible/roles/unattended_upgrades/`** + `playbooks/unattended-upgrades.yml`, enabled for
+prod in `inventory/group_vars/k3s_prod/vars.yml`. A **new** role rather than an extension of
+`node_baseline` — different lifecycle, and other open branches touch that role.
+
+**The design, and how it answers this repo's two standing objections to reboot automation.**
+`roles/node_baseline/tasks/main.yml:61-67` says "IT DOES NOT REBOOT" (k3s-prod-2 hung on a warm
+reboot 2026-09-03; physical power button required), and `docs/plans/cluster-upgrade-plan.md:291-299`
+rejected system-upgrade-controller for "automating away the human gate between nodes". Both still
+stand. **The human gate is the sentinel.** unattended-upgrades is security-only with the kernel
+blacklisted, so it essentially never produces `/var/run/reboot-required`; that file appears when an
+operator deliberately runs `node-baseline.yml --tags kernel`. kured then performs only the
+*mechanical* half the operator would otherwise type by hand — including the ordering rule
+`node_baseline`'s own output has to shout ("NEVER node 2 first"), which the operator keeps by
+running the kernel play with `--limit <host>` one node at a time. `cluster-upgrade-plan.md:329-330`
+names the condition under which its own rejection reverses (fast disks), and prod is that cluster.
+
+**Quorum: concurrent reboots are structurally impossible.** All three nodes are control-plane+etcd
+(quorum 2/3). `concurrency: 1` plus kured's single cluster-wide lock annotation on its own DaemonSet
+means a node reboots only while holding that lock, enforced by API-server optimistic concurrency
+rather than by timing. **`lockTtl: 0`** (expiry disabled) is the deliberate choice: a TTL exists so
+reboots continue after the lock holder dies, which on 3 etcd nodes is exactly the catastrophe — a
+hung node's lock expiring would let a second node take the cluster to 1/3. So a hang stops the cycle
+forever and pages a human. `lockReleaseDelay: 30m` holds the lock after a node returns.
+
+**The Longhorn gate.** Measured: 25 volumes, all `numberOfReplicas: 3`, one replica per node, with
+`replica-soft-anti-affinity: false` and `disable-scheduling-on-cordoned-node: true`. So every drain
+takes **all 25 volumes to degraded 2/3**, and recovery is serialized —
+`replica-replenishment-wait-interval: 600`, `concurrent-replica-rebuild-per-node-limit: 1`. Reboot
+the next node too early and volumes reach 1/3, where `node-drain-policy:
+block-if-contains-last-replica` can block a drain forever. The inter-node gate is therefore **volume
+robustness, not pod readiness**, and it is doubled: `lockReleaseDelay: 30m` (unconditional, covers
+the 600s wait) plus kured's Prometheus alert gate (semantic, covers a rebuild longer than 30m).
+
+**`alertFilterMatchOnly: true` is load-bearing.** The flag inverts the regexp's meaning: `false`
+(chart default) makes it an *ignore* list and kured blocks on **any** firing alert. Measured
+2026-10-17, five alerts were firing with no end in sight (`CNPGPrimariesCoLocated`,
+`KubeDeploymentReplicasMismatch`, `KubeDeploymentRolloutStuck`, `KubePodNotReady`,
+`OAuth2ProxyGateUnobserved` — the last three all tracing to `pm-agent`, ImagePullBackOff on a
+placeholder digest for 12 days). Under the default, kured would have been blocked on day one and
+never rebooted anything, indistinguishable from "no reboots needed". The allow-list is anchored:
+`^(LonghornVolumeDegraded|LonghornVolumeFaulted|KubeNodeNotReady|NodeExporterDown)$`.
+`NodeEtcdRequestsSlow` and the `NodeIOPressure*` family are **deliberately excluded** despite being
+exactly what you'd want to pause for — prod-1 has a documented etcd WAL I/O starvation history and
+including them risks a permanent wedge that looks like safety.
+
+**The kernel floor (the highest-risk item), handled with two independent exclusions.** Measured: all
+three nodes run `7.1.8+deb13-amd64` (`7.1.8-1~bpo13+1`, the backports metapackage), far above the
+6.13 memattr floor — **but `linux-image-6.12.107` and `linux-image-6.12.94` are also still
+installed, below the floor, bootable, and `apt-mark showhold` is empty on every node.** So: (1) the
+allowed-origins list is `label=Debian-Security` only, and the running kernel comes from
+`o=Debian Backports` (pin 100, the `NotAutomatic`+`ButAutomaticUpgrades` signature), so u-u cannot
+*see* a kernel upgrade; (2) `^linux-` is blacklisted anyway, so widening origins later — the most
+likely future edit — cannot re-arm the hazard. The role **asserts both against `apt-config dump`**,
+apt's own parse of the whole directory, rather than trusting the file it just wrote. Also blacklisted:
+`intel-microcode`, `amd64-microcode`, `grub-*`. `Remove-Unused-Kernel-Packages` is **false** against
+the Debian default, because those older kernels are the documented rollback path. None of this stops
+the kernel moving — the operator's deliberate Ansible run still does that, unchanged.
+
+**The silent-no-op trap this closes.** Measured: `update-notifier-common`, `needrestart` and
+`unattended-upgrades` were **absent on all three nodes**, so *nothing could create
+`/var/run/reboot-required`*. kured deployed alone would have been permanently idle while 1/1 Running
+and reporting `kured_reboot_required=0` — green forever. The role installs the sentinel writer and
+**asserts a hook exists** by discovery (not a hard-coded path), failing loudly if not.
+`needrestart` runs in auto mode with `k3s.service` blacklisted — auto-restart is what makes a
+security-only policy effective, but restarting the control plane unattended is worse than a delayed
+library fix. (k3s is not an apt package at all here: no `k3s`, no `containerd` in dpkg.)
+
+**Alerts + runbook.** `platform/kured/alerts.yaml` — three rules over kured's *only* metric
+(`kured_reboot_required`, one GaugeVec, read from `cmd/kured/main.go:92-96`) plus `kube_node_info`:
+`KuredNotReporting` (`absent()` — the pipeline is dead and the other two are inert),
+`KuredNodeCoverageGap` (`count(kured) < count(nodes)` — a node silently uncovered, which `absent()`
+cannot see), `KuredRebootPending` (`for: 36h` — a reboot was offered and blocked). Companion
+promtool suite `alerts.promtool-test.yaml` pins all three both ways, including that the rule
+**stops** when a reboot succeeds. `runbooks/node-reboot-automation.md` registered in
+`runbooks/kustomization.yaml`.
+
+**The window is DAILY 03:00-06:00, and that choice is driven by the alert.** A weekly window means a
+pending reboot legitimately waits 6 days, forcing `for: 8d`+ — longer than this cluster's 7d
+Prometheus retention and reset by any restart past the 1h for-outage-tolerance. Daily caps the
+legitimate wait under 24h and makes `for: 36h` both robust and meaningful.
+
+### (b) Image / dependency automation
+
+**Renovate was aspirational. It now has a runner.** `.gitea/workflows/renovate.yaml` — scheduled
+daily 03:15 UTC, `workflow_dispatch` with `dry_run` defaulting to **true** for a safe first run.
+Preflight **fails hard (exit 1)** if `RENOVATE_TOKEN` is missing rather than skipping, because a
+skipped Gitea job reports `success` and that is precisely how `renovate.json` stayed inert.
+
+**`renovate.json`: Renovate now owns both halves, and Flux image-automation is formally closed.**
+ADR-0009 gave container image tags to Flux image-automation; that cannot be delivered on this
+repo's terms. (1) `ImageUpdateAutomation` **pushes commits, it does not open PRs**, so it bypasses
+the CLAUDE.md §1 gate; pointed at a side branch instead, nothing opens the PR. (2) It cannot work
+here anyway: `main` is protected `user_can_push: false` and the `flux-system` GitRepository has **no
+`spec.secretRef`** — the source is anonymous and read-only. (3) Enabling it means an in-cluster git
+**write** credential for the repo that defines the cluster. So `docker`/`kubernetes` — disabled
+*only* to avoid fighting that controller — are enabled, with `kubernetes.managerFilePatterns`
+covering `platform/`, `platform-config/`, `apps/`, **`catalog/`** (apps are generated from it; a bump
+only to the rendered output is reverted by the next render and fails `ci/render-golden.sh`),
+`runbooks/serving/` and `infra/agent/`. Added a no-automerge `blast-radius` rule for `kured`,
+`longhorn`, `cilium`, `cloudnative-pg`, `kube-prometheus-stack`, because the gate is static and does
+not boot a node or form a quorum. `platform/kyverno/tests/**` added to `ignorePaths`.
+
+### Sensitive paths — this PR needs a non-author approving review
+
+It touches **`clusters/**`**, **`conformance/**`** and **`.gitea/**`**, all in
+`infra/gitea/approval-policy.json`. It will **not** merge on a green gate alone. Per CLAUDE.md §1 I
+have **not** posted `/approve-override` and must not — that is the self-approval the stage exists to
+prevent.
+
+**The `conformance/` edit, justified.** `+34/-0` in
+`alert-metrics-have-scrape-targets/check.sh`: one new `SCRAPE_OWNERS` entry claiming `kured_`. It
+**weakens nothing** — no existing owner's evaluation changes — and it *tightens* the check for my
+own component, adding two `require` patterns (`service.create: true`, `metrics.create: true`; the
+chart defaults **both** to false) plus a `companion` requirement that `networkpolicy.yaml` stay in
+the kustomization. It had to be touched: assertion (3) fails any alert over an unclaimed metric, and
+the escape hatch `KNOWN_UNWIRED_METRICS` is **shrink-only**, so declaring `kured_*` as debt was not
+permitted and would have been wrong anyway. The alternative was shipping reboot automation with no
+alerting, which for a component whose every failure mode is silent is the worse choice. The entry
+carries a dated comment saying all of this.
+
+## What I verified
+
+Measured against the live cluster and nodes, read-only, on 2026-10-17.
+
+- **Flux image-automation controllers ARE installed and idle.** `-n flux-system get deploy` shows
+  `image-automation-controller` and `image-reflector-controller` both `1/1`, age **43d**; all three
+  `image.toolkit.fluxcd.io` CRDs exist. And **zero** `ImageRepository`/`ImagePolicy`/
+  `ImageUpdateAutomation` objects — "No resources found" for each. The controllers have run for 43
+  days with nothing to do. `platform/kustomization.yaml:52` has the component commented out.
+- **Renovate is aspirational, confirmed three ways**: no workflow in `.gitea/` or `ci/` (grep),
+  no Renovate CronJob/Deployment/Job in any namespace, and `renovate.json` itself disabled the
+  `docker`+`kubernetes` managers — so every plain `image:` tag was maintained by **nothing**, while
+  **seven files** claim "Renovate maintains this pin".
+- **`flux-system` GitRepository has no `secretRef`** (empty), URL is the in-cluster Gitea, and its
+  artifact revision was `main@sha1:1a0b208…`, matching my base.
+- **Nodes**: 3× Debian 13 trixie, all `control-plane,etcd`, k3s `v1.36.4+k3s1`, kernel
+  `7.1.8+deb13-amd64`. `apt-mark showhold` **empty**; `/etc/apt/preferences.d/` **empty**;
+  `/var/run/reboot-required` absent; `unattended-upgrades`/`update-notifier-common`/`needrestart`
+  all **absent**; `linux-image-6.12.107` and `6.12.94` installed alongside the backports kernel;
+  no `k3s`/`containerd` apt package; `apt-cache policy` origins and pins as quoted above.
+- **No node taints** on any of the three (so no tolerations needed), and `system-node-critical`
+  exists.
+- **Longhorn**: 1.12.1; `node-drain-policy: block-if-contains-last-replica`;
+  `replica-soft-anti-affinity: false`; `disable-scheduling-on-cordoned-node: true`;
+  `replica-replenishment-wait-interval: 600`; `concurrent-replica-rebuild-per-node-limit: 1`;
+  25 volumes all 3-replica/healthy; 25 replicas per node.
+- **6 PDBs with 0 allowed disruptions**: 3× `longhorn-system/instance-manager-*` (structurally 0 —
+  single-node selector, `minAvailable: 1`) and 3× CNPG `*-primary`. All three Postgres primaries
+  were on `k3s-prod-1`.
+- **kured drain flags do not exist.** Read `cmd/kured/main.go:438-449`: `Force`,
+  `DeleteEmptyDirData` and `IgnoreAllDaemonSets` are **hardcoded true**. Passing
+  `--ignore-daemonsets`/`--delete-emptydir-data`/`--force` as extraArgs would CrashLoop the
+  DaemonSet. All three are genuinely needed (12 DaemonSets, 135 emptyDir pods measured).
+- **kured exposes exactly one metric** — the only `MustRegister` in the project.
+- **Prometheus selectors are all `{}`** (`serviceMonitorSelector`, `serviceMonitorNamespaceSelector`,
+  `ruleSelector`, `ruleNamespaceSelector`), so a `PrometheusRule`/`ServiceMonitor` in a new namespace
+  is picked up with no label plumbing.
+- **`count(kube_node_info) == 3`** and `count(longhorn_volume_robustness) == 100`, queried through a
+  read-only port-forward. The 5 chronically-firing alerts were read from `/api/v1/alerts`.
+- **`LonghornVolumeDegraded`** is `longhorn_volume_robustness{state="degraded"} == 1`, `for: 10m` —
+  shorter than Longhorn's own 600s replenishment wait, so degradation reliably outlasts the latch.
+- **kured chart 6.1.0 → appVersion 1.23.0** is the newest of 80 entries in the chart index.
+- **Kyverno/PSA will not block it**: every policy is scoped
+  `app-namespace: "true"`; the namespace sets it explicitly `"false"`, and
+  `restrict-host-namespaces.yaml`'s own header states that carve-out. No cluster-wide PSA baseline.
+- **`renovate.json` parses** as valid JSON after my edits (`json.load`).
+- **kured is not installed** and the string `kured` appears nowhere in the repo — greenfield.
+
+### CORRECTION 2026-10-18 — the first real run FAILED, and the sentinel design was wrong
+
+`ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod` failed on all three
+nodes at the package task:
+
+```
+fatal: [k3s-prod-1]: FAILED! => "No package matching 'update-notifier-common' is available"
+```
+
+**`update-notifier-common` is an UBUNTU package and does not exist in Debian.** Measured on trixie:
+`apt-cache policy update-notifier-common` → `Candidate: (none)`. The original entry asserted this
+package "owns the hook that creates /var/run/reboot-required" — true on Ubuntu, false here, and the
+role could never have worked. This was the exact risk the first version of this entry flagged ("the
+`find` patterns are a guess at which package ships what") landing in a worse place than predicted:
+not a wrong glob, a nonexistent package.
+
+Three further measurements settled what to do instead:
+
+* `/usr/share/update-notifier/notify-reboot-required` is **absent**.
+* **No `linux-image-*` postinst references `notify-reboot-required`** — invoking it is an Ubuntu
+  patch, so merely placing a script at that path would never be called.
+* No hook under `/etc/apt/apt.conf.d` or `/etc/kernel/` mentions `reboot-required`.
+
+So Debian trixie has **no stock mechanism at all**, and the role now ships the writer itself:
+`/usr/local/sbin/k3s-notify-reboot-required` (compares the running kernel against the newest
+installed via `linux-version`, from `linux-base`, which every `linux-image` depends on — measured
+`linux-base 4.12.1` present) plus a `DPkg::Post-Invoke` hook at
+`/etc/apt/apt.conf.d/99-k3s-reboot-required`. `/run` is a tmpfs (measured), so the sentinel
+self-clears on boot and needs no cleanup task.
+
+The verify assert changed shape too: it was a `find` glob over `/etc/kernel/postinst.d` and
+`/etc/apt/apt.conf.d` for `*update-notifier*` / `*unattended-upgrades*`. That glob was **weaker
+than it looked** — it would have matched Debian's own `unattended-upgrades` apt conffile and
+reported "sentinel writer present" while no writer existed. It now stats the two exact paths this
+role owns and requires the script to be executable.
+
+**Newly verified (this correction):** the sentinel script was executed on prod-1 as the
+unprivileged `k3s` user under `sh -x`. It resolved `running=7.1.8+deb13-amd64`,
+`newest=7.1.8+deb13-amd64`, took the equal branch, exited 0 and wrote nothing — correct, since the
+running kernel *is* the newest. `sh -n` and `bash -n` clean.
+
+**Still NOT verified after this correction:** the *writing* half. Because running == newest on all
+three nodes, the `touch` path has never executed, so I have not seen the sentinel actually created,
+nor seen kured react to it. The honest test is the operator's next kernel upgrade. Also unverified:
+whether a `DPkg::Post-Invoke` hook fires in the `unattended-upgrades` code path specifically (it
+runs dpkg, so it should, but that is inference) — and whether `linux-version sort --reverse` orders
+a mixed backports/stable kernel set the way I assume when a 6.12 and a 7.1 are both installed. It
+produced the right answer with the current set; a future set could differ.
+
+## What I did NOT verify
+
+The honest list. Most of this needs an actual upgrade cycle, which I did not and could not run.
+
+- **I never ran an OS upgrade, never rebooted a node, and never watched kured take a lock.** No part
+  of the runtime behaviour of either half has been observed. `sudo` needs a password I do not have,
+  so I could not run `apt`, `unattended-upgrade --dry-run`, or install anything. **Everything about
+  (a) working is inference from configuration and measured cluster state, not observation.**
+- **The Ansible role has never been executed.** Not syntax-checked by `ansible-playbook --syntax-check`
+  either (no inventory access from the gate). Its asserts, its `apt-config dump` parsing, and the
+  sentinel-hook discovery are all unproven. The `find` patterns for the sentinel hook are a *guess at
+  which package ships what* — I could not read `update-notifier-common`'s file list (packages.debian.org
+  returned no such package for that suite/arch), which is exactly why the role discovers rather than
+  hard-codes. **If no hook matches, the role fails loudly — but it may fail on a correct node.**
+- **Whether CNPG actually switches the primary in response to a drain-induced eviction.** All three
+  `*-primary` PDBs are structurally 0-disruption and `.spec.nodeMaintenanceWindow` is unset on all
+  three clusters. If the operator does not switch over, draining `k3s-prod-1` blocks on three PDBs at
+  once and hits `drainTimeout`. **This is the most likely way the first real reboot fails.** It
+  should be tested with one manual, watched `kubectl drain` before trusting the timer.
+- **Whether `drainTimeout: 15m` is long enough.** Unmeasured. The `unifi` VMI has
+  `evictionStrategy: LiveMigrate` with `migrationMethod: BlockMigration`, i.e. a full root-disk copy
+  over the network on every `k3s-prod-1` drain. I do not know how long that takes.
+- **Whether the alert gate actually blocks as intended.** I never saw kured query Prometheus. The
+  assumption that it treats a query error as "do not proceed" is from its documented behaviour, not
+  observed. The NetworkPolicy allowing kured→Prometheus:9090 and Prometheus→kured:8080 is
+  **unproven**; if either is wrong the symptom is silence, which is why `KuredNotReporting` exists —
+  but that alert is itself unproven against a live scrape.
+- **The alert expressions have not run against real `kured_*` data**, because the metric does not
+  exist in this cluster yet. They pass promtool unit tests against synthetic series whose shape I
+  read from kured's source. `absent()` and the `count() < count()` comparison are the two most
+  likely to behave differently against real label sets.
+- **`enabledManagers: ["regex"]` may be the wrong name.** Modern Renovate expects `custom.regex` for
+  `customManagers`. I left the pre-existing value alone rather than guess, but **the inline
+  `# renovate:` markers may simply not run.** Unverifiable without executing Renovate. Two markers
+  are additionally known-broken independent of this: `external-snapshotter`'s sits two comment lines
+  above its `newTag`, and Keycloak's `kustomization.yaml` has no marker at all despite a comment
+  claiming Renovate maintains it.
+- **Renovate has still never run.** I added the runner; `RENOVATE_TOKEN` does not exist yet, so the
+  first scheduled run **will fail** (by design). Nothing about the config is proven correct until a
+  `dry_run: true` dispatch is read by a human. Expect it to want to open ~30 PRs.
+- **No alert for "automatic patching silently stopped".** u-u's success lives in
+  `/var/log/unattended-upgrades/` on each node and nothing reads it. Likewise **no alert for the
+  Renovate schedule ceasing to fire**. Both are real gaps I did not close.
+- **`coredns` is 1 replica and I did not fix it.** It is k3s-managed, out of this PR's scope, and it
+  means every `k3s-prod-1` reboot causes a cluster-wide in-cluster DNS gap. Recommended follow-up.
+- **kured cannot honour "NEVER node 2 first".** It has no ordering control. The mitigation — run the
+  kernel play with `--limit` one host at a time — is a *procedure*, documented in the runbook, not an
+  enforced property. If the operator flags all three nodes at once, kured picks the order.
+- **kured cannot press a power button.** The 2026-09-03 warm-reboot hang is unmitigated. Bounded, not
+  prevented: quorum holds at 2/3, the lock is held forever so nothing cascades, and
+  `KuredRebootPending` eventually fires.
+- **Upstream version claims are a snapshot, partly from search results**, not all from authoritative
+  release APIs. I could not determine latest for versitygw, crafty, nginx-unprivileged, act_runner,
+  `virt-template-*`, the linuxserver wireguard date-tag, or the exact latest PostgreSQL 16 patch
+  (sources conflicted 16.10 vs 16.15). **The gitea `1.24.3 → v28.1.0` gap is almost certainly an
+  upstream versioning-scheme change, not 26 majors — do not act on that framing.**
+- **I did not verify Renovate's `kustomize` manager still processes `images:`/`newTag`** now that
+  `docker` is enabled, which decides whether `external-snapshotter` and `intel-gpu-plugin` are
+  managed at all.
+- **I did not raise any HelmRelease version ceiling**, so the 9 minor-locking ranges still cap
+  Renovate at patches. Bumping `kube-prometheus-stack` in particular will fail the gate until
+  `ci/versions.env`'s exact promtool/Prometheus `3.14.0` lockstep moves with it.
+- **L2/L3 conformance did not run** (`vars.CLUSTER_AVAILABLE` unset), so nothing here is proven
+  against a cluster by CI either.

decisions/2026-10-17-graceful-cluster-shutdown.md

@@ -0,0 +1,106 @@
+---
+date: 2026-10-17
+branch: feat/graceful-cluster-shutdown
+title: "Graceful node shutdown was never configured, and scaling a StatefulSet deleted its PVC"
+---
+
+## What changed
+
+All three nodes had to be powered off to move them onto a UPS. There was **no procedure for it**:
+`infra/cluster.sh stop` is `virsh`-based and only ever applied to the decommissioned dev cluster, so
+the whole sequence was improvised live. Two defects came out of doing it.
+
+**1. Kubelet graceful node shutdown was unconfigured on every prod node.** No `shutdownGracePeriod`
+anywhere in `infra/ansible/`, so `shutdown -h` stopped k3s, containerd went with it, and every pod
+was SIGKILLed where it stood — 9 CloudNativePG instances mid-checkpoint, the UniFi VM, and 25
+attached Longhorn volumes. `roles/k3s_server` now writes `kubelet-arg` into
+`/etc/rancher/k3s/config.yaml` with `shutdown-grace-period=90s` and
+`shutdown-grace-period-critical-pods=20s`.
+
+The half that is easy to miss and silently defeats the other half: **graceful node shutdown is
+enforced by logind, not by the kubelet.** The kubelet holds an inhibitor *delay* lock and systemd
+waits at most `InhibitDelayMaxSec` — **default 5 seconds** — before proceeding regardless. A grace
+period above that cap is truncated with no error, so the feature looks configured and still
+hard-kills pods. The role now ships a `logind.conf.d` drop-in setting it to 120, and **asserts**
+`k3s_logind_inhibit_delay_max_sec >= k3s_kubelet_shutdown_grace_period` rather than documenting the
+relationship, because the failure is invisible.
+
+**2. Scaling a StatefulSet to 0 DELETED its PVC and Longhorn volume.** `monitoring/loki` carries
+`persistentVolumeClaimRetentionPolicy: {whenDeleted: Delete, whenScaled: Delete}`, so the quiesce's
+`scale --replicas=0` destroyed `storage-loki-0` outright. Loki stores with `type: filesystem`, so
+that PVC was the only copy of its log data. This is the defect in this change set that actually
+destroyed something, and it was mine.
+
+New `infra/maintenance/quiesce.sh` (`quiesce` | `resume` | `status`) encodes the sequence that
+worked, and **refuses to scale any StatefulSet whose `whenScaled` is `Delete`** — it leaves it
+running, accepts that its volume hard-detaches at poweroff, and prints which volumes to expect
+`degraded`. A rebuild is a vastly better outcome than a deletion. Workload discovery is computed
+from the live cluster (every Deployment/StatefulSet whose pods mount a PVC, Longhorn or TrueNAS NFS)
+rather than hardcoded, so a newly onboarded service is covered without anyone remembering to add it.
+
+`runbooks/cluster-maintenance-shutdown.md` (new, registered in `runbooks/kustomization.yaml`) is the
+operator page: shutdown order, the TrueNAS-after-nodes constraint, power-on order, the
+un-hibernate-Postgres step Flux will not do for you, and a "what looks broken but is fine" section
+separating the `4/19`-Kustomizations-Ready cascade, the degraded-and-rebuilding volume and the
+self-recovering `keycloak-operator` from the one thing that genuinely needs a human (a stuck
+`oauth2-proxy` replica).
+
+## What I verified
+
+* **Graceful shutdown was genuinely absent**, not merely undocumented: `grep -rn
+  'shutdownGracePeriod|graceful-node-shutdown'` over `infra/ansible/` returns nothing, and no
+  `logind` or `InhibitDelay` setting exists anywhere in the tree.
+* **The whole shutdown/restart cycle was executed end to end on prod 2026-10-17** and the cluster
+  came back: 3/3 nodes Ready with etcd `healthz` ok, 19/19 Flux Kustomizations Ready,
+  `platform-config`'s `lastAppliedRevision` equal to the GitRepository artifact revision, all 3
+  CNPG clusters `healthy` with primaries elected, the UniFi VM `Running`, PiHole resolving
+  (`home.keithjackson.dev` → `192.168.2.20` via `192.168.2.23`), and the Gateway serving
+  `200/200/401` for apex/gitea/plex.
+* **The PVC-deletion mechanism is confirmed, not inferred:** `kubectl get sts loki -o
+  jsonpath={.spec.persistentVolumeClaimRetentionPolicy}` returns
+  `{"whenDeleted":"Delete","whenScaled":"Delete"}`, the PVC was `NotFound` after the cycle, and the
+  Longhorn volume count had gone 25 → 24.
+* **The damage was repaired and verified:** `scale sts loki --replicas=1` (restoring the replica
+  count the chart declares) recreated `storage-loki-0` as a fresh 50Gi Bound PVC, `loki-0` reached
+  `2/2 Running`, and the volume count returned to 25.
+* **The guard works against the live cluster**, tested both ways: `sts_deletes_pvc_on_scale
+  monitoring loki` returns true (would be skipped) and `pihole pihole` returns false (safe to
+  scale).
+* **The script's read-only paths run clean on prod:** `status` reports correctly, and the discovery
+  function returns the 13 workloads currently holding a PVC.
+* `bash -n` clean; the two edited Ansible YAML files parse.
+
+## What I did NOT verify
+
+* **The kubelet change has NEVER BEEN APPLIED OR TESTED.** This is the significant gap. I did not
+  run the role, did not restart k3s, and have **not** observed a node actually drain on
+  `shutdown -h`. Everything about it is reasoning from kubelet/systemd semantics plus the measured
+  absence of the setting. Applying it needs a **rolling** k3s restart (`serial: 1` — kube-vip keeps
+  the API up, etcd holds at 2/3), which is an operator action (CLAUDE.md §5). **Verify by draining
+  one node for real:** apply, then `shutdown -h now` on a single node and watch whether pods
+  terminate gracefully instead of vanishing.
+* **Whether 90s/20s are the right numbers.** Chosen to comfortably cover a Postgres fast shutdown
+  and a Longhorn flush; not measured against either. The budget is spent on every reboot, so if a
+  node takes noticeably longer to go down, this is why.
+* **Whether k3s accepts `kubelet-arg` in `config.yaml` as written.** The file already carries
+  `kube-apiserver-arg` as a list and `kubelet-arg` is the documented sibling, but I did not render
+  or apply it, so a typo would surface only at k3s restart — and a bad `config.yaml` is exactly the
+  kind of thing that makes k3s fail to come up. **Review this one before applying.**
+* **The `assert` task's `regex_replace` on the grace period** assumes a plain `90s`-style value; a
+  value expressed as `1m30s` would parse to the nonsense integer `130`. Fine for the committed
+  defaults, wrong if someone writes a compound duration.
+* **`resume`'s Flux-reconcile wait has never run**, and neither has `quiesce` as a script — the
+  2026-10-17 cycle was done with the equivalent commands by hand, so the script encodes a sequence
+  that worked but has not itself been executed end to end. Only `status` and the discovery and guard
+  functions have run.
+* **Loki's lost log window is gone for good.** Up to 72h of logs (its `retention_period`), destroyed
+  by my scale-down. Nothing recovers it; Velero's schedules cover the namespace but the PVC was
+  deleted, not corrupted, and I did not attempt a restore because the data self-expires anyway.
+* **No other `whenScaled: Delete` StatefulSet was audited.** The guard now catches them generically,
+  but I did not enumerate which other workloads carry the policy, so I cannot say loki was the only
+  one at risk. The runbook gives the one-liner to list them.
+* **TrueNAS and the router are entirely outside this change.** TrueNAS is not in the Ansible
+  inventory and refuses SSH, so its shutdown stays a web-UI step with no automation and no check.
+* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so CI proves the L0/L1
+  offline subset only.
+* I did not re-measure branch protection or runner state (CLAUDE.md §2b).

decisions/2026-10-17-keycloak-passkeys.md

@@ -0,0 +1,311 @@
+---
+date: 2026-10-17
+branch: feat/keycloak-passkeys
+title: Passkeys declared for the homelab realm — and the realm import will not apply them
+---
+
+## What changed
+
+`platform/keycloak/keycloak-realm-import.yaml` now declares passkey (WebAuthn Passwordless)
+login for the `homelab` realm, in two parts:
+
+1. **The WebAuthn Passwordless policy.** Ten `webAuthnPolicyPasswordless*` fields on the realm:
+   RP ID `${BASE_DOMAIN}`, RP entity name `Homelab SSO`, signature algorithms `[ES256, RS256]`,
+   `requireResidentKey: "Yes"`, `userVerificationRequirement: required`,
+   `authenticatorAttachment: "not specified"`, `attestationConveyancePreference: none`,
+   `acceptableAaguids: []`, `createTimeout: 300`, `avoidSameAuthenticatorRegister: true`.
+   Deliberately the **passwordless** family only; the two-factor `webAuthnPolicy*` family is left
+   at Keycloak's defaults because nothing in this realm uses `webauthn-authenticator`.
+
+2. **The browser flow.** `browser-mfa` keeps its alias and stays bound as `browserFlow`; its
+   interior is restructured so a passkey is an **ALTERNATIVE** beside the password form:
+
+   ```
+   browser-mfa                            (top level, unchanged)
+     browser-mfa-forms                    ALTERNATIVE
+       auth-username-form                   REQUIRED      <- was auth-username-password-form
+       browser-mfa-passkey-or-password      REQUIRED      <- NEW sub-flow
+         webauthn-authenticator-passwordless  ALTERNATIVE <- NEW: the passkey
+         browser-mfa-password-and-otp         ALTERNATIVE <- NEW sub-flow
+           auth-password-form                   REQUIRED  <- was half of the combined form
+           browser-mfa-conditional-otp          CONDITIONAL (unchanged, relocated intact)
+   ```
+
+The design decisions, with their reasoning, are written into the manifest comments in this repo's
+usual density. The ones that matter:
+
+* **Additive, not replacing. Enrolment optional, not forced.** This IdP gates Grafana, Gitea,
+  Nextcloud, Headlamp, Semaphore, PiHole, Crafty, the \*arr stack *and* `kubectl` via the
+  `kubernetes` OIDC client. A mandatory passkey flow that misfires is a total lock-out from the
+  platform and the cluster API at once, leaving only the Keycloak bootstrap admin and the
+  break-glass cert kubeconfig. The gain from forcing it (one human stops typing a password) does
+  not price that risk.
+* **No `requiredActions` block was added, on purpose.** Declaring one would have been the obvious
+  way to add `webauthn-register-passwordless`, and it would have been actively dangerous — see
+  "What I verified".
+* **RP ID is `${BASE_DOMAIN}`, not `keycloak.${BASE_DOMAIN}`.** A passkey's RP ID is fixed at
+  enrolment and can never be re-keyed, so it must not be bound to a name that can move. This realm
+  already declares an `auth.${BASE_DOMAIN}` redirect URI, and this repo already owns a reconciler
+  built because a hostname move silently failed to reach the realm. Residual risk, unavoidable:
+  changing `BASE_DOMAIN` itself invalidates every enrolled passkey.
+* **A passkey login is not also asked for TOTP.** `browser-mfa-conditional-otp` hangs off the
+  password branch only. The justification is `userVerificationRequirement: required` — the
+  ceremony already proves possession plus a verified human, phishing-resistantly and origin-bound.
+  Stacking a replayable shared secret on top of that buys friction. The manifest names the exact
+  one-line edit to reverse this decision.
+* **Login becomes two screens** (username, then password) for every user, passkey or not, because
+  the server must know who is logging in before it can decide whether a passkey exists. Accepted
+  cost; it is the enabling change.
+* **Usernameless login deliberately not used**, although Keycloak 26.0.6 supports it — see "What I
+  verified" for the measured reason it would have made every login open on a passkey prompt.
+
+`platform/keycloak/README.md` gains a `## Passkeys` section: the decision table, the fact that
+this does not reach the live realm, the two operator paths that would apply it, and the four-step
+browser verification nobody has run.
+
+No secret, key, credential or AAGUID was minted or committed. No file under `clusters/**`,
+`conformance/**`, `ci/**`, `platform/kyverno/**`, `**/rbac.yaml` or `**/*.sops.yaml` was touched,
+so this should not trip `ci/approval-policy.sh`'s sensitive-path review requirement.
+
+## What I verified
+
+**The live realm import has already completed and will never re-run.** Measured on prod,
+2026-10-17:
+
+```
+$ kubectl --context k3s-prod -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
+{"conditions":[{"message":"","status":"True","type":"Done"},
+               {"message":"","status":"False","type":"Started"},
+               {"message":"","status":"False","type":"HasErrors"}]}
+```
+
+`Done=True` / `Started=False`, exactly the one-shot state `platform/keycloak/README.md` already
+documents for client secrets and client URLs. **So the change in this PR has zero effect on how
+anyone logs in today**, and the client-secret/client-URL reconciler does not and cannot close the
+gap: its hard endpoint allow-list makes `/authentication` unreachable by design, and realm-level
+policy fields are not client fields.
+
+**The CRD accepts every field I used.** Measured against the live CRD (not the upstream docs):
+`kubectl --context k3s-prod get crd keycloakrealmimports.k8s.keycloak.org` lists all twelve
+`webAuthnPolicyPasswordless*` properties with the types I wrote (`RequireResidentKey`,
+`RpEntityName`, `RpId`, `UserVerificationRequirement`, `AttestationConveyancePreference`,
+`AuthenticatorAttachment` as `string`; `SignatureAlgorithms`, `AcceptableAaguids`, `ExtraOrigins`
+as `array/string`; `CreateTimeout` as `integer`; `AvoidSameAuthenticatorRegister` as `boolean`),
+plus `authenticationFlows`, `authenticatorConfig` and `requiredActions`. The `spec.realm` schema
+has **no** `x-kubernetes-preserve-unknown-fields`, so a misspelled field would have been pruned
+silently rather than rejected — which is why this was checked rather than assumed.
+
+**Running Keycloak is 26.0.6.** `quay.io/keycloak/keycloak:26.0.6` on `keycloak-0`/`keycloak-1`,
+operator `quay.io/keycloak/keycloak-operator:26.0.6`. Every source claim below is read from the
+`26.0.6` tag, not from memory or from docs.
+
+**Every provider ID is quoted from upstream source, not guessed.**
+
+| String I used | Where it is defined at tag 26.0.6 |
+|---|---|
+| `webauthn-authenticator-passwordless` | `WebAuthnPasswordlessAuthenticatorFactory.PROVIDER_ID`, display "WebAuthn Passwordless Authenticator" |
+| `auth-username-form` | `UsernameFormFactory.PROVIDER_ID`, display "Username Form" |
+| `auth-password-form` | `PasswordFormFactory.PROVIDER_ID`, display "Password Form" |
+| `webauthn-register-passwordless` | `WebAuthnPasswordlessRegisterFactory.PROVIDER_ID` (referenced in comments only) |
+
+**`requiredActions` is REPLACE, not merge — measured, and it is why I did not declare one.**
+`model/storage-private/.../DefaultExportImportManager.java` lines 327-338:
+
+```java
+if (rep.getRequiredActions() != null) {
+    for (RequiredActionProviderRepresentation action : rep.getRequiredActions()) { ... }
+    DefaultRequiredActions.addDeleteAccountAction(newRealm);
+} else {
+    DefaultRequiredActions.addActions(newRealm);
+}
+```
+
+Declaring one required action therefore skips the entire default set. `RealmManager` is no
+backstop: `setupRequiredActions` is `if (realm.getRequiredActionProvidersStream().count() == 0)`,
+and the count is already non-zero by then (`RepresentationToModel.importRealm` runs at
+`RealmManager.java:585`, the backstop at `:612`). A realm built that way would have had the passkey
+action plus `delete_account` and nothing else — **no `CONFIGURE_TOTP`**, which would leave
+`auth-otp-form` demanding a TOTP that no platform-admin could ever enrol: a hard lock-out
+delivered by a change whose purpose was to *add* a login method.
+
+**And declaring it would have bought nothing.** `DefaultRequiredActions.addWebAuthnPasswordlessRegisterAction`
+registers `webauthn-register-passwordless` with `setEnabled(true)`, `setDefaultAction(false)`,
+priority 80, gated on `Profile.Feature.WEB_AUTHN` — which is `Type.DEFAULT` in
+`common/.../Profile.java:74`, i.e. on unless explicitly disabled. `enabled=true` /
+`defaultAction=false` *is* the additive-and-optional posture this change wants.
+
+**`authenticationFlows` has the opposite semantics, which is why declaring flows is safe.**
+`importAuthenticationFlows` adds the declared flows, then backfills each unbound binding
+(browser, registration, direct grant, reset credentials, client auth, first broker login, docker
+auth) from the built-in defaults individually (`DefaultExportImportManager.java` ~1320-1410).
+
+**The graceful degradation to password is measured from the flow engine, not assumed.** For a user
+with no passkey: `WebAuthnPasswordlessAuthenticator.requiresUser()` returns `false`, so
+`DefaultAuthenticationFlow.processSingleFlowExecutionModel` takes the `else` branch (`daf:456-462`),
+whose four conditions all hold — a user is set (from `auth-username-form`), `configuredFor` is
+false, `WebAuthnPasswordlessAuthenticatorFactory.isUserSetupAllowed()` is `false`, and
+`WebAuthnAuthenticator implements Authenticator, CredentialValidator<...>` — so it raises
+`CREDENTIAL_SETUP_REQUIRED`. The ALTERNATIVE loop (`daf:289-302`) catches exactly that, marks the
+execution `ATTEMPTED` and proceeds to the next alternative, with the upstream comment *"the user,
+since he has alternatives, should be able to go to another alternative and continue"*. Hence: no
+passkey, no passkey screen, straight to the password form.
+
+**Why usernameless was rejected, measured.** The `configuredFor` guard sits inside the
+`requiresUser()` branch. With `requiresUser()` false and no user yet in context, that guard is
+skipped and the engine just calls `authenticate(context)` — so a passkey execution placed before
+the username form is offered to *everyone*, including users who will never own one.
+
+**The policy sentinels are read from the shipped code, and one of my initial assumptions was
+wrong.** `Constants.DEFAULT_WEBAUTHN_POLICY_NOT_SPECIFIED = "not specified"` is the
+realm-representation sentinel — but `WebAuthnConstants.OPTION_NOT_SPECIFIED = ""` (empty string) is
+the WebAuthn-API-level one. I checked both because they disagree. The login theme's
+`webauthnRegister.js` at 26.0.6 settles the behaviour:
+
+```js
+if (input.requireResidentKey !== 'not specified') {
+    if (input.requireResidentKey === 'Yes') { authenticatorSelection.requireResidentKey = true; }
+    else { authenticatorSelection.requireResidentKey = false; }
+}
+...
+if (input.createTimeout !== 0) { publicKey.timeout = input.createTimeout * 1000; }
+rp: {id: input.rpId, name: input.rpEntityName}
+```
+
+So `"Yes"` is a literal string compare (any other non-sentinel value means *false*), `"not
+specified"` is compared literally for four fields, and `createTimeout` is **seconds** with `0`
+meaning "send no timeout". It also shows 26.0.6 emits only the legacy `requireResidentKey` member,
+never `residentKey: "required"` — the client maps that per WebAuthn L2, but the policy cannot
+express `residentKey: "preferred"` at all.
+
+`[ES256, RS256]` is exactly `WebAuthnPolicy.DEFAULT_POLICY` /
+`Constants.DEFAULT_WEBAUTHN_POLICY_SIGNATURE_ALGORITHMS` — declared explicitly, not changed.
+`getWebAuthnPolicyPasswordless` was read field by field to confirm `[]` for `acceptableAaguids`
+means "no restriction" and that `RpEntityName` must be non-empty or the **update** path
+(`DefaultExportImportManager.java:842`) drops the whole policy silently.
+
+**The manifest parses and renders.** `yaml.safe_load` round-trips the file and the flow graph reads
+back exactly as designed (five flows, the executions and priorities above, `browserFlow:
+browser-mfa`, `requiredActions` absent). `python3 ci/render-keycloak-client-urls.py --check` →
+`client-urls.yaml is in sync`, rc 0 (no client URL changed, so no regeneration was needed).
+`yamllint -c ci/yamllint.yaml platform/keycloak/keycloak-realm-import.yaml` → rc 0; no line exceeds
+the 160-column ceiling.
+
+**`browser-mfa` keeps its alias deliberately.** `conformance/tests/l2-behavioral/keycloak-config-as-code`
+asserts `kubectl get keycloakrealmimport homelab -o jsonpath={.spec.realm.browserFlow} | grep -q
+'browser-mfa'`. Renaming the top-level flow would have failed that test.
+
+**Envsubst safety.** `platform-config`'s Flux Kustomization declares
+`postBuild.substituteFrom: cluster-vars` (`clusters/base/platform-config.yaml:26-29`), so this file
+is substituted. The only brace token I added is `${BASE_DOMAIN}`, which is declared at
+`clusters/base/cluster-vars.yaml:42` (`home.keithjackson.dev`). Every comment I wrote avoids braces
+entirely — the RP-ID discussion says `keycloak.<BASE_DOMAIN>` with angle brackets precisely so an
+unparseable token cannot fail the whole Kustomization.
+
+**Gate result: rc 3, zero failures, `gitea-settings` the only stage the gate counts as not-run** —
+the documented honest best local outcome. Run in the pinned toolbox from this worktree with all
+three §3 workarounds (`DEV_KUBECONFIG=/dev/null`, plus an `ENGINE` shim bind-mounting the main
+checkout's `.git` and the sibling spec repo with `SPEC_REPO=/home-network-k3s`):
+
+```
+[ ok ] stage spec-immutability: pass      <- NOT skipped; both shim mounts worked, R-TEST-06 covered
+[ ok ] stage decisions-log: pass          <- archive frozen, 48 entries well-formed
+[ ok ] stage lint: pass
+[ ok ] stage l0: pass                     <- incl. flux-postbuild-substitution + keycloak-client-secret-reconciler
+[ ok ] stage render-golden: pass
+[ ok ] stage l1: pass
+[ ok ] stage approval-policy: pass        <- no sensitive path touched, so no non-author review needed
+[skip] stage gitea-settings: SKIPPED      <- needs SETTINGS_API_TOKEN, CI-only; expected
+[skip] stage conformance: SKIPPED         <- cluster-gated L2/L3; expected
+MERGE GATE: UNDETERMINED — Stages that did not run: gitea-settings
+```
+
+No lint finding and no check failure names either file I changed. Touched requirements resolved to
+`R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06`; coverage 51/51, 0 gaps.
+
+**A note on how that number was obtained, because the first attempt was worthless.** My first gate
+run returned the same rc 3 and the same seven passes — against a tree that did not contain this
+change at all. The gate's `tracked_files()` reads **git**, and my edits were still uncommitted, so
+every stage gated `origin/main`. The tell was in the log and I nearly missed it: the
+touched-requirement subset reported UNDETERMINED with *"check out the feature branch you mean to
+gate (not the base branch)"*, and `decisions-log` counted **47** entries rather than 48. **A green
+gate run on an uncommitted worktree is a gate run on the base branch.** The numbers above are from
+the second run, after committing and rebasing.
+
+**The base moved under me mid-run.** `origin/main` advanced from `9a41a8e` to `1a0b208` (PR #70,
+oauth2-proxy OIDC startup race) while I was working. This branch is rebased onto `1a0b208`, so the
+gate above ran on the base CI will actually merge into. The rebase was clean — PR #70 touched
+`platform/oauth2-proxy/helmrelease.yaml` and its own decisions entry, disjoint from these files.
+
+## What I did NOT verify
+
+**This change is unapplied. It changes nothing about the live realm, and I did not attempt to make
+it.** `Done=True` / `Started=False` was measured, not inferred, and I ran no mutating `kubectl`,
+no `flux reconcile`, no admin-API call and no `kcadm`. Whether the operator's eventual
+admin-console reconstruction of this flow matches what is declared here is, by construction,
+unverifiable from git.
+
+**I never enrolled a passkey and never pointed a browser at any of this.** Nothing here has been
+exercised by a real WebAuthn ceremony on any device. Specifically unverified:
+
+* **That the RP ID works.** `home.keithjackson.dev` is a registrable-domain suffix of
+  `keycloak.home.keithjackson.dev`, so it *should* satisfy the browser's check — but I verified
+  that by reading the WebAuthn rule, not by watching a browser accept it. A wrong RP ID fails
+  **silently, in the browser, before Keycloak is involved**, with no server-side log line. If
+  enrolment produces no OS prompt and no error, this is the first suspect.
+* **That `"not specified"` is the right literal for `authenticatorAttachment` in a realm import.**
+  I traced it to `Constants.DEFAULT_WEBAUTHN_POLICY_NOT_SPECIFIED` and to the theme's string
+  compare, but I have not seen a realm import accept it. Omitting the field would default to the
+  same value; if an import ever rejects the literal, drop the line.
+* **That `createTimeout: 300` behaves.** Seconds-vs-milliseconds is read from the theme JS. The
+  claim that 300 is long enough for the cross-device QR/hybrid flow, and that Chrome clamps rather
+  than rejects it, is reasoning about browser behaviour, not a measurement.
+* **Whether a passkey user can still reach the password form.** I argued it from
+  `createAuthenticationSelectionList` producing a "Try another way" option, and I noted that for a
+  user holding both credentials the *first* screen is chosen by the user's own credential ordering
+  rather than by my priorities. Neither was observed.
+* **The two-screen login regression for ordinary users.** `auth-username-form` +
+  `auth-password-form` replacing `auth-username-password-form` is a visible change for every user
+  including non-passkey family accounts, and nobody has logged in through it.
+* **That conditional OTP for platform-admins still fires on the password branch.** The sub-flow and
+  its `conditional-otp-platform-admin` config are byte-identical and merely relocated one level
+  deeper, but "identical YAML" is not "still enforced at runtime". This is the highest-value
+  regression check and it is untested. The previous breakage of this exact step
+  (`condUserRole` vs a non-existent property key) 403'd every platform-admin login.
+* **That a platform-admin authenticating by passkey is not challenged for TOTP.** That is the
+  intended behaviour of the shape I wrote; it is unobserved, and it is also a deliberate
+  relaxation of R-IDP-04 as previously implemented, which the operator may want to reverse.
+
+**Whether the realm import reconciles into an existing realm: measured as NO for the completed
+one-shot, but I did not test a re-import.** I did not delete and recreate the
+`KeycloakRealmImport`, so I cannot say what a re-run would do to the existing `homelab` realm —
+whether it would merge the new policy and flows, duplicate flow aliases, or fail. The 2026-09-06
+total SSO outage documented in `platform/keycloak/README.md` is the reason I did not experiment.
+**Treat the re-import path in the README as untested advice, not a runbook.**
+
+**Nothing was proved about the live realm's current state.** I did not read the realm's existing
+required actions, its existing WebAuthn policies, or its live flow executions — doing so needs the
+admin credential and I did not decrypt or use it. So I cannot confirm that the live realm's
+required-action set actually contains `webauthn-register-passwordless` today; I can only show that
+Keycloak 26.0.6's defaults register it and that this repo's import never declared any required
+actions to suppress them. **That inference has not been checked against the realm.**
+
+**L2/L3 conformance did not run** (no cluster from the gate container; `vars.CLUSTER_AVAILABLE` is
+unset in CI anyway), so the existing `keycloak-config-as-code` behavioural test did not re-execute
+against this change. I reasoned about its assertion by reading it.
+
+**No new conformance test was added.** Nothing in `conformance/` asserts the passkey policy or the
+new flow shape, so a future edit can silently remove either and no gate stage will notice. That is
+a real gap this PR leaves open; closing it would touch `conformance/**`, which is a sensitive path
+needing non-author review.
+
+**The quoted gate run predates this file's own gate-result paragraph.** It gated commit `207f5db`;
+the gate numbers and the rebase note above were then written into this entry, so the pushed tree
+differs from the gated one by this handoff's prose only — no manifest, no check, no generated file.
+`decisions-log` is the only stage that reads this file, and it validates structure (frontmatter,
+the three headings, non-stub bodies), which is unchanged. I did not re-run the full gate for a
+prose edit; CI will gate the pushed commit.
+
+**`gitea-settings` did not run locally** — it needs `SETTINGS_API_TOKEN`, which exists only in CI.
+I did not re-measure branch protection, the runner, or `required_approvals` against the Gitea API
+this session; the claims in `CLAUDE.md` §2 are taken on trust here and are explicitly flagged there
+as needing re-measurement.

decisions/2026-10-17-oauth2-proxy-oidc-startup-race.md

@@ -0,0 +1,99 @@
+---
+date: 2026-10-17
+branch: fix/oauth2-proxy-oidc-startup-race
+title: "oauth2-proxy dies on a cold start because OIDC discovery runs once, before Keycloak serves"
+---
+
+## What changed
+
+Exposed by the 2026-10-17 whole-cluster shutdown for the UPS move. oauth2-proxy performs OIDC
+discovery **once, at startup**, and `os.Exit(1)`s if it fails. On a cold boot Keycloak is not
+serving yet, so the shared Gateway answers 503 and the proxy dies:
+
+```
+[provider.go:55] Performing OIDC Discovery...
+[main.go:73] ERROR: Failed to initialise OAuth2 Proxy: error initialising provider: ...
+             failed to discover OIDC configuration: unexpected status "503": no healthy upstream
+```
+
+It then sits in **exponential CrashLoopBackOff long after Keycloak is healthy** — one of the two
+replicas was still down minutes later and only a manual `kubectl delete pod` cleared it. There is no
+retry flag; the process does not survive a cold start. A readinessProbe cannot help (the container
+is already dead), and CrashLoopBackOff backoff is not tunable per-pod.
+
+`platform/oauth2-proxy/helmrelease.yaml` gains an `extraInitContainers` gate (`wait-for-idp`) that
+polls Keycloak's discovery doc until it answers, then lets the proxy start. A pod in `Init` does not
+accrue CrashLoopBackOff backoff and retries promptly, which is the point.
+
+**Bounded, and fails open.** After ~5 minutes it gives up and starts the proxy anyway, so the worst
+case is exactly today's behaviour and never something new. An unbounded wait would trade a CrashLoop
+for a pod wedged in `Init` forever, which is not an improvement.
+
+## Why this shape and not the alternatives
+
+* **`extraInitContainers` is the chart's own documented hook for precisely this.** Its commented
+  example in `values.yaml` is literally `- name: wait-for-idp`. The key was **confirmed against the
+  pinned chart** (`helm show values oauth2-proxy/oauth2-proxy --version 10.7.1` → `extraInitContainers: []`
+  at line 428), not assumed — a wrong Helm values key is silently ignored, which is the
+  "reviewed, merged, apparently done, still broken" shape this repo keeps getting bitten by.
+* **No new base image.** The chart already runs an `alpine` init container for its built-in
+  `initContainers.waitForRedis` gate, so alpine is a base this chart uses regardless. Pinned to
+  `3.22` (verified to exist upstream) rather than the chart's own `latest`.
+* **Flux `postRenderers` rejected:** it would work independently of the chart schema, but nothing in
+  this tree uses postRenderers, so it would introduce a mechanism with no precedent for a problem
+  the chart already has a hook for.
+* **`--skip-oidc-discovery` rejected:** it removes the startup fetch entirely, which is arguably
+  more elegant, but it means hardcoding four Keycloak endpoints and I could not verify offline that
+  oauth2-proxy then fetches JWKS lazily rather than at startup. oauth2-proxy gates SSO for PiHole,
+  the \*arr apps, qBittorrent, Homepage, Prometheus, Alertmanager and Hubble — a wrong guess locks
+  the operator out of all of them, so the reversible, additive option wins.
+
+The `$$`-free convention matters here: shell variables are written bare (`$i`, `$url`, `$(( ))`)
+with **no braced form anywhere**, because `platform` declares `postBuild.substituteFrom cluster-vars`
+and envsubst consumes braced expansions — a braced shell variable renders to the empty string and
+silently breaks the loop. `${BASE_DOMAIN}` is braced on purpose. Same hazard and same convention as
+the gitea and nextcloud OIDC bootstrap hooks, and `ci: flux-postbuild-substitution` enforces it.
+
+## What I verified
+
+* **The failure is measured, not theorised.** The log above is from the live pod
+  (`oauth2-proxy-5d6c74876b-bzb2p`, `lastState.terminated: {reason: Error, exitCode: 1}`), captured
+  with `--previous` after the cold boot. The surviving replica's log shows the same startup
+  succeeding once Keycloak was up, which is what identifies this as a race rather than a config
+  fault.
+* **Deleting the pod is a sufficient workaround** — done on prod 2026-10-17, the replacement came up
+  clean and both replicas have been healthy since.
+* **The chart key is real**, at the pinned version range: `helm search repo` resolves
+  `>=10.0.0 <11.0.0` to chart `10.7.1` (app `7.15.5`), whose values carry `extraInitContainers: []`,
+  `extraContainers: []` and the built-in `initContainers.waitForRedis` (which uses `alpine`).
+* **`docker.io/library/alpine:3.22` exists** — confirmed against the registry tag list rather than
+  assumed from memory.
+* The edited HelmRelease parses as YAML.
+
+## What I did NOT verify
+
+* **UNAPPLIED. The init container has never run.** I did not reconcile the HelmRelease, so I have not
+  seen `wait-for-idp` execute, exit 0, or gate the proxy. Applying it rolls both oauth2-proxy
+  replicas, which is a mutating action (CLAUDE.md §5) — and it rolls the component that gates SSO for
+  seven apps, so **do it when you can watch it**, not unattended.
+* **I did not re-create the cold-start race to prove the fix.** That needs Keycloak down while
+  oauth2-proxy starts; the honest test is the next whole-cluster boot. Until then this is reasoned
+  from the measured failure, not demonstrated against it.
+* **busybox `wget`'s TLS behaviour here is assumed, not tested.** The claim is that alpine's
+  `ca-certificates-bundle` lets it validate the Gateway's LE-prod certificate, so a genuine TLS
+  failure exits non-zero rather than passing falsely. If that is wrong in either direction the gate
+  degrades safely — a never-succeeding probe just burns the 5-minute budget and then starts the proxy
+  anyway — but the fast path would be silently dead and this would be doing nothing. **Check the
+  init container's log on first apply**: it should print "Keycloak discovery reachable" within
+  seconds on a warm cluster, not the give-up message.
+* **Whether a pod can reach the Gateway hostname at all from this namespace is inherited, not
+  re-proven.** The main container does exactly this fetch today and succeeds, so the path is known
+  good — but note the standing trap that an arbitrary pod gets 403 from every Gateway hostname
+  (`decisions/2026-10-04-plex-advertise-plexdirect.md`); oauth2-proxy is exempt only because
+  Keycloak's ingress policy admits it. An init container in the same pod inherits that identity, so
+  this should hold — unverified.
+* **60 attempts x 5s is not tuned.** It is "comfortably longer than Keycloak's observed cold start",
+  not a measured bound.
+* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline
+  subset only.
+* I did not re-measure branch protection or runner state (CLAUDE.md §2b).

decisions/2026-10-18-coredns-ha.md

@@ -0,0 +1,108 @@
+---
+date: 2026-10-18
+branch: fix/coredns-ha
+title: "coredns ran at one replica, so any node drain was a cluster-wide DNS outage"
+---
+
+## What changed
+
+**MEASURED on prod 2026-10-18: `kube-system/coredns` had `replicas: 1`.** In-cluster DNS was a
+single pod on a single node, so draining or rebooting that node was a **cluster-wide in-cluster DNS
+outage** until the pod rescheduled and became ready — every Service lookup, every CNPG replica
+finding its primary, every operator addressing the API by name. It was invisible because `1/1
+Ready` looks perfectly healthy.
+
+It also became urgent rather than latent: the scheduled reboot automation on
+`feat/automated-upgrades` drains and reboots one node at a time, unattended, overnight — turning a
+single point of failure into a recurring one.
+
+New `platform/coredns/hpa.yaml`: a `HorizontalPodAutoscaler` on the coredns Deployment with
+`minReplicas: 2`, `maxReplicas: 3`, registered in the existing `platform/coredns/kustomization.yaml`
+alongside the split-horizon ConfigMap already there.
+
+**Why an HPA and not `replicas: 2` — this is the whole decision.** k3s owns the coredns Deployment
+and we cannot durably edit it: it is applied by k3s's own deploy controller from a bundled manifest
+(`AddOn/coredns`, source `/data/k3s/server/manifests/coredns.yaml`, `managedFields` managers
+`deploy@k3s-prod-1` and `k3s`, plus an `objectset.rio.cattle.io/hash` label). k3s re-stages its
+bundled manifests on startup, so a `replicas: 2` written there — or a `kubectl scale` — is reverted
+on the next k3s restart or upgrade. Two alternatives were weighed and rejected:
+
+* **`coredns.yaml.skip` + our own templated copy** (the pattern `roles/kube_vip` already uses for a
+  file in `server/manifests`). Deterministic and never flaps, but it takes ownership of coredns away
+  from k3s permanently — k3s upgrades would stop updating it and the version would need refreshing
+  by hand forever. Too high a price for a replica count.
+* **A Flux-managed patch of the Deployment.** Rejected outright: Flux and k3s's deploy controller
+  would both own one object and fight continuously — the exact shape of the `default-deny`
+  NetworkPolicy drift loop this cluster already suffered (~2500 events/10min).
+
+An HPA avoids both: a separate object k3s does not own, driving the `/scale` subresource rather than
+the Deployment spec, so there is no shared-ownership fight. If k3s re-stages `replicas: 1`, the HPA
+sees it below `minReplicas` and scales back within one reconcile.
+
+It is a **floor-holder, not an autoscaler**: the CPU metric exists only because an `autoscaling/v2`
+HPA needs one to be valid. `scaleDown.stabilizationWindowSeconds: 600` because DNS is the worst
+thing in the cluster to flap; `scaleUp` is immediate, since that is the path that restores redundancy
+after a k3s restart.
+
+**Immediate action taken outside this PR:** `kubectl -n kube-system scale deploy coredns
+--replicas=2` was run at the operator's explicit request, so the cluster has two replicas **now**
+rather than after merge. That hand scale is what this PR makes durable.
+
+## What I verified
+
+* **The defect:** `kubectl -n kube-system get deploy coredns` → `replicas=1 available=1`.
+* **k3s ownership:** the `AddOn/coredns` object exists with source
+  `/data/k3s/server/manifests/coredns.yaml`; the Deployment has no `ownerReferences` but carries
+  `objectset.rio.cattle.io/hash` and `managedFields` managers `deploy@k3s-prod-1` / `k3s`. So it is
+  k3s-managed, not Flux-managed — independently corroborated by another agent, which confirmed
+  `platform/coredns/` contains **only** the `coredns-custom` ConfigMap and does not touch the
+  Deployment or its replica count.
+* **The hand scale works and holds:** coredns went to `2/2 Running` on two different nodes, and
+  in-cluster DNS kept answering throughout (`nslookup kubernetes.default.svc.cluster.local
+  10.43.0.10` → `10.43.0.1`). Still `spec.replicas=2 ready=2` after ~2 minutes, with the AddOn
+  checksum **unchanged** — so k3s's deploy controller does not revert on a drift timer; it
+  re-applies on manifest change and on restart.
+* **The HPA's preconditions hold:** coredns ships `requests: {cpu: 100m, memory: 70Mi}` (so a
+  Resource/Utilization metric is meaningful), and metrics-server is installed as a k3s addon and
+  verified serving — `kubectl top pod -l k8s-app=kube-dns` returns **5m and 4m** against that 100m
+  request, i.e. ~5% utilisation, confirming the 80% target will never be approached and the HPA will
+  sit at `minReplicas`.
+* **The kustomization edit is additive:** `git diff --stat` is `1 file changed, 4 insertions(+)`,
+  and `- coredns` was already registered in `platform/kustomization.yaml`, so no change was needed
+  there.
+* All three touched/added YAML files parse.
+
+## What I did NOT verify
+
+* **The HPA has never been applied.** I did not reconcile it, so I have **not** observed it create,
+  reach `ScalingActive=True`, resolve its metric against real coredns data, or hold `minReplicas`.
+  Applying is an operator action (CLAUDE.md §5). The failure mode to watch for is
+  `ScalingActive=False` with `FailedGetResourceMetric` — the HPA would then be present and inert,
+  which looks identical to working because the hand scale already left 2 replicas in place. **Check
+  `kubectl -n kube-system describe hpa coredns` after merge**, not just the replica count.
+* **The central claim that k3s reverts `replicas` on restart is REASONED, NOT OBSERVED.** It rests
+  on k3s re-staging bundled manifests at startup plus the measured ownership evidence above. I did
+  **not** restart k3s to watch the revert happen — doing so is a rolling control-plane restart, an
+  operator action. If k3s in fact preserves a modified replica count, this HPA is unnecessary
+  (though harmless).
+* **I did not confirm the HPA wins the race against k3s's deploy controller after a restart.** Both
+  write the scale; the expectation is the HPA re-asserts within a reconcile, leaving a brief window
+  at 1 replica. Neither the window's length nor the outcome has been measured.
+* **I nearly shipped a destructive change.** My first write of
+  `platform/coredns/kustomization.yaml` replaced the file wholesale, listing only `hpa.yaml` — which
+  would have pruned the existing `coredns-custom` ConfigMap and **broken in-cluster split-horizon
+  DNS (R-DNS-01/03)**. It was caught by a guard assertion before commit and restored from `HEAD`, and
+  the diff is now additive. Recorded because the near-miss is the useful part: `platform/coredns/`
+  already existed and I assumed it did not. **No conformance check would obviously have caught the
+  removal** — it is valid YAML and a legitimate-looking prune.
+* **Whether 2 is enough.** Three nodes, and coredns' anti-affinity only *prefers* spreading, so both
+  replicas could in principle land on one node and a single drain would still take DNS down. I did
+  not add a `topologySpreadConstraint` or a PodDisruptionBudget — both arguably belong here and
+  neither is in this change.
+* **No alert fires if coredns drops to 1 replica**, so a future silent regression (k3s upgrade
+  changing the manifest, the HPA going inert) would not be reported. That gap is not closed here.
+* **No conformance test asserts this**, so a future edit could remove the HPA with no gate stage
+  noticing.
+* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline
+  subset only.
+* I did not re-measure branch protection or runner state (CLAUDE.md §2b).

decisions/2026-10-18-ha-check-catastrophic-regex.md

@@ -0,0 +1,76 @@
+---
+date: 2026-10-18
+branch: fix/ha-check-catastrophic-regex
+title: "One regex made the merge gate take 28 minutes instead of 2"
+---
+
+## What changed
+
+The merge gate started taking **~28 minutes** instead of its usual **<2 minutes**, on every change
+touching `infra/**`. One regex was responsible for essentially all of it.
+
+`conformance/tests/l0-static/ha-control-plane-as-code/check.sh` contained:
+
+```python
+join_when = re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:\s*k3s_server_mode\s*==\s*'secondary'",
+                      K3S_TEXT, re.S)
+```
+
+With `re.S`, every `.` also matches newlines, so `.*?` and the nested `(?:.*\n){0,10}?` can each
+consume the entire document. That is **nested unbounded repetition** — ten levels of a quantifier
+that can each swallow everything — and a failed or late match backtracks exponentially.
+
+Replaced `.` with `[^\n]*` and dropped `re.S`, so the `{0,10}` is a genuine ten-line window with
+nothing to backtrack into. Same intent, same verdict, no DOTALL.
+
+**Why it surfaced now, and it was mine.** `K3S_TEXT` is every `*.yml` in `roles/k3s_server`
+concatenated. The regex was latent for as long as it existed; **PR #71 added ~100 lines to that role
+(the kubelet graceful-shutdown config), which grew the input past the point where exponential
+backtracking stopped being survivable.** Nothing about the check changed — my change grew its input.
+
+**Why only `infra/**` changes were slow.** `ci/touched-requirements.sh:268` maps `infra/*` to
+`R-ENV-01`, and the resulting L0 run is `--include-scope whole-repo --requirements R-ENV-01`, which
+pulls in this whole-repo check. Doc-only and `platform/**` changes never selected it, which is why
+#69 (1m55s) and #72 (1m53s) stayed fast while #71 (27m44s), the #73 merge (27m46s) and #75 all ran
+~28 minutes.
+
+## What I verified
+
+* **The hot spot, by measurement rather than reading.** `docker top` on the live CI job container for
+  run #299 showed `python3 -` with **19:54 of CPU time** under
+  `ha-control-plane-as-code/check.sh`, inside
+  `conformance/run.sh --layer l0 --include-scope whole-repo --requirements R-ENV-01`.
+* **Reproduced locally**: the check did not finish within a 100-second budget on an unmodified
+  checkout.
+* **Localised to the exact line** by extracting the inline Python and running it under
+  `faulthandler.dump_traceback_later(25)`, which named `re/__init__.py:177 in search` called from
+  the `join_when` line — not inferred from reading the patterns.
+* **The pattern is unique in the repo**: `grep -rlnE "\{0,[0-9]+\}\?" conformance/tests/l0-static/*/check.sh`
+  returns only this file, so there is no sibling with the same hazard.
+* **The fix is fast AND verdict-preserving**, tested against the real `K3S_TEXT` (22,179 chars, 403
+  lines): the new regex matches (`YES`) in **0.2 ms**. Sanity-checked that the thing it looks for
+  genuinely exists — one `--server https://` and three `k3s_server_mode == 'secondary'` occurrences.
+* **The whole check now runs in 0.199s, rc 0, 115 assertions**, with the same success message as
+  before.
+
+## What I did NOT verify
+
+* **I did not re-time the full gate in CI after the fix.** The expectation is that `infra/**` changes
+  return to the ~2-minute range, but that is arithmetic from the 19m54s figure, not a measured
+  post-fix run. The PR's own CI run is the first real datapoint — read it.
+* **I did not audit the other 23 L0 checks for different performance pathologies.** This one was
+  found because it was actively burning CPU in a live job; a check that is merely slow (not
+  exponential) would not have stood out, and `ci/gate.sh` has **no per-stage timing** at all
+  (`grep -nE "SECONDS|date \+|elapsed"` finds nothing), so there is no data on where the remaining
+  time goes.
+* **Nothing prevents this class of bug returning.** No check asserts gate stage runtime, and no lint
+  forbids nested quantifiers under `re.S`. A timing budget per stage, or a `re` lint, would close it;
+  neither is written here.
+* **I did not verify my claim about *why* only `infra/**` was affected beyond the path mapping and
+  the observed durations.** The correlation is strong (5 runs, two clear clusters) and the live
+  process list confirms the `--requirements R-ENV-01` scope, but I did not read
+  `conformance/run.sh`'s scope-selection logic end to end.
+* The regex rewrite assumes the `when:` clause is indented with spaces or tabs only (`[ \t]*`). A
+  file using some other whitespace before `when:` would now fail to match where the old DOTALL
+  version would have succeeded. Unlikely in YAML, and the current input matches, but it is a
+  narrowing.

decisions/2026-10-18-renovate-cron-window.md

@@ -0,0 +1,78 @@
+---
+date: 2026-10-18
+branch: fix/renovate-cron-window
+title: "The Renovate cron fired outside Renovate's own schedule, so it would never have opened a PR"
+---
+
+## What changed
+
+`.gitea/workflows/renovate.yaml` shipped (PR #73) with `cron: "15 3 * * *"`. **That schedule could
+never have produced a single PR.**
+
+Gitea Actions cron is **UTC**. `renovate.json` carries `"schedule": ["after 3am and before 6am"]`
+with `"timezone": "America/New_York"`, and that inner schedule is **authoritative** — Renovate
+re-checks it and declines to open PRs outside it. So:
+
+```
+cron 03:15 UTC  ->  23:15 EDT, the PREVIOUS EVENING in New York
+window          ->  03:00-06:00 local
+inside?         ->  NO (3h45m before the window opens)
+```
+
+The original comment claimed the cron was "slightly ahead of the repo window ... so a run is always
+in flight when the window opens". It is 3h45m *behind*, on the previous day. Every nightly run would
+have started, logged "not within schedule", opened **zero** PRs and exited **successfully** — a
+green tick forever, indistinguishable from "nothing needed updating". That is the same
+"failure-mode-is-a-green-tick" shape this repo keeps hitting, and it would have hidden the ~30
+pending updates PR #73 was written to surface.
+
+Changed to `cron: "15 8 * * *"`, chosen to survive DST rather than to be tidy. The window's UTC
+position moves:
+
+| | local window | in UTC |
+|---|---|---|
+| EDT (UTC-4, Mar–Nov) | 03:00–06:00 | 07:00–10:00 |
+| EST (UTC-5, Nov–Mar) | 03:00–06:00 | 08:00–11:00 |
+
+`08:15 UTC` is inside **both** (04:15 EDT / 03:15 EST), so no edit is needed at either switchover.
+A cron cannot follow DST, so this is the only way to get it right once.
+
+## What I verified
+
+* **The defect, by arithmetic rather than assertion:** `datetime(2026,10,18,03,15, UTC)` converted to
+  `America/New_York` via `zoneinfo` is `2026-10-17 23:15 EDT`; `3 <= hour < 6` is **False**.
+* **The fix, at both DST extremes:** `08:15 UTC` → `04:15 EDT` in July and `03:15 EST` in December,
+  both inside `03:00–06:00`. Computed, not reasoned.
+* **The inner schedule really is authoritative and really is set:** `renovate.json` on `main` carries
+  `timezone = America/New_York` and `schedule = ['after 3am and before 6am']`.
+* **`RENOVATE_TOKEN` now exists** as a repository Actions secret (listed by name via the Gitea API
+  alongside `SETTINGS_API_TOKEN`), so the workflow's fail-hard preflight will pass. It did not exist
+  when PR #73 was authored.
+* The edited workflow parses as YAML.
+
+## What I did NOT verify
+
+* **Renovate has still never executed.** This fixes *when* it runs, not *whether it works*. No run
+  has happened, so nothing here proves it extracts dependencies, authenticates to Gitea, or opens a
+  usable PR.
+* **`enabledManagers` may silently disable the custom managers — UNRESOLVED and the likelier
+  remaining bug.** `renovate.json` lists `enabledManagers: [..., 'regex', ...]` and defines
+  `customManagers` (correct modern key; `regexManagers` is absent, which is right). But modern
+  Renovate names the custom manager **`custom.regex`** in `enabledManagers`, not bare `regex`. If
+  the bare name is invalid or ignored, every inline `# renovate:` marker in the tree does nothing —
+  including the `external-snapshotter` `newTag` pin PR #73 flagged as probably inert. **I did not
+  change it**, because guessing at a config key that could fail validation and take the whole run
+  down is worse than leaving a question that one dry run answers definitively. The `dry_run: true`
+  dispatch log settles it: look for a per-manager dependency-extraction count, and for any config
+  validation warning naming `regex`.
+* **Whether a dry run inside vs outside the window differs usefully.** Renovate honours its schedule
+  even in dry-run, so a manual dispatch outside 03:00–06:00 local will also log "not within
+  schedule". That makes the first dispatch's output less informative than it looks; to see what
+  Renovate *would* do, either dispatch inside the local window or override the schedule for that
+  run. Not tested either way.
+* **No alert fires if the schedule stops firing.** PR #73 recorded this gap and it is still open: a
+  Renovate workflow that silently stops running (disabled Actions, expired token, deleted secret)
+  produces no signal at all. Closing it needs a check on last-successful-run age, which nothing
+  does.
+* No conformance test asserts the cron falls inside `renovate.json`'s window, so this exact bug can
+  return. A static check comparing the two is feasible and is not written.

docs/BULK-NFS-STORAGE.md

@@ -1,61 +1,43 @@
 # Bulk NFS storage — the media library and the legacy Nextcloud archive
 
-The operator-facing companion to the `class: bulk` storage entries in `catalog/{plex,sonarr,radarr,nextcloud}.yaml`.
-Everything here was **measured** on the storage host on 2026-09-07, over read-only SSH; nothing is
-assumed. It resolves the long-standing **OQ-2** stub (`platform/longhorn/storageclasses.yaml`) for the
+The operator-facing companion to the `class: bulk` storage entries in
+`catalog/{plex,sonarr,radarr,qbittorrent,nextcloud}.yaml`.
+
+**Rewritten 2026-10-07 for the move off Proxmox onto a standalone TrueNAS at 192.168.2.30.** The
+layout (§2), the mount design (§3, §4) and the uid model (§6) are unchanged by the move and were
+measured on the old host on 2026-09-07. Everything about the storage host itself — §1, §5, §10 —
+was re-measured on the NAS on 2026-10-07. Nothing here is assumed. It resolves the long-standing
+**OQ-2** stub (`platform/longhorn/storageclasses.yaml`) for the
 case that actually exists here — attaching datasets that already hold data — and leaves dynamic bulk
 provisioning unimplemented, because nothing needs it.
 
 ---
 
-## 1. The storage host, and why prod cannot reach it
+## 1. The storage host
 
-The bulk data lives on the **Proxmox host's ZFS pool**. That host has exactly one interface:
+The bulk data lives on a **standalone TrueNAS** (Community Edition) at **192.168.2.30** — the same
+`192.168.2.0/24` as prod's nodes (`192.168.2.10/.11/.12`), so it is reached directly, with no NAT
+and no routing decision.
 
-```
-vmbr0  192.168.1.9/24        default via 192.168.1.1   (pfSense LAN)
-```
-
-| | nodes | reaches 192.168.1.9? |
-|---|---|---|
-| **dev** | 192.168.1.40 / .41 / .42 | **Yes** — same L2 segment, and it already holds an NFS export from this host (`/mnt/Storage/k8s-volumes`, in `/etc/exports`) |
-| **prod** | 192.168.2.10 / .11 / .12 | **No** — 192.168.2.0/24 is pfSense's **WAN**. Ports 22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. 2049 and 111 do not. |
-
-Opening the storage path for prod is a **pending operator decision**. Its two candidate outcomes give
-different values for the `BULK_NFS_SERVER` cluster-var:
-
-| outcome | `BULK_NFS_SERVER` for prod |
-|---|---|
-| port-forward `192.168.2.102:2049` → `192.168.1.9:2049` | `192.168.2.102` |
-| a WAN→LAN route to 192.168.1.0/24 | `192.168.1.9` |
-
-Until it lands, `clusters/prod` holds `TBD-PROD-BULK-NFS-SERVER`. **See §7 for why that placeholder
-behaves differently from the repo's other ones.**
+**This section used to be the hard part of this page.** The datasets were on the Proxmox host, whose
+only interface was `vmbr0 192.168.1.9/24` on pfSense's **LAN**, while prod sits on pfSense's **WAN**.
+tcp/2049 did not traverse that, so prod mounted through a DNAT at `192.168.2.102`, and which address
+belonged in `BULK_NFS_SERVER` depended on an undecided operator choice. Both pools were physically
+moved on 2026-10-07 (`docs/plans/proxmox-to-n5-air-migration.md`, Phase C/D) and all of that is
+simply gone. Measured 2026-10-07: **192.168.2.102:2049 no longer answers; 192.168.2.30:2049 and
+:111 do.**
 
 ### It only needs ONE port
 
 The PVs mount `nfsvers=4.1`. **NFSv4 carries mount and lock in-protocol**, so the only thing that
 must be reachable is **tcp/2049** — no rpcbind/111, no mountd, no statd. NFSv3 would have needed
-four ports, three of them on ephemeral numbers, which is a materially worse firewall story. The
-server advertises `+3 +4 +4.1 +4.2` (`/proc/fs/nfsd/versions`), so v4.1 is available and is the
-conservative floor.
+four ports, three of them on ephemeral numbers. The NAS advertises `+3 +4 +4.1 +4.2`
+(`/proc/fs/nfsd/versions`, measured 2026-10-07), so 4.1 is available and is the conservative floor.
 
-### The export's client list is the SAME under both outcomes — measured, not assumed
-
-**pfSense DNATs the existing port-forward WITHOUT source NAT.** Connecting from this workstation
-(192.168.2.132) to `192.168.2.102:22`, the storage host logged the peer as `192.168.2.132`:
-
-```
-SSH_CONNECTION=192.168.2.132 39768 192.168.1.9 22
-ss -tn  ->  192.168.1.9:22   192.168.2.132:39768
-```
-
-The original WAN-side source address survives the translation (return traffic finds its way back via
-the host's default route, 192.168.1.1). So under **either** outcome the NFS server will see prod's
-**real** node addresses — 192.168.2.10/.11/.12 — and the `sharenfs` change in §5 is the same either
-way. **It does not have to wait for the routing decision.**
-
----
+**NFSv4 is not on by default and its absence does not look like a storage problem.** TrueNAS ships
+NFSv3-only until NFSv4 is added to *Services → NFS → Enabled Protocols*. Without it the mount never
+completes and the pods sit in `ContainerCreating` — no permission error, nothing in the app logs,
+just no volume.
 
 ## 2. The layout, as found
 
@@ -107,16 +89,18 @@ why it is mounted read-only; see §6.
 | plex | `/mnt/Storage/plex` | `/media` | — | **ro** | ReadOnlyMany |
 | sonarr | `/mnt/Storage/plex` | `/media` | — | rw | ReadWriteMany |
 | radarr | `/mnt/Storage/plex` | `/media` | — | rw | ReadWriteMany |
+| qbittorrent | `/mnt/Storage/plex` | `/downloads` | `downloads` | rw | ReadWriteMany |
 | nextcloud | `/mnt/Storage/nextcloud` | `/mnt/legacy-nextcloud` | `nextcloud` | **ro** | ReadOnlyMany |
 
 In-app paths that follow: Plex libraries at `/media/movies`, `/media/tv`, `/media/photos`; Sonarr
 Root Folder `/media/tv`; Radarr Root Folder `/media/movies`; both importing from
 `/media/downloads/complete/{tv-sonarr,radarr}`.
 
-**All three media consumers use the identical mountPath on purpose.** When qbittorrent is eventually
-given the same export (it is not in this change), the arrs and the download client will already agree
-on absolute paths, which is the second precondition for hardlinked imports — otherwise every import
-needs a Remote Path Mapping and quietly falls back to copying when one is missing.
+**The three library consumers use the identical mountPath on purpose**, and qbittorrent has since
+been given the same export (narrowed by `subPath: downloads`, presented at `/downloads` to match its
+existing `Session\DefaultSavePath`). The arrs and the download client therefore agree on absolute
+paths, which is the second precondition for hardlinked imports — otherwise every import needs a
+Remote Path Mapping and quietly falls back to copying when one is missing.
 
 **`/media` and not `/data`, and that is not cosmetic.** The arrs' Longhorn config PVC renders at the
 renderer default `/data/config`. A media mount at `/data` would be mounted **over** it and mask the
@@ -147,62 +131,123 @@ Narrowing that would cost hardlinks, which is the worse trade.
 
 ---
 
-## 5. The export change the operator must apply — STRICTLY ADDITIVE
+## 5. The exports, as configured on TrueNAS — and the settings that decide permissions
 
-**Do not apply this from the repo.** It is an operator step on the storage host, recorded here so the
-exact value is reviewable. The existing consumers **must keep working**: 192.168.1.6, 10.123.1.10 and
-192.168.1.18 on the media tree, and 192.168.1.6 and 192.168.1.5 on the Nextcloud dataset, are the
-live non-k3s prod stack.
+**Do not apply this from the repo.** It is operator configuration on the NAS, recorded here so the
+exact values are reviewable. Everything below was **measured on 192.168.2.30 on 2026-10-07**, after
+the move.
 
-Current values (`zfs get sharenfs Storage`):
+### The exports
 
 ```
-Storage/plex       rw=@192.168.1.6:@10.123.1.10:@192.168.1.18,no_subtree_check,no_root_squash
-Storage/nextcloud  [email protected]:@192.168.1.5,no_subtree_check,no_root_squash
+/mnt/Storage/plex       192.168.2.10 192.168.2.11 192.168.2.12
+                        (sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,root_squash,no_all_squash)
+/mnt/Storage/nextcloud  192.168.2.10 192.168.2.11 192.168.2.12
+                        (sync,wdelay,hide,no_subtree_check,sec=sys,ro,secure,root_squash,no_all_squash)
 ```
 
-### `Storage/plex` — append the six k3s node addresses to the existing `rw=` list
+Per-node addresses rather than `@192.168.2.0/24`, following the precedent of the old host. A `/24`
+would work and would survive adding a node; it also grants the whole subnet.
+
+**All three nodes must appear on BOTH shares.** Any media pod can be scheduled on any node, so a
+missing node is a latent failure, not an immediate one: it surfaces the first time a pod reschedules
+there, as `access denied by server while mounting`, with the cause weeks stale. One was caught
+exactly this way during setup — the plex share listed `192.168.2.13`, an address no node holds, and
+so was missing `192.168.2.12` (k3s-prod-3). Nothing was failing at the time.
+
+### The option flags ARE the permission contract
+
+| flag | why it matters here |
+|---|---|
+| **`no_all_squash`** | The one the whole model rests on. Client uids pass through untouched, so the arrs' `PUID=1000` arrives as the owner of a `0755 1000:3000` tree. In the UI this is **Mapall User/Group left EMPTY** — setting either squashes every client to one identity and erases the reader/writer distinction entirely. |
+| **`sec=sys`** | AUTH_SYS. Numeric uid/gid, not Kerberos principals. §6 is only true because of this. |
+| **`root_squash`** | Maproot left empty. The Proxmox export carried `no_root_squash`; **nothing here needs it** — every write to the tree happens after the LSIO drop to uid 1000, and the images only chown their Longhorn `/config`. This is a deliberate tightening over the old export, not parity. |
+| **`rw` / `ro`** | The archive's read-only posture is **server-enforced**, as it was before. The media tree cannot get the same treatment — see below. |
+| **`secure`** | Reserved source ports required; "Allow non-root mount" off. The kubelet mounts as root from a reserved port and does not need the exception. |
+
+**Plex's read-only posture is still a CLIENT-side guarantee.** Plex is ro and the arrs are rw, but
+all four can land on any node, so the server sees the same three source addresses for all of them
+and cannot distinguish the reader from the writers. `spec.nfs.readOnly: true` on `plex-media` plus
+the volumeMount flag is the whole enforcement. Unchanged by the move; an honest limitation, not an
+oversight. If it ever needs to be server-enforced, the lever is a dedicated node pool for Plex.
+
+### Service settings that are NOT obvious from the export list
+
+- **Enabled Protocols must include NFSv4** (above).
+- **Require Kerberos for NFSv4: off.** The numeric AUTH_SYS pass-through is the model.
+- **Manage Groups Server-side: off.** This is `rpc.mountd --manage-gids`: the server *discards the
+  client's supplied group list* and looks that uid's groups up in TrueNAS's own user database
+  instead. With no local uid 1000 defined, the arrs' `PGID=3000` would simply vanish. Writes would
+  survive anyway — uid 1000 **owns** the tree, so the owner bits grant access without the group —
+  but it is a free way to lose group semantics.
+- **NFSv4 DNS Domain: blank.** It only feeds `idmapd.conf`, which numeric mapping bypasses.
+- **Bind IP 192.168.2.30** means loopback does not answer: check with `showmount -e 192.168.2.30`,
+  not `localhost`.
+
+There is **no** "NFSv3 ownership model for NFSv4" control on current TrueNAS — it was an older
+TrueNAS/CORE option, and no replacement is needed: on a Linux client against a Linux server over
+AUTH_SYS, both the `nfs` and `nfsd` modules default to `nfs4_disable_idmapping=Y`, so NFSv4 carries
+raw numeric ids rather than `user@domain` principals. §9's `ls -n` probe is how you confirm it.
+
+### The ZFS `sharenfs` properties must be CLEARED, not edited
+
+The pools arrived with the old host's exports still set as dataset properties:
+
+```
+Storage/plex              [email protected]:@10.123.1.10:@192.168.1.18:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash
+Storage/nextcloud         [email protected]:@192.168.1.5,[email protected]:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash
+Storage/docker-volumes    [email protected]:@192.168.1.5:@192.168.1.10:@192.168.1.18,no_subtree_check,no_root_squash
+```
+
+TrueNAS generates `/etc/exports` from its own middleware database and will fight a property it did
+not set — the failure mode is two differently-configured exports of one path, with no indication
+which is in force. Cleared 2026-10-07:
 
 ```sh
-zfs set sharenfs='[email protected]:@10.123.1.10:@192.168.1.18:@192.168.1.40:@192.168.1.41:@192.168.1.42:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash' Storage/plex
+zfs inherit sharenfs Storage/plex
+zfs inherit sharenfs Storage/nextcloud
+zfs inherit sharenfs Storage/docker-volumes
 ```
 
-Every pre-existing host is preserved verbatim and no option changes; only clients are added.
+`Storage` is `off`, so each resolves to `off`.
 
-### `Storage/nextcloud` — add the k3s nodes to a **`ro=`** list, not to `rw=`
+### Do NOT touch the dataset's permissions or ACLs
 
-```sh
-zfs set sharenfs='[email protected]:@192.168.1.5,[email protected]:@192.168.1.41:@192.168.1.42:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash' Storage/nextcloud
-```
+The tree's ownership **is** the access story (§6), and TrueNAS offers two ways to destroy it
+irreversibly across 8.33 T.
 
-This is the one place read-only can be enforced **server-side**, and it is worth taking: only
-Nextcloud consumes this dataset from k3s, so the export itself can refuse writes. Even a
-misconfigured client cannot then damage the archive.
+`Storage/plex` carries `acltype=nfsv4` (SCALE's built-in default, `SOURCE: default`) and
+`aclinherit=passthrough` (inherited). Measured 2026-10-07: the directories carry **trivial ACLs
+only** — `nfs4xdr_getfacl` shows just `owner@`/`group@`/`everyone@` matching the mode — so `0755
+1000:3000` governs exactly as it did under Proxmox.
 
-**The media tree cannot get the same treatment.** Plex is read-only and the arrs are read-write, but
-all three may be scheduled on any node, so the server sees the same source addresses for all of
-them and cannot distinguish the reader from the writers. **Plex's read-only posture is therefore a
-client-side guarantee** (`spec.nfs.readOnly: true` on `plex-media`, plus the volumeMount flag) — an
-honest limitation, not an oversight. If it ever needs to be server-enforced, the lever is a
-dedicated node pool for Plex, which is a much larger change.
+- **Stay out of the dataset's Edit Permissions / Edit ACL screen.** On an `acltype=nfsv4` dataset
+  that is a full ACL editor, and its presets write *real* NFSv4 ACLs, recursively if permitted.
+  There is no undo.
+- `aclinherit=passthrough` means a single ACL written at `/mnt/Storage/plex` would be **inherited by
+  everything created under it afterwards** — the blast radius is the whole future library, not one
+  directory.
+- Note GNU `ls`'s `+` marker is **not** a reliable ACL indicator here: it looks for a POSIX ACL
+  xattr, and this dataset stores NFSv4 ACLs elsewhere. Use `nfs4xdr_getfacl`.
 
-Per-node addresses rather than `@192.168.2.0/24` follows the precedent already on this host
-(`/etc/exports` lists the dev nodes individually, as `/32`). A `/24` would work and would survive
-adding a node; it also grants the whole subnet. Per-node is the tighter default.
+`xattr=on` rather than `sa` is a performance detail, not a permissions one, and changing it would
+not affect existing files.
 
 ### Verifying, without changing anything
 
 ```sh
-exportfs -v                  # the new client lines appear, existing ones unchanged
-zfs get sharenfs Storage/plex Storage/nextcloud
+cat /proc/fs/nfsd/versions          # +4.1 must be present
+exportfs -v                         # the two shares, three nodes each, flags as tabulated above
+showmount -e 192.168.2.30           # NOT localhost -- the service is bound to one address
+zfs get -r -o name,property,value,source sharenfs Storage    # every row off or -
 ```
 
----
-
 ## 6. uid / gid — the failure this design exists to avoid
 
-The export uses `sec=sys` and `no_root_squash`, so **the client's numeric uid is passed through
-verbatim** and the server's on-disk ownership is the entire access story. The media tree is
+The export uses `sec=sys` and `no_all_squash`, so **the client's numeric uid is passed through
+verbatim** and the server's on-disk ownership is the entire access story. (It no longer carries
+`no_root_squash`, which the Proxmox export did — see §5. Nothing here needs it, and the uid that
+matters is 1000, not 0.) The media tree is
 `0755 1000:3000` throughout (some directories `1000:1000`), so **only uid 1000 can write it**.
 
 **`fsGroup` cannot fix this, and reaching for it is how this gets misdiagnosed.** The in-tree NFS
@@ -233,24 +278,27 @@ Nextcloud needs no uid work at all, which is one more reason its mount is read-o
 
 ---
 
-## 7. The prod placeholder does NOT fail at apply time — read this before merging
+## 7. Why this knob is unvalidated at create AND immutable at update
+
+**The `TBD-PROD-BULK-NFS-SERVER` placeholder is gone** (2026-10-07) — prod now carries a real
+address, because the NAS is on prod's own subnet and there is no longer a decision to defer. The
+hazard that made it load-bearing is worth keeping on the page, because it is the half that explains
+why §10 exists.
 
 Every other `TBD-PROD-*` in `clusters/prod/kustomization.yaml` is rejected loudly by CRD validation
-(Cilium, LB-IPAM, CNPG, Velero) because it is not a valid IP or URL. **This one is not.**
+(Cilium, LB-IPAM, CNPG, Velero) because it is not a valid IP or URL. **This one was not.**
 
-A PersistentVolume's `spec.nfs.server` is an **unvalidated string**. The API server accepts
-`TBD-PROD-BULK-NFS-SERVER`, the PV is created, and the PVC **binds** to it — binding is pure
-control-plane and never touches the server. The failure appears only when a pod mounts:
+A PersistentVolume's `spec.nfs.server` is an **unvalidated string**. The API server accepted
+`TBD-PROD-BULK-NFS-SERVER`, the PV was created, and the PVC **bound** to it — binding is pure
+control-plane and never touches the server. The failure appeared only when a pod mounted:
 
 ```
 MountVolume.SetUp failed ... Failed to resolve server TBD-PROD-BULK-NFS-SERVER
 ```
 
-so **plex, sonarr, radarr and nextcloud would sit in ContainerCreating on prod.**
-
-**Consequently: set `BULK_NFS_SERVER` in `clusters/prod/kustomization.yaml` BEFORE prod reconciles a
-revision that carries these bulk volumes.** Dev renders a working value and is where these manifests
-are meant to be proven first.
+**The same field is immutable on UPDATE** (§10), which is the other half of the same awkwardness: a
+wrong value is accepted silently at create, and then cannot be corrected in place. Together they are
+why repointing this knob is a delete-and-recreate rather than a reconcile.
 
 ### The static guard this convention leans on is missing from `main` — but the gate now refuses the value anyway
 
@@ -287,10 +335,9 @@ Without the arm, HALF A is what holds the line, by design.
    EPHEMERAL layer"). Plex was fixed; sonarr, radarr, qbittorrent and jackett were not. Fixing it
    means those apps start from an empty config volume, so it belongs in its own change with the
    operator present. **This change does not make it worse** — that is why the media mount is `/media`.
-2. **qbittorrent has no bulk mount.** Today the arrs' import source is written by the **legacy
-   non-k3s** download client, which is coherent while the legacy stack stays live. Giving the k3s
-   qbittorrent the same export at the same `/media` path is the natural follow-up, and the paths were
-   chosen so it needs no Remote Path Mapping.
+2. **RESOLVED — qbittorrent now has the bulk mount** (`subPath: downloads` at `/downloads`), so the
+   import source is written by the k3s download client rather than the legacy non-k3s one. See
+   `catalog/qbittorrent.yaml`.
 3. **`/mnt/Storage/plex/downloads/downloads` is a symlink to `/shared/downloads`**, which is outside
    the export. Over NFS it is a dangling symlink on every client. Harmless, but expect Sonarr to log
    it while scanning `downloads/`.
@@ -300,10 +347,10 @@ Without the arm, HALF A is what holds the line, by design.
 5. **`nconnect` was not set.** It would likely help throughput on the media tree, but it is a tuning
    change that should be measured, not guessed, and it changes client behaviour under server
    failure.
-6. **Nothing was verified against a live cluster.** No `kubectl` context is usable from this
-   workstation (browser-OIDC only), and this change is manifest-authoring under an explicit
-   do-not-apply constraint. Every claim about the *host* is measured; every claim about *binding and
-   mounting* is reasoned from the manifests.
+6. **Every claim about the NAS in §5 is measured (2026-10-07); the post-move MOUNT is not.** The PVs
+   still have to be recreated (§10) before any pod mounts 192.168.2.30, so the §9 probes are the
+   outstanding verification, not a formality. The pre-move equivalents did pass against the old host
+   on 2026-09-12.
 7. **The bulk-tier requirement ID is deliberately not typed on any non-`.md` line of this change.**
    `ci/touched-requirements.sh` attributes requirements from R-IDs appearing in added lines, and the
    in-cluster conformance test for the bulk tier is still a gap, so typing it would select a
@@ -312,36 +359,95 @@ Without the arm, HALF A is what holds the line, by design.
 
 ---
 
-## 9. Validating on dev, once the export is extended
+## 9. Validating, after the PVs are recreated
 
-Dev can prove all of this before prod's network path exists. Read-only checks first:
+These run against **prod**; dev is decommissioned. They are the outstanding verification for the
+move — nothing below has been run against 192.168.2.30. Read-only checks first:
 
 ```sh
-infra/devcontainer/dev.sh run kubectl --context k3s-dev get pv
-infra/devcontainer/dev.sh run kubectl --context k3s-dev get pvc -A | grep -E 'media|legacy-archive'
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr get pod -o wide
+./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pv -o custom-columns=\
+NAME:.metadata.name,SERVER:.spec.nfs.server,PATH:.spec.nfs.path,STATUS:.status.phase'
 ```
 
-Expect **four** PVs — `plex-media`, `sonarr-media`, `radarr-media`, `nextcloud-legacy-archive` (three
-of them naming the *same* export, one per namespace, which is the cross-namespace model) — each
-`Bound` to the same-named PVC, with `RECLAIM POLICY Retain` and an empty `STORAGECLASS`.
+Expect **five** NFS PVs — `plex-media`, `sonarr-media`, `radarr-media`, `qbittorrent-downloads`,
+`nextcloud-legacy-archive` (four of them naming the *same* export, one per namespace, which is the
+cross-namespace model) — each `Bound`, `RECLAIM POLICY Retain`, empty `STORAGECLASS`, and
+**`SERVER 192.168.2.30`**. A PV still showing `192.168.2.102` means the recreate in §10 did not
+happen; Flux will NOT have updated it in place.
 
-Then the two things that actually matter, from inside a pod:
+Then the four things that actually matter, from inside a pod:
 
 ```sh
-# (a) the writer can write — this is the uid check, and it is the one that fails in practice
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr exec deploy/sonarr -- \
-  sh -c 'id; touch /media/tv/.k3s-write-probe && rm /media/tv/.k3s-write-probe && echo WRITE-OK'
+# (a) numeric ids survived the wire -- the idmapping check, new since the TrueNAS move
+kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- sh -c 'ls -n /media | head -3'
+#     expect uid 1000 / gid 3000 on the entries. 65534 / nobody means ids are being mapped, not
+#     passed through, and every import will fail EACCES while the pod still reports Ready.
 
-# (b) hardlinks work BETWEEN the library and the download path — the root-mount check
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr exec deploy/sonarr -- \
+# (b) the writer can write -- the uid check, and the one that fails in practice
+kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- \
+  sh -c 'touch /media/tv/.k3s-write-probe && rm /media/tv/.k3s-write-probe && echo WRITE-OK'
+
+# (c) hardlinks work BETWEEN the library and the download path -- the root-mount check
+kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- \
   sh -c 'set -e; d=/media/downloads/complete/tv-sonarr/.probe; t=/media/tv/.probe-link;
          echo x > "$d"; ln "$d" "$t"; ls -i "$d" "$t"; rm -f "$d" "$t"; echo HARDLINK-OK'
 
-# (c) the reader really is read-only
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n plex exec deploy/plex -- \
-  sh -c 'touch /media/.probe 2>&1 | head -1'      # expect "Read-only file system"
+# (d) the reader really is read-only
+kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c 'touch /media/.probe 2>&1 | head -1'
+#     expect "Read-only file system"
 ```
 
-`(b)` printing two identical inode numbers is the proof that §4's argument holds in practice. If it
+`(c)` printing two identical inode numbers is the proof that §4's argument holds in practice. If it
 prints `EXDEV`, something has split the single mount back into two.
+
+Reading the failures together is what localises the fault: **(b) EACCES with (a) showing `nobody`**
+is idmapping (§5 service settings). **(b) EACCES with (a) showing `1000`** is on-disk ownership —
+stop and check `ls -ldn` on the NAS before changing anything else.
+
+---
+
+## 10. Repointing BULK_NFS_SERVER means DELETING AND RECREATING THE PVs
+
+**A bound PersistentVolume's NFS source is immutable.** Measured by server-side dry-run on
+2026-10-07:
+
+```
+$ kubectl patch pv sonarr-media --type=json --dry-run=server \
+    -p '[{"op":"replace","path":"/spec/nfs/server","value":"192.168.2.30"}]'
+The PersistentVolume "sonarr-media" is invalid: spec.persistentvolumesource: Forbidden:
+  spec.persistentvolumesource is immutable after creation
+```
+
+Note the pairing with §7: `spec.nfs.server` is **unvalidated at create** (any string is accepted and
+the PVC binds) and **immutable at update**. So editing `BULK_NFS_SERVER` alone does not fail loudly
+and does not work either — Flux's apply errors on the five PVs and the pods keep their previous,
+now-dead mount.
+
+This is safe to do because `persistentVolumeReclaimPolicy: Retain` means deleting a PV never touches
+the data, and `catalog/render.py` emits a **pre-set `claimRef`** on each one, so a recreated PV
+rebinds its own PVC with no manual step.
+
+**Operator sequence** (mutating `kubectl` is an operator action, not an agent action):
+
+```sh
+# 1. Stop the consumers. This also releases the hung hard-mounts against the old server:
+#    the pods report Ready, but any real I/O on /media blocks indefinitely once 2049 stops
+#    answering, because the PVs mount `hard`.
+for ns in plex sonarr radarr qbittorrent nextcloud; do
+  kubectl --context k3s-prod -n $ns scale deploy --all --replicas=0
+done
+
+# 2. Delete the five PVs. Retain + no live claim means nothing is reclaimed or erased.
+kubectl --context k3s-prod delete pv plex-media sonarr-media radarr-media \
+  qbittorrent-downloads nextcloud-legacy-archive
+
+# 3. Let Flux reconcile the revision carrying BULK_NFS_SERVER=192.168.2.30, which recreates them.
+#    Confirm .status.lastAppliedRevision matches the GitRepository artifact before continuing.
+
+# 4. Bring the consumers back, then run every probe in §9.
+```
+
+The PVCs pass through `Lost` between steps 2 and 3; that is expected and resolves when the PV with
+the matching `claimRef` reappears. If a PVC is still `Lost` after Flux reconciles, the PV name or
+its `claimRef` namespace does not match — do **not** delete the PVC to "fix" it.
+

infra/ansible/inventory/group_vars/k3s_prod/vars.yml

@@ -420,3 +420,34 @@ flux_bootstrap_env_path: "clusters/prod"
 # rather than leaving an inert drop-in behind, so this group-wide `true` is still correct
 # for it.
 etcd_io_priority_enabled: true
+
+# --- automatic security patching (roles/unattended_upgrades) -------------------------
+# TURNS ON roles/unattended_upgrades FOR PROD ONLY. Role default is false. Same seam as
+# node_baseline_backports_kernel and etcd_io_priority_enabled above.
+#
+# WHAT WAS THERE BEFORE: nothing at all. Measured on all three prod nodes 2026-10-17 —
+# `unattended-upgrades`, `update-notifier-common` and `needrestart` were ALL absent,
+# `apt-mark showhold` was empty on every node, /etc/apt/preferences.d/ was empty, and
+# nothing on the node could create /var/run/reboot-required. These boxes have had ZERO
+# automatic patching since provisioning, and no way to signal that a reboot was needed.
+#
+# WHAT ENABLING THIS DOES *NOT* DO: nothing, until the operator runs
+# playbooks/unattended-upgrades.yml. It is not applied by site.yml (deliberately — see
+# that playbook's header). And once run, it still never upgrades the KERNEL, never
+# restarts k3s, and never reboots:
+#   * kernel — blacklisted `^linux-`, AND the trixie-backports archive the running
+#     kernel comes from (o=Debian Backports, measured) is not an allowed origin. Two
+#     independent exclusions, because this is the EFI-memattr constraint: these HP minis
+#     need >= 6.13 or they can boot with a fraction of their RAM
+#     (PROD-PROVISIONING.md §0b), and 6.12 kernels are still installed and bootable here.
+#     The kernel continues to move ONLY under a deliberate
+#     `node-baseline.yml --tags kernel` run, exactly as before.
+#   * k3s — needrestart blacklist. k3s is not even an apt package on these nodes
+#     (measured: no `k3s`, no `containerd` in dpkg), so the control plane and runtime are
+#     outside apt's reach regardless.
+#   * reboots — Automatic-Reboot "false", asserted by the role against apt's own
+#     effective config. Reboots belong to kured (platform/kured): one node at a time,
+#     under a cluster-wide lock, inside 03:00-06:00, after cordon + drain. Three
+#     control-plane nodes with etcd quorum 2/3 must never reboot themselves
+#     independently, which is what Automatic-Reboot "true" would mean.
+unattended_upgrades_enabled: true

infra/ansible/playbooks/unattended-upgrades.yml

@@ -0,0 +1,36 @@
+---
+# Automatic Debian SECURITY patching for the k3s nodes — the package half of the automated upgrade
+# process. The reboot half is kured (platform/kured), reconciled by Flux; the two are designed
+# together and neither is useful alone. Full rationale for every decision:
+# roles/unattended_upgrades/defaults/main.yml and that role's README.md.
+#
+# TARGET (R-ENV-02): `k3s_hosts` defaults to dev's `k3s_servers`; prod is
+#   ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod
+# Same pattern node-baseline.yml and node-health.yml established — `hosts:` is templated before
+# group_vars exist, so the target is an extra var, never a group_vars value.
+#
+# The role is a NO-OP unless `unattended_upgrades_enabled` is true, which is set for prod only in
+# inventory/group_vars/k3s_prod/vars.yml. Running this against dev changes nothing.
+#
+# DELIBERATELY NOT IN site.yml. site.yml is the bring-up chain (node-baseline -> k3s ->
+# etcd-io-priority -> kube-vip -> cilium -> verify -> flux-bootstrap) and is "not run against prod
+# casually". Automatic patching is not a prerequisite for anything in that chain and nothing in the
+# chain depends on it, so adding it there would only widen what a cluster rebuild touches. It is
+# applied on its own, once, and re-applied when its configuration changes.
+#
+# SAFE TO RUN ON A LIVE CLUSTER, AT ANY TIME. This playbook installs and configures the upgrader. It
+# never runs `apt upgrade`, never restarts k3s, and never reboots — the role asserts the last of
+# those against apt's own effective config rather than trusting the file it just wrote. The first
+# unattended upgrade happens at the next timer firing (04:00 +/-60m), and any reboot that upgrade
+# implies is kured's to schedule, one node at a time, inside 03:00-06:00.
+#
+# TAGS. `--tags verify` re-runs only the read-only assertions and the per-node summary, which is the
+# way to answer "is automatic patching actually working on these boxes, and is anything waiting on a
+# reboot" without rewriting any configuration:
+#   ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify
+- name: Automatic security upgrades on the k3s nodes (kernel excluded; reboots delegated to kured)
+  hosts: "{{ k3s_hosts | default('k3s_servers') }}"
+  become: true
+  gather_facts: true
+  roles:
+    - unattended_upgrades

infra/ansible/roles/k3s_server/defaults/main.yml

@@ -107,3 +107,31 @@ k3s_base_domain: "home.keithjackson.dev"
 
 # Derived, so the realm path is stated once. Override k3s_base_domain, not this.
 k3s_oidc_issuer_url: "https://keycloak.{{ k3s_base_domain }}/realms/homelab"
+
+# --- Graceful node shutdown (kubelet) ----------------------------------------------------------
+# MEASURED 2026-10-17, during the UPS move: graceful node shutdown was NOT configured on any prod
+# node, so `shutdown -h` did NOT drain pods. systemd stops k3s, containerd goes with it, and every
+# pod is SIGKILLed where it stands -- including 9 CloudNativePG instances mid-checkpoint, the UniFi
+# VM, and 25 attached Longhorn volumes. The whole-cluster shutdown that day had to quiesce
+# everything BY HAND first (runbooks/cluster-maintenance-shutdown.md), and the one volume that did
+# not finish detaching came back `degraded` and had to rebuild.
+#
+# With these set, the kubelet takes a systemd inhibitor lock and, on shutdown, evicts ordinary pods
+# first and critical ones last, so a plain `shutdown -h`/reboot drains the node. That makes the
+# manual scale-down in the runbook a belt-and-braces step for a PLANNED whole-cluster shutdown
+# rather than the only thing standing between a reboot and a hard kill.
+#
+# 90s/20s, not larger: the budget is spent on every reboot and node upgrade, and the kubelet waits
+# the FULL ordinary period before it starts on critical pods. Postgres does a fast shutdown in
+# well under 90s here, and Longhorn needs only enough time to flush and detach.
+k3s_kubelet_shutdown_grace_period: "90s"
+k3s_kubelet_shutdown_grace_period_critical_pods: "20s"
+
+# *** THE HALF THAT IS EASY TO MISS, AND SILENTLY BREAKS THE OTHER HALF. ***
+# Graceful node shutdown works by the kubelet holding a logind INHIBITOR DELAY lock. systemd caps
+# how long any inhibitor may delay shutdown at `InhibitDelayMaxSec`, which DEFAULTS TO 5 SECONDS.
+# If that cap is below the grace period above, systemd proceeds anyway and the drain is truncated
+# to ~5s -- i.e. the feature appears configured, reports no error, and still hard-kills pods. So
+# this MUST be >= k3s_kubelet_shutdown_grace_period; the role asserts it rather than trusting it.
+# Set a little above the kubelet's total (90s + 20s) to leave headroom.
+k3s_logind_inhibit_delay_max_sec: 120

infra/ansible/roles/k3s_server/handlers/main.yml

@@ -15,3 +15,12 @@
     port: "{{ k3s_api_port }}"
     timeout: 180
   listen: Restart k3s
+
+# logind re-reads InhibitDelayMaxSec only on restart. Restarting logind does NOT log users out or
+# disturb k3s -- it manages login sessions and inhibitor locks, not workloads. The kubelet picks up
+# the new cap when it next takes its lock, so no k3s restart is needed for THIS half.
+- name: Restart logind
+  ansible.builtin.systemd_service:
+    name: systemd-logind
+    state: restarted
+  listen: Restart logind

infra/ansible/roles/k3s_server/tasks/main.yml

@@ -86,6 +86,16 @@
       # k3s_base_domain (defaults/main.yml), whose [parity] contract with the cluster's
       # BASE_DOMAIN is documented there. Dev's default renders the value it already runs, so
       # config.yaml is unchanged and the Restart handler does not fire.
+      # ── Graceful node shutdown ──────────────────────────────────────────────────────────────
+      # Without these the kubelet takes no systemd inhibitor lock, so `shutdown -h` stops
+      # containerd and SIGKILLs every pod where it stands -- MEASURED on all three prod nodes
+      # 2026-10-17 (defaults/main.yml has the detail). The companion logind drop-in below raises
+      # InhibitDelayMaxSec, WITHOUT WHICH systemd caps the drain at ~5s and these values are
+      # quietly ignored.
+      kubelet-arg:
+        - "shutdown-grace-period={{ k3s_kubelet_shutdown_grace_period }}"
+        - "shutdown-grace-period-critical-pods={{ k3s_kubelet_shutdown_grace_period_critical_pods }}"
+
       kube-apiserver-arg:
         - "oidc-issuer-url={{ k3s_oidc_issuer_url }}"
         - "oidc-client-id=kubernetes"
@@ -183,3 +193,46 @@
     host: "127.0.0.1"
     port: "{{ k3s_api_port }}"
     timeout: 180
+
+# ── Graceful node shutdown: the systemd half — tag `shutdown` ──────────────────────────────────
+# The kubelet's shutdown-grace-period is ENFORCED BY logind, not by the kubelet: the kubelet holds
+# an inhibitor DELAY lock, and systemd will wait at most `InhibitDelayMaxSec` for it (default 5s)
+# before proceeding regardless. A grace period above that cap is therefore silently truncated --
+# the feature looks configured, logs no error, and still hard-kills pods. Assert the relationship
+# instead of documenting it, because the failure is invisible.
+- name: Assert the logind inhibitor cap can actually cover the kubelet grace period
+  ansible.builtin.assert:
+    that:
+      - (k3s_logind_inhibit_delay_max_sec | int) >= (k3s_kubelet_shutdown_grace_period | regex_replace('[^0-9]', '') | int)
+    fail_msg: >-
+      k3s_logind_inhibit_delay_max_sec ({{ k3s_logind_inhibit_delay_max_sec }}s) is below
+      k3s_kubelet_shutdown_grace_period ({{ k3s_kubelet_shutdown_grace_period }}). systemd would
+      stop waiting for the kubelet's inhibitor lock before the drain finished, so pods would be
+      SIGKILLed anyway and graceful node shutdown would be configured but ineffective.
+    quiet: true
+  tags: [shutdown]
+
+- name: Ensure /etc/systemd/logind.conf.d exists
+  ansible.builtin.file:
+    path: /etc/systemd/logind.conf.d
+    state: directory
+    owner: root
+    group: root
+    mode: "0755"
+  tags: [shutdown]
+
+# A drop-in, not an edit of logind.conf: the distro file stays pristine and `state: absent` here
+# would fully revert the change.
+- name: Raise the logind inhibitor delay cap so the kubelet can finish draining
+  ansible.builtin.copy:
+    dest: /etc/systemd/logind.conf.d/10-k3s-graceful-shutdown.conf
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      # Managed by Ansible (roles/k3s_server). See defaults/main.yml.
+      # MUST stay >= kubelet shutdown-grace-period, or the drain is truncated to this value.
+      [Login]
+      InhibitDelayMaxSec={{ k3s_logind_inhibit_delay_max_sec }}
+  notify: Restart logind
+  tags: [shutdown]

infra/ansible/roles/unattended_upgrades/README.md

@@ -0,0 +1,195 @@
+# roles/unattended_upgrades
+
+Automatic Debian **security** patching on the k3s nodes. The kernel is excluded, k3s is never
+restarted, and reboots are delegated to **kured** (`platform/kured`).
+
+- **Playbook:** `playbooks/unattended-upgrades.yml`
+- **Enabled by:** `unattended_upgrades_enabled` — role default `false`, set `true` for prod in
+  `inventory/group_vars/k3s_prod/vars.yml`
+- **Companion component:** `platform/kured` (the reboot half). **Neither is useful alone** — see
+  "The two halves" below.
+- **Every decision and its reasoning:** `defaults/main.yml`. That file is the argument; this one is
+  the orientation.
+
+## The two halves, and why they ship together
+
+| | what it does | what it must NOT do |
+|---|---|---|
+| **this role** (Ansible, host) | installs security updates continuously; leaves `/var/run/reboot-required` behind when one needs a reboot | upgrade the kernel; restart k3s; reboot |
+| **kured** (Flux, cluster) | cordons, drains, reboots and uncordons **one node at a time** when that file exists | decide *whether* a reboot is needed |
+
+Deploying either alone produces a mechanism that looks healthy and does nothing:
+
+- **kured alone** watches a file that **nothing on Debian ever creates**. Measured 2026-10-18,
+  after a first attempt to install Ubuntu's `update-notifier-common` failed on all three nodes
+  with *"No package matching 'update-notifier-common' is available"*:
+  - `apt-cache policy update-notifier-common` → `Candidate: (none)` — the package does not exist
+    in Debian; it is Ubuntu-only.
+  - `/usr/share/update-notifier/notify-reboot-required` → absent.
+  - **No `linux-image-*` postinst references `notify-reboot-required`** — calling it is an Ubuntu
+    patch, so even dropping a script at that path would never be invoked.
+  - No hook under `/etc/apt/apt.conf.d` or `/etc/kernel/` mentions `reboot-required`.
+
+  So **this role ships the writer itself** (`/usr/local/sbin/k3s-notify-reboot-required` plus a
+  `DPkg::Post-Invoke` hook) rather than installing a package. Without it kured would sit at 1/1
+  Running reporting `kured_reboot_required=0` forever.
+- **this role alone** installs patches that a long-running process never picks up, and when a
+  reboot *is* needed there is nothing safe to take it. `Automatic-Reboot "true"` is not the
+  answer: three control-plane nodes with etcd quorum 2/3, each rebooting at its own time, is how
+  the cluster is lost.
+
+## What it was replacing: nothing
+
+Measured on all three prod nodes, 2026-10-17:
+
+| | |
+|---|---|
+| `unattended-upgrades` | **not installed** |
+| `update-notifier-common` | **not installable** — Ubuntu-only, `Candidate: (none)` on trixie |
+| `needrestart` | **not installed** |
+| `apt-mark showhold` | **empty** — no package held on any node |
+| `/etc/apt/preferences.d/` | **empty** — no apt pin declares the backports-kernel intent |
+| `/var/run/reboot-required` | absent, and uncreatable |
+
+These nodes had received **no automatic patching since provisioning**.
+
+## The three decisions
+
+### 1. Security only
+
+Allowed origins are `label=Debian-Security` and nothing else. `trixie-updates` is excluded
+deliberately (not overlooked) — it carries non-security fixes that are useful but are the archive
+through which an unreviewed change would most plausibly arrive on a control-plane node. The knob to
+add it is `unattended_upgrades_include_updates`.
+
+### 2. The kernel is excluded twice over
+
+**This is the highest-risk part of the change.** These HP EliteDesk 800 G5 minis publish a corrupt
+EFI Memory Attributes Table on R21 firmware; Linux `< 6.13` does not validate it before reserving
+memory from its contents, and `k3s-prod-2` twice booted with **2.6 GiB of 16 GB**. Upstream
+`8fbe4c49c0cc` (first in v6.13) is the only fix — there is no kernel cmdline workaround. Full
+write-up: `infra/PROD-PROVISIONING.md` §0b and `roles/node_baseline/defaults/main.yml:21-51`.
+
+Measured 2026-10-17: all three nodes run `7.1.8+deb13-amd64` (`7.1.8-1~bpo13+1`, i.e. the backports
+metapackage), comfortably above the floor — **but `linux-image-6.12.107+deb13-amd64` and
+`linux-image-6.12.94+deb13-amd64` are also still installed, below the floor, bootable, and held by
+nothing.**
+
+So an unattended process must not be able to touch the kernel at all:
+
+1. **Origin exclusion (structural).** The running kernel comes from `o=Debian Backports` (measured
+   via `apt-cache policy`, pin priority 100 — the `NotAutomatic` + `ButAutomaticUpgrades`
+   signature). That origin is not in `Origins-Pattern`, so unattended-upgrades cannot see a kernel
+   upgrade at all.
+2. **Package blacklist (belt and braces).** `^linux-` is blacklisted, so even if the origin list is
+   later widened — the most likely future edit to this role, and the one that would silently re-arm
+   the hazard — the kernel is still untouchable.
+
+The role **asserts both** against `apt-config dump` (apt's own parse of the whole
+`/etc/apt/apt.conf.d/` directory) rather than trusting the file it just wrote.
+
+**This does not stop the kernel moving.** The operator still advances it deliberately with
+
+```
+ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel --limit <host>
+```
+
+which is the only thing that maintains the backports source, the metapackage and the on-node
+provenance stamp. `node_baseline`'s own note that `apt upgrade` keeps the backports kernel current
+still holds for an operator-run apt. What changed is that a **new unattended actor** is forbidden
+from it.
+
+Also blacklisted, for the same "operator-only" reason: `intel-microcode`, `amd64-microcode`,
+`grub-*`. Microcode and bootloader changes take effect on reboot and alter boot behaviour on the
+very firmware that is already misbehaving here.
+
+`Remove-Unused-Kernel-Packages` is **false**, against the Debian default: those older kernels are
+the documented rollback path, and an unattended process pruning them could remove the fallback an
+operator needs in front of a box that will not boot.
+
+### 3. needrestart restarts services — but never k3s
+
+A security-only policy that restarts nothing patches bytes on disk that no running process reads.
+`needrestart` in mode `a` restarts services still mapped to a deleted library, which is what makes
+the policy actually effective.
+
+`k3s.service` is blacklisted. Restarting the Kubernetes control plane unattended at 04:00 is a
+worse outcome than a delayed library fix; that fix lands at the next kured reboot instead. Measured
+2026-10-17, `k3s.service` is the **only** matching unit — k3s bundles its own containerd and
+kubelet, and neither `containerd` nor `containerd.io` is an apt package on these nodes — so this
+one entry is complete today.
+
+## Schedule
+
+`apt-daily` at 03:30 and `apt-daily-upgrade` at 04:00, both `±60m` randomised, via systemd drop-ins.
+Chosen to land **inside** kured's 03:00–06:00 reboot window so that a sentinel written by an upgrade
+can be acted on the same night rather than ~24h later.
+
+Two traps the role handles explicitly:
+
+- `/etc/apt/apt.conf.d/20auto-upgrades` must set the four `APT::Periodic::*` counters. Without
+  them the apt-daily timers fire and **do nothing** — installed, configured, green, and inert.
+- The timer drop-ins clear `OnCalendar=` before setting it. systemd **accumulates** `OnCalendar`
+  entries across drop-ins, so omitting the reset keeps the shipped schedule *in addition* to ours.
+
+## Why a new role rather than extending `node_baseline`
+
+`node_baseline` is the repo's only other apt surface and would have been a defensible home. Two
+reasons against:
+
+1. **Single responsibility.** `node_baseline` is the *first-boot floor* — it runs before k3s exists
+   and is the thing that makes a node installable. Continuous patching policy is a different
+   lifecycle with a different blast radius, and it wants to be runnable (and re-runnable) without
+   going anywhere near that role's data-disk tasks.
+2. **Concurrent work.** Other open branches touch `roles/node_baseline/`. A new role has nothing to
+   merge.
+
+Nothing here duplicates `node_baseline`: it does not add apt sources, does not touch the kernel, and
+does not manage disks.
+
+## Post-merge verification
+
+Running the role is safe on a live cluster at any time — it installs and configures the upgrader, and
+never runs `apt upgrade`, restarts k3s, or reboots.
+
+```bash
+# Apply (prod). Idempotent; re-runnable.
+cd infra/ansible && ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod
+
+# Re-run ONLY the read-only assertions + the per-node pending-reboot summary.
+cd infra/ansible && ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify
+```
+
+Then, on a node, confirm the three properties that matter (all read-only, no sudo needed for the
+first two):
+
+```bash
+ssh [email protected] 'apt-config dump | grep -iE "unattended-upgrade::(automatic-reboot|package-blacklist|origins)"'
+ssh [email protected] 'systemctl list-timers apt-daily-upgrade.timer --no-pager'
+ssh [email protected] 'ls -la /var/run/reboot-required* 2>&1 | head -2
+  ls -l /usr/local/sbin/k3s-notify-reboot-required /etc/apt/apt.conf.d/99-k3s-reboot-required 2>&1
+  dpkg -l needrestart unattended-upgrades | tail -3'
+```
+
+Expect: `Automatic-Reboot "false"`, a blacklist containing `^linux-`, an `Origins-Pattern`
+containing `Debian-Security` and **not** `Backports`, a next-elapse around 04:00, and all three
+packages `ii`.
+
+A dry run of what the upgrader would actually do (needs root, so this one is an operator step):
+
+```bash
+sudo unattended-upgrade --dry-run --debug
+```
+
+The kernel must appear in its blacklist output, not its upgrade list.
+
+## What this role cannot tell you
+
+- **That a security upgrade has ever succeeded.** That is in
+  `/var/log/unattended-upgrades/unattended-upgrades.log` on each node, and nothing in this repo or
+  in Prometheus reads it. There is no alert for "automatic patching has silently stopped working".
+  That is a real, known gap — see the handoff entry
+  `decisions/2026-10-17-automated-upgrades.md`.
+- **That a reboot actually happened.** That is kured's half, and it *is* alerted on
+  (`KuredRebootPending`, `KuredNodeCoverageGap`, `KuredNotReporting` in
+  `platform/kured/alerts.yaml`).

infra/ansible/roles/unattended_upgrades/defaults/main.yml

@@ -0,0 +1,199 @@
+---
+# roles/unattended_upgrades — automatic Debian SECURITY patching on the k3s nodes, with the kernel
+# deliberately excluded and with reboots delegated to kured (platform/kured).
+#
+# OFF BY DEFAULT, ON FOR PROD. Same seam as node_baseline_backports_kernel / etcd_io_priority_enabled:
+# a plain boolean, default false, turned on in inventory/group_vars/k3s_prod/vars.yml. Not a hostname
+# test — that would encode the environment into the role (see node_baseline's defaults for the full
+# argument, which this role follows rather than restates).
+#
+# ═════════════════════════════════════════════════════════════════════════════════════════════════
+# STATE BEFORE THIS ROLE (measured on all three prod nodes, 2026-10-17, via `dpkg-query -W`)
+# ═════════════════════════════════════════════════════════════════════════════════════════════════
+#   unattended-upgrades      NOT INSTALLED
+#   update-notifier-common   NOT INSTALLED
+#   needrestart              NOT INSTALLED
+#   apt-mark showhold        EMPTY (no package is held on any node)
+#   /var/run/reboot-required ABSENT, and nothing on the node can create it
+#   /etc/apt/preferences.d/  EMPTY (no apt pin declares the backports-kernel intent)
+# So: zero automatic patching, and no mechanism by which a reboot could ever be signalled. That last
+# point is why this role and platform/kured must land together — kured watches
+# /var/run/reboot-required, and on these nodes today NOTHING WRITES IT. Deploying kured alone would
+# produce a daemon that is permanently, silently idle while looking perfectly healthy.
+#
+# ═════════════════════════════════════════════════════════════════════════════════════════════════
+# DECISION 1 — SECURITY ONLY, NOT FULL UPGRADES
+# ═════════════════════════════════════════════════════════════════════════════════════════════════
+# Measured `apt-cache policy` on k3s-prod-1, 2026-10-17, gives the four archives and their pins:
+#   trixie            o=Debian,            a=stable,            l=Debian          pin 500
+#   trixie-updates    o=Debian,            a=stable-updates,    l=Debian          pin 500
+#   trixie-security   o=Debian,            a=stable-security,   l=Debian-Security pin 500
+#   trixie-backports  o=Debian Backports,  a=stable-backports,  l=Debian Backports pin 100
+#
+# Only `Debian-Security` is allowed below. The reasoning, in order of weight:
+#
+#   * Security updates are the class where the cost of NOT applying them is unbounded and the
+#     change itself is minimal by Debian policy (targeted fixes against the frozen stable version,
+#     no feature churn). It is the one class worth applying with nobody watching.
+#   * A full `apt upgrade` on these nodes would also pull trixie point-release changes, which is a
+#     larger surface than anyone has reviewed for a 3-node etcd cluster, for no urgency.
+#   * `trixie-updates` is DELIBERATELY EXCLUDED, not overlooked. It carries non-security fixes that
+#     are too urgent to wait for a point release — useful, but it is the archive through which an
+#     unreviewed change would most plausibly arrive, and it is not what "unattended" should mean on
+#     a control-plane node. Add it consciously later if the operator wants it; the knob is below.
+#   * `trixie-backports` is EXCLUDED, AND THAT IS THE KERNEL ARGUMENT — see DECISION 2.
+#
+# ═════════════════════════════════════════════════════════════════════════════════════════════════
+# DECISION 2 — THE KERNEL IS EXCLUDED TWICE OVER. THIS IS THE HIGHEST-RISK PART OF THE CHANGE.
+# ═════════════════════════════════════════════════════════════════════════════════════════════════
+# The constraint: HP's R21 firmware on these EliteDesk 800 G5 minis publishes a corrupt EFI Memory
+# Attributes Table, and Linux < 6.13 does not validate it before reserving memory from its contents
+# — k3s-prod-2 twice booted with 2.6 GiB of 16 GB. Upstream 8fbe4c49c0cc (first in v6.13) is the
+# ONLY fix; there is no kernel cmdline workaround. So these nodes run the backports metapackage
+# `linux-image-amd64` (measured 2026-10-17: version 7.1.8-1~bpo13+1 installed and running on all
+# three, well above the floor). Full write-up: infra/PROD-PROVISIONING.md §0b, and the rationale
+# block in roles/node_baseline/defaults/main.yml:21-51.
+#
+# A NODE THAT COMES BACK FROM AN UNATTENDED UPGRADE ON A 6.12 KERNEL IS A NODE THAT MAY COME BACK
+# WITH A FRACTION OF ITS RAM. Note that 6.12 kernels are not hypothetical here — measured
+# 2026-10-17, `linux-image-6.12.107+deb13-amd64` and `linux-image-6.12.94+deb13-amd64` are BOTH
+# still installed on all three nodes, below the floor, bootable, and nothing holds them.
+#
+# Two independent mechanisms keep unattended-upgrades away from the kernel, and the role asserts
+# both rather than trusting either:
+#
+#   1. ORIGIN EXCLUSION (structural). The running kernel comes from `o=Debian Backports`, which is
+#      not in Allowed-Origins. u-u only ever considers packages from allowed origins, so it cannot
+#      see a backports kernel upgrade at all. This also means the backports kernel receives no
+#      automatic patching — accepted, and in any case backports carries no security guarantees
+#      (node_baseline's defaults state that cost plainly: "no security-tracker commitment, no DSAs").
+#   2. PACKAGE BLACKLIST (belt and braces). `linux-*` is blacklisted below, so even if the origin
+#      list is later widened — the most likely future edit to this file, and the one that would
+#      silently re-arm the hazard — the kernel is still untouchable. Without this, widening origins
+#      to include backports would hand an unattended process the EFI-memattr-critical package.
+#
+# WHAT THIS DOES *NOT* DO, AND MUST NOT BE READ AS DOING: it does not stop the kernel moving. The
+# operator still advances it deliberately with
+#   ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel --limit <host>
+# which is the only thing that maintains the backports source, the metapackage and the on-node
+# provenance stamp. node_baseline's own comment ("`ButAutomaticUpgrades: yes` ... means `apt upgrade`
+# keeps it current once installed") still holds for an operator-run apt; the measured pin priority of
+# 100 is exactly the NotAutomatic + ButAutomaticUpgrades signature that produces that behaviour.
+# Nothing here changes it. What changes is that a NEW unattended actor is forbidden from the kernel.
+#
+# NOT BLACKLISTED, BECAUSE THEY ARE NOT APT PACKAGES AT ALL — measured 2026-10-17: `dpkg-query` finds
+# no `k3s`, no `containerd` and no `containerd.io`. k3s is a single static binary at
+# /usr/local/bin/k3s installed by roles/k3s_server, and it bundles its own containerd and kubelet.
+# So the entire Kubernetes control plane and container runtime are OUTSIDE apt's reach and cannot be
+# upgraded by this role even in principle. That is worth knowing before worrying about it.
+unattended_upgrades_enabled: false
+
+# The packages this role installs. Each one is load-bearing; see the notes.
+unattended_upgrades_packages:
+  # The upgrader itself.
+  - unattended-upgrades
+  # NOTE: `update-notifier-common` was listed here and REMOVED (2026-10-18). It is an UBUNTU
+  # package and does not exist in Debian at all -- `apt-cache policy update-notifier-common` on
+  # trixie returns `Candidate: (none)`, so the play failed outright with "No package matching
+  # 'update-notifier-common' is available" on all three nodes. The sentinel writer is now provided
+  # by this role itself; see unattended_upgrades_sentinel_* below and tasks/main.yml.
+  # Restarts services still mapped to a deleted library after a security upgrade. Without it a
+  # security-only policy patches bytes on disk that no running process reads — see DECISION 3.
+  - needrestart
+
+# --- Allowed origins -----------------------------------------------------------------------------
+# Verbatim apt origin expressions, substituted into 50unattended-upgrades. The ${distro_*} tokens
+# here are UNATTENDED-UPGRADES' OWN placeholders, expanded by it at runtime — they are not Ansible
+# or Flux substitutions, which is why they are single-quoted and why this role's files never travel
+# through Flux envsubst.
+#
+# `l=Debian-Security` plus the `-security` codename is the form Debian's own shipped config uses, and
+# both halves were confirmed against the measured archive metadata above (l=Debian-Security,
+# n=trixie-security). Matching on the LABEL rather than the suite name is what makes this survive the
+# next Debian release.
+unattended_upgrades_allowed_origins:
+  - 'origin=Debian,codename=${distro_codename},label=Debian-Security'
+  - 'origin=Debian,codename=${distro_codename}-security,label=Debian-Security'
+# Set this to add 'origin=Debian,codename=${distro_codename}-updates,label=Debian' — a deliberate
+# widening, kept as a knob so that turning it on is a reviewable one-line change rather than an edit
+# to a heredoc. See DECISION 1 for why it is off.
+unattended_upgrades_include_updates: false
+
+# --- Package blacklist ---------------------------------------------------------------------------
+# Regular expressions matched against package names. See DECISION 2: `linux-` is the EFI-memattr
+# guard and must not be removed without re-reading PROD-PROVISIONING.md §0b.
+unattended_upgrades_blacklist:
+  # Every kernel package: linux-image-*, linux-headers-*, linux-image-amd64 (the metapackage that
+  # actually tracks backports), plus linux-kbuild / linux-libc-dev. Anchored at the start so it
+  # cannot accidentally match an unrelated package containing the substring.
+  - '^linux-'
+  # Microcode changes take effect only on reboot and can alter boot behaviour on the very firmware
+  # that is already misbehaving here. Operator-only, same argument as the kernel.
+  - '^intel-microcode$'
+  - '^amd64-microcode$'
+  # The bootloader. A grub upgrade rewrites boot configuration on a machine with a documented
+  # warm-reboot hang (k3s-prod-2, 2026-09-03). Never unattended.
+  - '^grub-'
+  - '^grub2-common$'
+
+# --- Reboot policy -------------------------------------------------------------------------------
+# FALSE, AND THIS IS THE DIVISION OF LABOUR WITH kured, NOT AN OVERSIGHT.
+# unattended-upgrades can reboot a node itself (Automatic-Reboot "true"). It must not here: it has
+# no notion of a cluster, so it would not cordon, would not drain, would not evict the 25 attached
+# Longhorn volumes cleanly, would not check whether another node is already down, and — on three
+# control-plane nodes where etcd quorum is 2 of 3 — three nodes each deciding independently to
+# reboot at their own `Automatic-Reboot-Time` is precisely how you lose the cluster.
+# u-u therefore only ever leaves the sentinel behind; kured owns every reboot, one node at a time,
+# under a cluster-wide lock. See platform/kured/helmrelease.yaml.
+unattended_upgrades_automatic_reboot: false
+
+# --- needrestart (DECISION 3) --------------------------------------------------------------------
+# See the role README for the full argument. Summary: 'a' (restart services automatically) is what
+# makes a security-only policy actually effective — a patched libssl on disk does nothing for a
+# process still mapping the old one — but k3s.service is EXCLUDED, because restarting the
+# Kubernetes control plane unattended at 06:00 is a worse outcome than a delayed library fix.
+# Measured 2026-10-17: the only matching unit on these nodes is `k3s.service` (no separate
+# containerd or kubelet unit exists; k3s bundles both), so this blacklist is complete today.
+unattended_upgrades_needrestart_mode: a
+unattended_upgrades_needrestart_blacklist:
+  - '^k3s'
+
+# --- Schedule ------------------------------------------------------------------------------------
+# apt's own systemd timers (apt-daily, apt-daily-upgrade) drive u-u; these set their windows.
+# 04:00 with a 1h randomised delay, so the upgrade lands INSIDE kured's 03:00-06:00 reboot window
+# (platform/kured/helmrelease.yaml) rather than hours away from it: if an upgrade ever does set the
+# sentinel, the reboot can be taken the same night instead of waiting ~24h. The randomised delay
+# keeps the three nodes from hitting the mirror simultaneously; it does NOT serialise anything, and
+# it does not need to — u-u never reboots, so three concurrent upgrades are harmless.
+unattended_upgrades_download_time: "03:30"
+unattended_upgrades_upgrade_time: "04:00"
+unattended_upgrades_randomised_delay: 60m
+
+
+# ── THE REBOOT SENTINEL, PROVIDED BY THIS ROLE ──────────────────────────────────────────────────
+# /var/run/reboot-required is the ONLY thing kured watches, and on Debian trixie **nothing creates
+# it**. Measured on all three nodes 2026-10-18:
+#   * `update-notifier-common` (which owns that hook on Ubuntu) has `Candidate: (none)` — the
+#     package does not exist in Debian. Installing it is what broke the first run of this play.
+#   * `/usr/share/update-notifier/notify-reboot-required` is absent.
+#   * NO `linux-image-*` postinst references `notify-reboot-required` — calling it is an Ubuntu
+#     patch, not Debian behaviour, so merely dropping a script at that path would never be invoked.
+#   * No hook under /etc/apt/apt.conf.d or /etc/kernel/ mentions reboot-required.
+# So the writer has to be ours. Without it kured is a permanent, INVISIBLE no-op: it stays 1/1
+# Running and reports kured_reboot_required=0 forever, which is the "failure mode is a green tick"
+# shape this repo keeps getting burned by.
+#
+# MECHANISM: a dpkg Post-Invoke hook runs a script that compares the RUNNING kernel against the
+# newest INSTALLED one and touches the sentinel when they differ. Deliberately kernel-only, which
+# matches the design: unattended-upgrades is security-only with `^linux-` blacklisted, so it will
+# essentially never set this — the sentinel appears when the OPERATOR deliberately runs the
+# node_baseline kernel play, and kured then automates only the mechanical reboot.
+#
+# It uses `linux-version` (from `linux-base`, a dependency of every linux-image package, so
+# guaranteed present — measured `linux-base 4.12.1` installed) rather than parsing `dpkg -l`
+# output, because kernel version ordering is not string ordering.
+#
+# /run is a tmpfs (measured), so the sentinel self-clears on boot: after a reboot the running
+# kernel IS the newest and the script writes nothing. No cleanup task is needed, and none exists.
+unattended_upgrades_sentinel_script: /usr/local/sbin/k3s-notify-reboot-required
+unattended_upgrades_sentinel_hook: /etc/apt/apt.conf.d/99-k3s-reboot-required

infra/ansible/roles/unattended_upgrades/handlers/main.yml

@@ -0,0 +1,29 @@
+---
+# Re-arm apt's periodic timers after a config or drop-in change.
+#
+# WHY A daemon_reload IS REQUIRED AND NOT OPTIONAL. The timer windows are set by drop-ins under
+# /etc/systemd/system/apt-daily{,-upgrade}.timer.d/. systemd does not re-read unit files on its own,
+# so until it is told to, the RUNNING manager keeps the shipped schedule and the new window is
+# ignored — the files look right in git and on disk, `systemctl cat` shows them, and the upgrade
+# still fires at the old time. That is a silent config no-op, which is the failure mode this whole
+# role is written against.
+#
+# `daemon_reload` re-parses unit files and re-runs generators. It starts and stops nothing, so it
+# cannot disturb k3s, containerd or Longhorn — the same reasoning node_baseline's "Reload systemd
+# unit files" handler records.
+#
+# RESTARTING THE TIMERS IS ALSO SAFE, and is not the same as running the upgrade. `*.timer` units
+# only hold the schedule; restarting one recomputes its next elapse and does NOT trigger its
+# `*.service`. With `Persistent=true` set in the drop-in, a window missed while the node was down is
+# caught up on the next boot rather than skipped.
+#
+# Both fire only when a task actually changed, so a re-run of the role is silent.
+- name: Restart unattended-upgrades timers
+  ansible.builtin.systemd_service:
+    name: "{{ item }}"
+    state: restarted
+    enabled: true
+    daemon_reload: true
+  loop:
+    - apt-daily.timer
+    - apt-daily-upgrade.timer

infra/ansible/roles/unattended_upgrades/tasks/main.yml

@@ -0,0 +1,369 @@
+---
+# roles/unattended_upgrades — automatic Debian security patching, kernel excluded, reboots delegated
+# to kured. Every decision and its reasoning lives in defaults/main.yml; this file is the mechanism.
+#
+# TAGS — following roles/node_baseline, every task carries one of `packages`, `config` or `verify`.
+# An untagged run does all three in that order. The split exists because `verify` is the half an
+# operator will want to re-run on its own to answer "is automatic patching actually working on this
+# box", without re-writing any configuration:
+#   ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify
+#
+# IT DOES NOT UPGRADE ANYTHING, AND IT DOES NOT REBOOT. This role installs and CONFIGURES the
+# upgrader; it never runs `apt upgrade` itself and never takes a reboot. Both of those are deliberate
+# and both follow roles/node_baseline/tasks/main.yml:61-67 ("IT DOES NOT REBOOT", after k3s-prod-2
+# hung on a warm reboot on 2026-09-03 and needed a physical power button). Applying this role is
+# therefore a SAFE operation on a live cluster at any time of day: the first unattended upgrade
+# happens at the next timer firing, and any reboot it implies is kured's to schedule inside the
+# 03:00-06:00 window.
+- name: Automatic security upgrades (unattended-upgrades; kernel excluded, reboots via kured)
+  when: unattended_upgrades_enabled | bool
+  tags: [packages, config, verify]
+  block:
+
+    # ── PREFLIGHT ───────────────────────────────────────────────────────────────────────────────
+    - name: Assert this is a Debian-family host
+      # The config file layout, the origin expressions and the package names below are all
+      # Debian-specific. On anything else this role would write files that are silently ignored,
+      # which is the failure mode this whole PR exists to avoid.
+      ansible.builtin.assert:
+        that:
+          - ansible_facts['os_family'] == 'Debian'
+        fail_msg: >-
+          REFUSING TO CONFIGURE unattended-upgrades on {{ inventory_hostname }}: os_family is
+          '{{ ansible_facts["os_family"] }}', not 'Debian'. The allowed-origin expressions and
+          config paths in this role are Debian-specific.
+      tags: [packages, config, verify]
+
+    # ── PACKAGES ────────────────────────────────────────────────────────────────────────────────
+    - name: Install unattended-upgrades and needrestart
+      # The reboot-sentinel writer is NOT a package here. On Debian nothing ships one -- see
+      # defaults/main.yml -- so this role installs its own script and dpkg hook, two tasks below.
+      ansible.builtin.apt:
+        name: "{{ unattended_upgrades_packages }}"
+        state: present
+        update_cache: true
+        cache_valid_time: 3600
+      tags: [packages]
+
+    # ── THE REBOOT SENTINEL WRITER (ours, because Debian ships none) ────────────────────────────
+    # Full rationale and the four measurements behind it: defaults/main.yml. Short version: kured
+    # watches /var/run/reboot-required and nothing on Debian creates it, so without these two files
+    # the entire reboot half is an invisible no-op.
+    - name: Install the reboot-sentinel writer
+      ansible.builtin.copy:
+        dest: "{{ unattended_upgrades_sentinel_script }}"
+        owner: root
+        group: root
+        mode: "0755"
+        content: |
+          #!/bin/sh
+          # MANAGED BY ANSIBLE -- infra/ansible/roles/unattended_upgrades. Local edits are reverted.
+          #
+          # Touch /var/run/reboot-required when the RUNNING kernel is not the newest INSTALLED one.
+          # kured watches that path; see the role's defaults/main.yml for why this is ours to ship.
+          #
+          # `linux-version` comes from linux-base (a dependency of every linux-image package), and
+          # is used rather than sorting `dpkg -l` output because kernel version ordering is not
+          # string ordering. If it is somehow missing, exit 0 and write nothing: a dpkg hook must
+          # never fail an apt transaction, and a missing sentinel is caught by the verify assert.
+          set -eu
+          command -v linux-version >/dev/null 2>&1 || exit 0
+
+          running="$(uname -r)"
+          newest="$(linux-version list 2>/dev/null | linux-version sort --reverse 2>/dev/null | head -n1)"
+
+          # No newest (nothing installed we can read) -> nothing to say.
+          [ -n "$newest" ] || exit 0
+
+          if [ "$running" != "$newest" ]; then
+            touch /var/run/reboot-required 2>/dev/null || exit 0
+            # kured only needs the file, but the .pkgs companion is what an operator reads to find
+            # out WHY a node is waiting, so write a reason rather than an empty marker.
+            printf 'linux-image-%s\n' "$newest" > /var/run/reboot-required.pkgs 2>/dev/null || true
+          fi
+          exit 0
+      tags: [packages, config]
+
+    - name: Install the dpkg hook that runs the sentinel writer
+      # Post-Invoke, so it runs after dpkg has finished and a newly installed kernel is already
+      # visible to `linux-version list`. `|| true` so a failure here can never abort an apt
+      # transaction -- an unwritten sentinel is a missed reboot, but a broken apt is a broken node.
+      ansible.builtin.copy:
+        dest: "{{ unattended_upgrades_sentinel_hook }}"
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          // MANAGED BY ANSIBLE -- infra/ansible/roles/unattended_upgrades. Local edits reverted.
+          // Writes /var/run/reboot-required when the running kernel is not the newest installed.
+          DPkg::Post-Invoke { "{{ unattended_upgrades_sentinel_script }} || true"; };
+      tags: [packages, config]
+
+    # ── CONFIG ──────────────────────────────────────────────────────────────────────────────────
+    # Written as a whole file rather than patched in place, for the reason node_baseline gives for
+    # its own apt source: an idempotent full-file write with a managed-by header is reviewable and
+    # re-runnable, whereas line-editing a conffile fights dpkg on every package upgrade.
+    #
+    # 52- RATHER THAN 50-: Debian ships /etc/apt/apt.conf.d/50unattended-upgrades as a CONFFILE of
+    # the unattended-upgrades package. Overwriting it would make every future apt upgrade of that
+    # package prompt about a locally-modified conffile (or, under the non-interactive defaults a
+    # scripted run uses, silently keep one side and discard the other). apt reads the directory in
+    # lexical order and later files win, so a separate 52- file overrides the shipped defaults
+    # without ever touching them, and `apt-config dump` in the verify block below proves which value
+    # actually won rather than assuming this ordering holds.
+    - name: Configure unattended-upgrades (52-k3s-unattended-upgrades)
+      ansible.builtin.copy:
+        dest: /etc/apt/apt.conf.d/52-k3s-unattended-upgrades
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          // MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades. Local edits are reverted
+          // on the next run. Rationale for every value: that role's defaults/main.yml.
+          //
+          // Overrides the shipped 50unattended-upgrades by lexical ordering (52 > 50). The shipped
+          // file is a dpkg conffile and is deliberately left untouched.
+
+          // SECURITY ONLY. Deliberately excludes trixie-updates (non-security fixes) and
+          // trixie-backports (o=Debian Backports) -- and excluding backports is what makes it
+          // structurally impossible for this upgrader to move the EFI-memattr-critical kernel,
+          // which is installed from there. See defaults/main.yml DECISION 2.
+          Unattended-Upgrade::Origins-Pattern {
+          {% for origin in unattended_upgrades_allowed_origins %}
+                  "{{ origin }}";
+          {% endfor %}
+          {% if unattended_upgrades_include_updates | bool %}
+                  "origin=Debian,codename=${distro_codename}-updates,label=Debian";
+          {% endif %}
+          };
+
+          // NEVER AUTOMATICALLY UPGRADED, whatever the origin list later says. `^linux-` is the
+          // EFI-memattr guard: these HP minis need kernel >= 6.13 or they can boot with a fraction
+          // of their RAM, and 6.12 kernels are still installed and bootable on all three nodes.
+          // Do not remove without re-reading infra/PROD-PROVISIONING.md §0b.
+          Unattended-Upgrade::Package-Blacklist {
+          {% for pkg in unattended_upgrades_blacklist %}
+                  "{{ pkg }}";
+          {% endfor %}
+          };
+
+          // REBOOTS BELONG TO kured (platform/kured), NOT TO apt. All three nodes are
+          // control-plane + etcd with quorum 2 of 3, so three hosts independently deciding to
+          // reboot at their own Automatic-Reboot-Time is how the cluster is lost. u-u leaves
+          // /var/run/reboot-required behind; kured takes a cluster-wide lock and reboots exactly
+          // one node at a time, after cordon + drain, inside 03:00-06:00, gated on Longhorn volume
+          // health.
+          Unattended-Upgrade::Automatic-Reboot "{{ unattended_upgrades_automatic_reboot | bool | lower }}";
+          Unattended-Upgrade::Automatic-Reboot-WithUsers "false";
+
+          // Keep the node's disk from filling with superseded .debs, and keep a log an operator can
+          // read after the fact (/var/log/unattended-upgrades/). MinAge guards against removing a
+          // package still needed by a rollback.
+          Unattended-Upgrade::Remove-Unused-Kernel-Packages "false";
+          Unattended-Upgrade::Remove-New-Unused-Dependencies "true";
+          Unattended-Upgrade::Remove-Unused-Dependencies "false";
+          Unattended-Upgrade::AutoFixInterruptedDpkg "true";
+          Unattended-Upgrade::MinimalSteps "true";
+          Unattended-Upgrade::SyslogEnable "true";
+
+          // Remove-Unused-Kernel-Packages is FALSE on purpose, against the Debian default. These
+          // nodes deliberately carry more than one kernel: the backports kernel they boot and the
+          // distro kernels that are the documented rollback path (node_baseline writes the rollback
+          // procedure onto the node itself). An unattended process pruning kernels could remove the
+          // fallback an operator needs at 3am in front of a box that will not boot.
+      notify: Restart unattended-upgrades timers
+      tags: [config]
+
+    - name: Enable the apt periodic timers that actually run unattended-upgrades
+      # Installing the package is NOT sufficient to make it run on a schedule: without these
+      # counters apt's daily units do nothing, and the whole mechanism is inert while looking
+      # installed. `1` means "every day".
+      ansible.builtin.copy:
+        dest: /etc/apt/apt.conf.d/20auto-upgrades
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          // MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades.
+          // Without these four counters the apt-daily timers fire and do nothing, so
+          // unattended-upgrades would be installed, configured, and never once invoked.
+          APT::Periodic::Update-Package-Lists "1";
+          APT::Periodic::Download-Upgradeable-Packages "1";
+          APT::Periodic::Unattended-Upgrade "1";
+          APT::Periodic::AutocleanInterval "7";
+      tags: [config]
+
+    - name: Ensure the systemd timer drop-in directories exist
+      # `copy` does not create parent directories, and these two .timer.d/ dirs do not exist on a
+      # stock Debian node — without this task the next task fails with "Destination directory does
+      # not exist" on a first run.
+      ansible.builtin.file:
+        path: "/etc/systemd/system/{{ item }}.timer.d"
+        state: directory
+        owner: root
+        group: root
+        mode: "0755"
+      loop:
+        - apt-daily
+        - apt-daily-upgrade
+      tags: [config]
+
+    - name: Pin the apt timer windows into kured's reboot window
+      # Overriding the shipped timers rather than editing them: a drop-in survives package upgrades.
+      # OnCalendar is cleared first (`OnCalendar=` with no value) because systemd ACCUMULATES
+      # OnCalendar entries across drop-ins — omit the reset and the unit keeps its shipped schedule
+      # IN ADDITION to ours, which would run the upgrade at both times and look like it worked.
+      ansible.builtin.copy:
+        dest: "/etc/systemd/system/{{ item.unit }}.timer.d/override.conf"
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          # MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades.
+          # Lands the upgrade inside kured's 03:00-06:00 reboot window so a sentinel written by an
+          # upgrade can be acted on the same night rather than ~24h later.
+          [Timer]
+          OnCalendar=
+          OnCalendar={{ item.time }}
+          RandomizedDelaySec={{ unattended_upgrades_randomised_delay }}
+          Persistent=true
+        force: true
+      loop:
+        - { unit: apt-daily, time: "{{ unattended_upgrades_download_time }}" }
+        - { unit: apt-daily-upgrade, time: "{{ unattended_upgrades_upgrade_time }}" }
+      loop_control:
+        label: "{{ item.unit }} at {{ item.time }}"
+      notify: Restart unattended-upgrades timers
+      tags: [config]
+
+    - name: Configure needrestart (restart services, but never k3s)
+      # See defaults/main.yml DECISION 3. Auto-restart is what makes a security-only policy real --
+      # a patched library on disk does nothing for a process still mapping the deleted one -- but
+      # k3s.service is excluded because restarting the control plane unattended is worse than a
+      # delayed library fix. `$nrconf{restart}` is Perl config read by needrestart itself.
+      ansible.builtin.copy:
+        dest: /etc/needrestart/conf.d/50-k3s.conf
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          # MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades.
+          # Restart mode: '{{ unattended_upgrades_needrestart_mode }}'  (a = automatically, l = list only, i = interactive)
+          $nrconf{restart} = '{{ unattended_upgrades_needrestart_mode }}';
+          # Units matching these patterns are NEVER restarted automatically. k3s bundles containerd
+          # and the kubelet, so this one entry covers the entire control plane and runtime on these
+          # nodes (measured 2026-10-17: k3s.service is the only matching unit). A library fix that
+          # needs k3s restarted therefore waits for the reboot kured will take.
+          $nrconf{blacklist_rc} = [
+          {% for pat in unattended_upgrades_needrestart_blacklist %}
+              qr({{ pat }}),
+          {% endfor %}
+          ];
+      tags: [config]
+
+    # ── VERIFY ──────────────────────────────────────────────────────────────────────────────────
+    # These tasks exist because every component of this mechanism fails SILENTLY. An installed
+    # package, a written config file and a green play are all compatible with nothing ever being
+    # upgraded and no reboot ever being signalled.
+
+    - name: Ask apt-config what the effective settings actually are
+      # `apt-config dump` is apt's OWN parse of the whole /etc/apt/apt.conf.d/ directory, so it
+      # proves what WON after lexical ordering rather than what this role wrote. If the shipped
+      # 50unattended-upgrades were ever renamed above 52-, this is the task that would catch it.
+      ansible.builtin.command: apt-config dump
+      register: unattended_upgrades_aptconfig
+      changed_when: false
+      check_mode: false
+      tags: [verify]
+
+    - name: Assert the kernel is blacklisted and automatic reboots are off
+      # The two properties that must never silently regress. Asserted against apt's own view.
+      ansible.builtin.assert:
+        that:
+          # apt-config renders each list entry as `Unattended-Upgrade::Package-Blacklist:: "^linux-";`
+          # so the anchored pattern itself is the thing to look for. Checking for the bare string
+          # `linux-` would pass on almost any dump and prove nothing.
+          - "'\"^linux-\"' in unattended_upgrades_aptconfig.stdout"
+          - "'Unattended-Upgrade::Automatic-Reboot \"false\"' in unattended_upgrades_aptconfig.stdout"
+          # The origin list must be non-empty, or u-u runs nightly and upgrades nothing — installed,
+          # scheduled, green, and inert.
+          - "'Debian-Security' in unattended_upgrades_aptconfig.stdout"
+        fail_msg: >-
+          EFFECTIVE apt CONFIG ON {{ inventory_hostname }} IS NOT WHAT THIS ROLE INTENDED.
+          Either the `^linux-` kernel blacklist or `Automatic-Reboot "false"` is not winning.
+          Both are safety properties, not preferences: without the blacklist an unattended process
+          can move a kernel these HP minis must keep >= 6.13 (PROD-PROVISIONING.md §0b), and
+          without Automatic-Reboot=false three control-plane nodes can each reboot themselves
+          independently and take etcd below quorum. Inspect with `apt-config dump | grep -i
+          unattended` and check for a file sorting after 52- in /etc/apt/apt.conf.d/.
+        success_msg: >-
+          Effective apt config confirmed: kernel blacklisted, automatic reboot disabled.
+      tags: [verify]
+
+    # THE SENTINEL IS THE SINGLE POINT OF FAILURE FOR THE ENTIRE REBOOT HALF OF THIS DESIGN.
+    # This was originally a `find` over /etc/kernel/postinst.d and /etc/apt/apt.conf.d, on the
+    # theory that the writer came from a package and the role could only discover whatever that
+    # package happened to ship. That theory was wrong: no such package exists on Debian (see
+    # defaults/main.yml), so the writer is now OURS and lives at two known paths. Checking two
+    # exact paths is strictly stronger than a glob -- a glob would have matched Debian's own
+    # unattended-upgrades apt conffile and passed while no sentinel writer existed at all.
+    - name: Stat the sentinel writer script
+      ansible.builtin.stat:
+        path: "{{ unattended_upgrades_sentinel_script }}"
+      register: unattended_upgrades_sentinel_script_stat
+      changed_when: false
+      tags: [verify]
+
+    - name: Stat the sentinel dpkg hook
+      ansible.builtin.stat:
+        path: "{{ unattended_upgrades_sentinel_hook }}"
+      register: unattended_upgrades_sentinel_hook_stat
+      changed_when: false
+      tags: [verify]
+
+    - name: Assert something on this node can write the reboot sentinel
+      # If this fails, kured is a no-op: it watches /var/run/reboot-required forever, the file is
+      # never created, no reboot is ever taken, and NOTHING GOES RED -- the DaemonSet stays 1/1
+      # Running and `kured_reboot_required` reports a steady 0 on every node. That is the exact
+      # shape of failure (a check whose failure mode is a green tick) this repo has been burned by
+      # before, so it is an assert and not a debug message.
+      ansible.builtin.assert:
+        that:
+          - unattended_upgrades_sentinel_script_stat.stat.exists
+          - unattended_upgrades_sentinel_script_stat.stat.executable | default(false)
+          - unattended_upgrades_sentinel_hook_stat.stat.exists
+        fail_msg: >-
+          NOTHING ON {{ inventory_hostname }} CAN CREATE /var/run/reboot-required.
+          {{ unattended_upgrades_sentinel_script }} (executable) and
+          {{ unattended_upgrades_sentinel_hook }} must both exist -- this role ships both, because
+          Debian provides no sentinel writer of its own (update-notifier-common is an UBUNTU
+          package and is not installable here). kured watches that path and reboots only when it
+          exists, so with no writer the reboot automation is permanently and INVISIBLY idle --
+          kured stays 1/1 Running and reports kured_reboot_required=0 on every node forever.
+          Re-run this role without `--tags verify` to install them.
+        success_msg: >-
+          Reboot sentinel writer present ({{ unattended_upgrades_sentinel_script }} + dpkg hook)
+          -- a kernel change will signal kured on the next apt transaction.
+      tags: [verify]
+
+    - name: Report the pending-reboot state of this node
+      # Read-only. Tells the operator whether this node is currently waiting on kured for a reboot,
+      # which is the question `--tags verify` is usually being run to answer.
+      ansible.builtin.stat:
+        path: /var/run/reboot-required
+      register: unattended_upgrades_reboot_required
+      changed_when: false
+      tags: [verify]
+
+    - name: Summary
+      ansible.builtin.debug:
+        msg: >-
+          {{ inventory_hostname }}: security-only unattended upgrades ACTIVE
+          (upgrade timer {{ unattended_upgrades_upgrade_time }} +/-{{ unattended_upgrades_randomised_delay }}),
+          kernel and microcode blacklisted, automatic reboot DISABLED.
+          Reboot currently pending: {{ unattended_upgrades_reboot_required.stat.exists }}.
+          {% if unattended_upgrades_reboot_required.stat.exists %}
+          This node is waiting for kured to reboot it in the next 03:00-06:00 window; it will be
+          cordoned and drained first, and no other node will be rebooted at the same time.
+          {% endif %}
+      tags: [verify]

infra/maintenance/quiesce.sh

@@ -0,0 +1,265 @@
+#!/usr/bin/env bash
+# =================================================================================================
+# quiesce.sh — bring the prod cluster to a safe standstill before powering nodes off, and bring it
+#              back afterwards. Companion to runbooks/cluster-maintenance-shutdown.md.
+# =================================================================================================
+# WHY THIS EXISTS. On 2026-10-17 all three nodes had to be powered off to move them onto a UPS, and
+# there was NO procedure for it: `infra/cluster.sh stop` is `virsh`-based and only ever applied to
+# the decommissioned dev cluster. The whole sequence was improvised live. This script is that
+# sequence, so the next person does not improvise it.
+#
+# WHY IT IS NOT JUST `shutdown -h`. Graceful node shutdown was unconfigured on every prod node that
+# day, so systemd stopped containerd and SIGKILLed every pod where it stood. roles/k3s_server now
+# sets shutdown-grace-period (+ the logind InhibitDelayMaxSec cap that actually enforces it), which
+# makes a bare reboot safe. This script is still the right tool for a PLANNED whole-cluster
+# shutdown, for three reasons the kubelet's drain does not cover:
+#   1. The UniFi VM needs an ACPI guest shutdown, not an eviction.
+#   2. CloudNativePG wants hibernation, not pod deletion -- and hibernation is IMPERATIVE state
+#      that Flux will not set or clear for you.
+#   3. Draining three nodes one at a time makes pods RESCHEDULE onto the survivors, churning
+#      Longhorn replicas for no reason when the whole cluster is going down anyway. Scaling in
+#      place avoids all of it.
+#
+# SAFE TO RE-RUN. Every step is idempotent and `resume` is the exact inverse of `quiesce`.
+#
+# WHAT IT DELIBERATELY DOES NOT DO: power anything off. Node poweroff is an operator action
+# (CLAUDE.md §5) and needs the become password; the runbook gives the ansible one-liner. It also
+# does not touch TrueNAS (not in the inventory, SSH refused -- web UI only) or the router.
+set -euo pipefail
+
+CONTEXT="${CONTEXT:-k3s-prod}"
+KUBECTL=(kubectl --context "$CONTEXT" --request-timeout=30s)
+TIMEOUT="${TIMEOUT:-600}"
+
+# The three CloudNativePG clusters, as namespace/name. Hibernation is per-Cluster.
+CNPG=(keycloak/keycloak-db nextcloud-db/nextcloud-db semaphore-db/semaphore-db)
+
+log()  { printf '\033[34m[quiesce]\033[0m %s\n' "$*"; }
+ok()   { printf '\033[32m[ ok ]\033[0m %s\n' "$*"; }
+warn() { printf '\033[33m[warn]\033[0m %s\n' "$*"; }
+die()  { printf '\033[31m[fail]\033[0m %s\n' "$*" >&2; exit 1; }
+
+need() { command -v "$1" >/dev/null || die "$1 not on PATH — run this inside ./infra/devcontainer/dev.sh"; }
+
+# ---------------------------------------------------------------- discovery
+# Every Deployment/StatefulSet whose pods mount a PVC — Longhorn OR the TrueNAS NFS bulk tier.
+# Computed from the LIVE cluster rather than hardcoded, so a newly onboarded service is covered
+# without anyone remembering to add it here. PROTECTED namespaces are excluded because scaling
+# them down would remove the very machinery that performs the detach (longhorn-system), keep the
+# API reachable (kube-system), or finish the hibernation and the VM shutdown (cnpg-system,
+# kubevirt).
+pvc_workloads() {
+  "${KUBECTL[@]}" get pods -A -o json | python3 -c '
+import json,sys
+PROTECT={"longhorn-system","kube-system","cilium-secrets","cnpg-system","kubevirt","flux-system","cdi"}
+out={}
+for p in json.load(sys.stdin)["items"]:
+    ns=p["metadata"]["namespace"]
+    if ns in PROTECT: continue
+    if not any(v.get("persistentVolumeClaim") for v in (p["spec"].get("volumes") or [])): continue
+    ow=(p["metadata"].get("ownerReferences") or [])
+    if not ow: continue
+    k,n=ow[0]["kind"],ow[0]["name"]
+    if k=="ReplicaSet": k,n="deployment",n.rsplit("-",1)[0]
+    elif k=="StatefulSet": k="statefulset"
+    else: continue          # Jobs, and CNPG Clusters, are handled separately
+    out[(ns,k,n)]=1
+for ns,k,n in sorted(out): print(ns,k,n)
+'
+}
+
+# ---------------------------------------------------------------- the PVC-deletion guard
+# *** DO NOT REMOVE THIS CHECK. IT EXISTS BECAUSE SCALING A StatefulSet DESTROYED DATA. ***
+# A StatefulSet may carry `persistentVolumeClaimRetentionPolicy.whenScaled: Delete`, and then
+# scaling it to 0 DELETES ITS PVCs -- and with them the Longhorn volumes. MEASURED 2026-10-17: the
+# quiesce for the UPS move scaled `monitoring/loki` (whenScaled: Delete) to 0 and its
+# `storage-loki-0` PVC and volume were deleted outright. Loki stores with `type: filesystem`, so
+# that PVC was the only copy; 72h retention bounded the loss to a rolling window, but nothing about
+# the mechanism is bounded -- the same action on a Delete-policy StatefulSet holding real state
+# would be unrecoverable. The volume count going 25 -> 24 was the only visible symptom, and it was
+# first misread as a miscount.
+#
+# So: such a StatefulSet is NEVER scaled. It is left RUNNING and its volume hard-detaches at
+# poweroff, which costs a replica rebuild on restart (degraded -> healthy, automatic). A rebuild is
+# an enormously better outcome than a deletion, and that is the whole trade being made here.
+sts_deletes_pvc_on_scale() { # ns name -> 0 if scaling it would delete its PVCs
+  local policy
+  policy=$("${KUBECTL[@]}" -n "$1" get statefulset "$2" \
+            -o jsonpath='{.spec.persistentVolumeClaimRetentionPolicy.whenScaled}' 2>/dev/null || true)
+  [ "$policy" = "Delete" ]
+}
+
+wait_for() { # wait_for <description> <shell-condition>
+  local desc="$1" cond="$2" waited=0
+  log "waiting: $desc (up to ${TIMEOUT}s)"
+  until eval "$cond"; do
+    (( waited += 10 )); (( waited >= TIMEOUT )) && { warn "TIMED OUT waiting: $desc"; return 1; }
+    sleep 10
+  done
+  ok "$desc"
+}
+
+# ---------------------------------------------------------------- quiesce
+do_quiesce() {
+  need kubectl; need python3
+
+  # Refuse to start if Longhorn is already unhealthy. Powering off on top of a degraded volume is
+  # how one replica becomes zero; better to stop here and let it rebuild first.
+  local unhealthy
+  unhealthy=$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io \
+    -o jsonpath='{range .items[*]}{.status.robustness}{"\n"}{end}' | grep -cvE '^(healthy|unknown)$' || true)
+  [ "$unhealthy" = "0" ] || die "$unhealthy Longhorn volume(s) are not healthy — fix or wait for rebuild BEFORE powering off"
+  ok "Longhorn: all volumes healthy"
+
+  # 1. Flux first, or it reverts every scale-down below within its reconcile interval.
+  log "suspending Flux Kustomizations"
+  local k
+  for k in $("${KUBECTL[@]}" -n flux-system get kustomizations -o name); do
+    "${KUBECTL[@]}" -n flux-system patch "${k}" --type=merge -p '{"spec":{"suspend":true}}' >/dev/null
+  done
+  ok "all Kustomizations suspended"
+
+  # 2. The VM needs an ACPI shutdown of the GUEST. KubeVirt does that on running:false; a plain
+  #    pod eviction would be a power cut to the VM.
+  if "${KUBECTL[@]}" -n unifi get vm unifi >/dev/null 2>&1; then
+    log "ACPI-stopping the UniFi VM"
+    "${KUBECTL[@]}" -n unifi patch vm unifi --type=merge -p '{"spec":{"running":false}}' >/dev/null
+  fi
+
+  # 3. Postgres: hibernation is a clean smart shutdown that KEEPS the PVCs. CNPG does one instance
+  #    at a time, primary last-ish, so this is not instant.
+  log "hibernating CloudNativePG clusters"
+  local c ns name
+  for c in "${CNPG[@]}"; do
+    ns="${c%%/*}"; name="${c##*/}"
+    "${KUBECTL[@]}" -n "$ns" annotate cluster.postgresql.cnpg.io "$name" \
+      cnpg.io/hibernation=on --overwrite >/dev/null 2>&1 || warn "could not hibernate $c"
+  done
+
+  # 4. Everything holding a PVC, scaled in place. NFS consumers first so nothing holds a hard
+  #    TrueNAS mount; PiHole last because it is the LAN resolver.
+  log "scaling down PVC-holding workloads"
+  local line ns2 kind nm deferred=() skipped_delete_policy=()
+  while read -r ns2 kind nm; do
+    [ -z "${ns2:-}" ] && continue
+    if [ "$kind" = "statefulset" ] && sts_deletes_pvc_on_scale "$ns2" "$nm"; then
+      warn "SKIPPING $ns2/$nm — whenScaled: Delete would DELETE its PVC (see the guard above)."
+      warn "  Left running; its volume hard-detaches at poweroff and rebuilds on restart."
+      skipped_delete_policy+=("$ns2/$nm")
+      continue
+    fi
+    if [ "$ns2" = "pihole" ]; then deferred+=("$ns2 $kind $nm"); continue; fi
+    "${KUBECTL[@]}" -n "$ns2" scale "$kind" "$nm" --replicas=0 >/dev/null 2>&1 \
+      && printf '    scaled 0  %s/%s %s\n' "$ns2" "$kind" "$nm" \
+      || warn "could not scale $ns2/$kind/$nm"
+  done < <(pvc_workloads)
+  for line in "${deferred[@]:-}"; do
+    [ -z "${line:-}" ] && continue
+    # shellcheck disable=SC2086
+    set -- $line
+    "${KUBECTL[@]}" -n "$1" scale "$2" "$3" --replicas=0 >/dev/null 2>&1 \
+      && printf '    scaled 0  %s/%s %s  (LAN DNS — expect name resolution to stop)\n' "$1" "$2" "$3"
+  done
+
+  # 5. The only assertion that matters. An attached volume at poweroff comes back degraded and
+  #    rebuilds; this is what we are buying with all of the above.
+  wait_for "UniFi VMI gone" \
+    '[ "$('"${KUBECTL[*]}"' -n unifi get vmi --no-headers 2>/dev/null | wc -l)" = "0" ]' || true
+  wait_for "all Postgres instances stopped" \
+    '[ "$('"${KUBECTL[*]}"' get pods -A --no-headers 2>/dev/null | grep -cE "(keycloak|nextcloud|semaphore)-db-[0-9]")" = "0" ]' || true
+  wait_for "all Longhorn volumes detached" \
+    '[ "$('"${KUBECTL[*]}"' -n longhorn-system get volumes.longhorn.io -o jsonpath="{range .items[*]}{.status.state}{\"\n\"}{end}" 2>/dev/null | grep -c attached)" = "0" ]' || \
+      warn "some volumes are STILL ATTACHED — they will come back degraded and rebuild; acceptable, but note which"
+
+  if [ "${#skipped_delete_policy[@]}" -gt 0 ]; then
+    echo
+    warn "NOT scaled down (whenScaled: Delete — scaling them would delete their PVCs):"
+    printf '      %s\n' "${skipped_delete_policy[@]}"
+    warn "Expect these volumes to come back 'degraded' and rebuild. That is correct and automatic."
+  fi
+  echo
+  ok "QUIESCED. Safe to power off."
+  cat <<'NEXT'
+
+  Next, from inside the devcontainer (it holds the become password):
+    cd /work/infra/ansible && ansible -i inventory/hosts.ini k3s_prod -b \
+      -m shell -a 'shutdown -h +1 "maintenance"'
+
+  THEN, in this order and not before:
+    1. TrueNAS (192.168.2.30) — web UI only; not in the ansible inventory and SSH is refused.
+    2. Router / switch last.
+NEXT
+}
+
+# ---------------------------------------------------------------- resume
+do_resume() {
+  need kubectl
+
+  # THE STEP THAT IS ALWAYS FORGOTTEN. Hibernation is imperative state: Flux neither sets nor
+  # clears it, so resuming Flux alone leaves all three databases down and every app that needs
+  # one looking broken for reasons that point nowhere near Postgres.
+  log "un-hibernating CloudNativePG clusters"
+  local c ns name
+  for c in "${CNPG[@]}"; do
+    ns="${c%%/*}"; name="${c##*/}"
+    "${KUBECTL[@]}" -n "$ns" annotate cluster.postgresql.cnpg.io "$name" \
+      cnpg.io/hibernation=off --overwrite >/dev/null 2>&1 || warn "could not un-hibernate $c"
+  done
+
+  # Flux restores every replica count AND the VM's running:true from git, so nothing here needs to
+  # undo the scale-downs by hand.
+  log "resuming Flux Kustomizations"
+  local k
+  for k in $("${KUBECTL[@]}" -n flux-system get kustomizations -o name); do
+    "${KUBECTL[@]}" -n flux-system patch "${k}" --type=merge -p '{"spec":{"suspend":false}}' >/dev/null
+  done
+  ok "all Kustomizations resumed"
+
+  wait_for "Flux fully reconciled" \
+    '[ "$('"${KUBECTL[*]}"' -n flux-system get kustomizations --no-headers 2>/dev/null | grep -vc True)" = "0" ]' || \
+      warn "some Kustomizations are not Ready — the graph serialises behind platform-config, so give it a few minutes before worrying"
+
+  echo
+  ok "RESUMED."
+  cat <<'NEXT'
+
+  Known cold-start race, not a fault in this script: oauth2-proxy performs OIDC discovery at
+  startup and exits 1 on the 503 it gets while Keycloak is still coming up, then sits in
+  exponential CrashLoopBackOff. If a replica is stuck, delete the pod once:
+    kubectl --context k3s-prod -n oauth2-proxy delete pod <name>
+NEXT
+}
+
+do_status() {
+  need kubectl
+  printf 'nodes:      %s Ready\n' "$("${KUBECTL[@]}" get nodes --no-headers | grep -c ' Ready')"
+  printf 'flux:       %s/%s Ready\n' \
+    "$("${KUBECTL[@]}" -n flux-system get kustomizations --no-headers | grep -c True)" \
+    "$("${KUBECTL[@]}" -n flux-system get kustomizations --no-headers | wc -l)"
+  printf 'longhorn:   %s attached, %s detached, %s not healthy\n' \
+    "$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.state}{"\n"}{end}' | grep -c attached)" \
+    "$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.state}{"\n"}{end}' | grep -c detached)" \
+    "$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.robustness}{"\n"}{end}' | grep -cvE '^(healthy|unknown)$' || true)"
+  printf 'postgres:   %s instance pod(s)\n' \
+    "$("${KUBECTL[@]}" get pods -A --no-headers | grep -cE '(keycloak|nextcloud|semaphore)-db-[0-9]' || true)"
+  printf 'unifi VM:   %s VMI\n' "$("${KUBECTL[@]}" -n unifi get vmi --no-headers 2>/dev/null | wc -l)"
+}
+
+case "${1:-}" in
+  quiesce) do_quiesce ;;
+  resume)  do_resume ;;
+  status)  do_status ;;
+  *) cat >&2 <<USAGE
+usage: $0 {quiesce|resume|status}
+
+  quiesce  suspend Flux, ACPI-stop the UniFi VM, hibernate Postgres, scale down every
+           PVC-holding workload, and wait for all Longhorn volumes to detach.
+           Does NOT power anything off — see runbooks/cluster-maintenance-shutdown.md.
+  resume   the exact inverse: un-hibernate Postgres, resume Flux, wait for reconcile.
+  status   one-screen readout of nodes / Flux / Longhorn / Postgres / VM.
+
+Run it inside the pinned toolbox, which supplies kubectl and the context:
+  ./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh status'
+USAGE
+     exit 2 ;;
+esac

infra/vpn-ingress/new-peer.sh

@@ -3,7 +3,7 @@
 # new-peer.sh — enrol one device on the remote-access VPN (platform/vpn-ingress).
 #
 # Generates a WireGuard keypair, renders the device's client profile, prints a QR code for a phone,
-# and prints the one `[Peer]` stanza to paste into platform/vpn-ingress/peers.yaml. The device's
+# and prints the one `[Peer]` stanza to paste into platform/vpn-ingress/peers.conf. The device's
 # private half is written to ONE 0600 file and nowhere else: not sent to the cluster, not committed,
 # never logged.
 #
@@ -62,7 +62,7 @@ set -euo pipefail
 die() { printf 'new-peer.sh: %s\n' "$*" >&2; exit 1; }
 
 REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
-PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.yaml"
+PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.conf"
 
 # --- defaults, each mirroring a value that is DECLARED elsewhere in this repo -------------------
 # These are duplicated here rather than parsed out of the YAML on purpose: this script is meant to
@@ -133,11 +133,22 @@ CLIENT_ADDR="${CLIENT_BASE}.${OCTET}"
 # --- refuse a collision, because the failure is otherwise invisible -----------------------------
 # Two devices sharing an address does not error anywhere: the server routes that address to
 # whichever peer handshook most recently, so BOTH devices appear to work intermittently.
-if [ -r "$PEERS_FILE" ] && grep -q "AllowedIPs *= *${CLIENT_ADDR}/32" "$PEERS_FILE"; then
-  die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.yaml. Pick another octet."
+# COMMENT LINES ARE NOT ALLOCATIONS. This used to grep the whole file and so matched the
+# COMMENTED-OUT EXAMPLE STANZA in the register, which made `.2` — the first address anyone would
+# ask for — permanently un-allocatable with a message claiming it was already taken. Strip
+# comments first: a real stanza's `AllowedIPs` line is never commented, while both the template
+# and every peer's `# <name> — <owner>` label are.
+# FAIL if the register is missing rather than skipping the check. The path was silently wrong once
+# (it still said peers.yaml after the file became peers.conf), and because the old guard was
+# `[ -r "$PEERS_FILE" ] && ...` the only symptom would have been duplicate detection quietly
+# never firing again — the exact silent-skip shape this repo treats as a defect.
+[ -r "$PEERS_FILE" ] || die "cannot read the peer register '$PEERS_FILE'.
+  Duplicate-address detection cannot run, so this script refuses to issue a profile."
+if grep -vE '^[[:space:]]*#' "$PEERS_FILE" | grep -q "AllowedIPs *= *${CLIENT_ADDR}/32"; then
+  die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.conf. Pick another octet."
 fi
-if [ -r "$PEERS_FILE" ] && grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then
-  printf 'new-peer.sh: WARNING a peer comment already mentions "%s" in peers.yaml.\n' "$NAME" >&2
+if grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then
+  printf 'new-peer.sh: WARNING a peer comment already mentions "%s" in peers.conf.\n' "$NAME" >&2
 fi
 
 # --- output location: 0700, and never inside the repo -------------------------------------------
@@ -204,7 +215,7 @@ cat <<EOF
   Profile     : ${PROFILE}   (0600; contains the private key)
 
   ------------------------------------------------------------------------------------------
-  STEP 1 — add this to platform/vpn-ingress/peers.yaml, under data.peers.conf, then PR it:
+  STEP 1 — append this to platform/vpn-ingress/peers.conf, then PR it:
 
     # ${NAME} — <owner>, enrolled $(date -u +%Y-%m-%d)
     [Peer]
@@ -223,7 +234,7 @@ cat <<EOF
   STEP 3 — delete the profile from this machine once the device has it:
     shred -u ${PROFILE} 2>/dev/null || rm -f ${PROFILE}
 
-  The tunnel will not work until the peers.yaml PR is MERGED and Flux has rolled the DaemonSet
+  The tunnel will not work until the peers.conf PR is MERGED and Flux has rolled the DaemonSet
   (~1 min). Until then the device handshakes and gets silence, exactly like an unknown key.
   ------------------------------------------------------------------------------------------
 EOF

platform/coredns/hpa.yaml

@@ -0,0 +1,112 @@
+---
+# =================================================================================================
+# coredns HA floor — hold in-cluster DNS at >= 2 replicas, which k3s does not.
+# =================================================================================================
+# MEASURED on prod 2026-10-18: `kube-system/coredns` ran with **replicas: 1**. In-cluster DNS is a
+# single pod on a single node, so **draining or rebooting that one node is a cluster-wide
+# in-cluster DNS outage** for as long as the pod takes to reschedule and become ready — every pod
+# that looks up a Service, every CNPG replica finding its primary, every operator talking to the
+# API by name. It is the single cheapest availability defect in the cluster, and it was invisible
+# because `1/1 Ready` looks perfectly healthy.
+#
+# This matters more now than it did: the reboot automation (kured + unattended-upgrades) drains and
+# reboots one node at a time on a schedule, which turns a latent single point of failure into a
+# recurring one, unattended, in the middle of the night.
+#
+# -------------------------------------------------------------------------------------------------
+# WHY AN HPA, AND NOT JUST `replicas: 2` — this is the whole decision
+# -------------------------------------------------------------------------------------------------
+# **k3s owns the coredns Deployment and we cannot durably edit it.** It is applied by k3s's own
+# deploy controller from a bundled manifest, not by Flux:
+#
+#   $ kubectl -n kube-system get addon coredns
+#   NAME      SOURCE                                        CHECKSUM
+#   coredns   /data/k3s/server/manifests/coredns.yaml        8cd91595...
+#   $ kubectl -n kube-system get deploy coredns -o jsonpath='{.metadata.managedFields[*].manager}'
+#   deploy@k3s-prod-1 k3s
+#
+# and the Deployment carries an `objectset.rio.cattle.io/hash` label — a wrangler objectset. k3s
+# RE-STAGES its bundled manifests on startup, so a `replicas: 2` written into that file (or a
+# `kubectl scale`) is reverted the next time k3s restarts or is upgraded. MEASURED the same day: a
+# hand `kubectl scale` to 2 was NOT actively reverted (it held, and the AddOn checksum did not
+# change) — the deploy controller re-applies on manifest change and on restart, not on a drift
+# timer. So the hand fix survives until exactly the event we are protecting against: a reboot.
+#
+# The two durable alternatives were weighed and rejected:
+#   * **`coredns.yaml.skip` + our own templated copy** (the pattern roles/kube_vip already uses for
+#     a manifest in server/manifests). Fully deterministic and never flaps — but it takes OWNERSHIP
+#     of coredns away from k3s permanently, so k3s upgrades would stop updating it and the version
+#     would have to be refreshed by hand forever. Too high a price for a replica count.
+#   * **A Flux-managed patch of the Deployment itself.** Rejected outright: Flux and k3s's deploy
+#     controller would both own one object and fight continuously. That is the shape of the
+#     `default-deny` NetworkPolicy drift loop this cluster already suffered (Kyverno generate + Flux
+#     both owning one resource, ~2500 events per 10 minutes).
+#
+# An HPA avoids both. It is a SEPARATE object that k3s does not own and will never reconcile away,
+# and it drives the Deployment's `/scale` subresource rather than its spec — so there is no
+# shared-ownership fight. If k3s re-stages `replicas: 1`, the HPA observes it under `minReplicas`
+# and scales back to 2 within one reconcile (~15s by default).
+#
+# THE COST, STATED PLAINLY: there is a brief window — seconds, after a k3s restart — where coredns
+# is back at 1 replica until the HPA notices. That is strictly better than the status quo, where it
+# is at 1 replica permanently, but it is not zero. Accepting that window is the trade being made
+# here; the `.skip` option above is what to reach for if it ever proves unacceptable.
+#
+# -------------------------------------------------------------------------------------------------
+# THIS IS A FLOOR-HOLDER, NOT A REAL AUTOSCALER
+# -------------------------------------------------------------------------------------------------
+# `minReplicas: 2` is the entire point. The CPU metric exists because an autoscaling/v2 HPA needs a
+# metric to be a valid object, not because coredns is expected to scale on load: measured via
+# metrics-server, the two pods use **5m and 4m** against a **100m** request — ~5% utilisation, so
+# the 80% target will never be approached and the HPA will sit at `minReplicas` indefinitely.
+#
+# The CPU request is what makes this legal: coredns ships `requests: {cpu: 100m, memory: 70Mi}`
+# (measured), and a Resource/Utilization metric is meaningless without a request. metrics-server is
+# installed as a k3s addon and verified serving (`kubectl top pod -l k8s-app=kube-dns` returns
+# figures), so the metric resolves rather than leaving the HPA with `ScalingActive=False`.
+#
+# `maxReplicas: 3` = the node count. coredns' pod anti-affinity prefers one pod per node, so a
+# fourth replica would add no failure-domain coverage; the ceiling exists so a metrics glitch can
+# never balloon DNS pods across the cluster.
+apiVersion: autoscaling/v2
+kind: HorizontalPodAutoscaler
+metadata:
+  name: coredns
+  # kube-system, because that is where k3s puts coredns. This is the only object this component
+  # places outside a namespace it owns, and it is additive: it creates nothing in kube-system and
+  # mutates only the coredns Deployment's scale subresource.
+  namespace: kube-system
+  labels:
+    app.kubernetes.io/name: coredns
+spec:
+  scaleTargetRef:
+    apiVersion: apps/v1
+    kind: Deployment
+    name: coredns
+  minReplicas: 2
+  maxReplicas: 3
+  metrics:
+    - type: Resource
+      resource:
+        name: cpu
+        target:
+          type: Utilization
+          # Deliberately high. See "floor-holder" above: this must not scale on normal DNS load.
+          averageUtilization: 80
+  behavior:
+    scaleDown:
+      # 10 minutes. DNS is the worst thing in the cluster to flap: a scale-down races in-flight
+      # queries and every client's negative cache. The HPA should be reluctant in this direction.
+      stabilizationWindowSeconds: 600
+      policies:
+        - type: Pods
+          value: 1
+          periodSeconds: 600
+    scaleUp:
+      # Prompt in the direction that restores redundancy — this is the path taken after a k3s
+      # restart has re-staged `replicas: 1`.
+      stabilizationWindowSeconds: 0
+      policies:
+        - type: Pods
+          value: 2
+          periodSeconds: 60

platform/coredns/kustomization.yaml

@@ -7,3 +7,7 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
   - coredns-custom.yaml
+  # HA floor for the coredns Deployment itself — a separate concern from the split-horizon server
+  # block above, and deliberately an HPA rather than a replica edit: k3s owns that Deployment via
+  # its own deploy controller and re-stages `replicas: 1` on restart. hpa.yaml has the reasoning.
+  - hpa.yaml

platform/keycloak/README.md

@@ -154,6 +154,74 @@ NEW client, a new protocol mapper or a changed scope still needs the realm impor
 (it will not on its own) or a deliberate admin-API/`kcadm` change. `Done=True` is never evidence that
 a realm edit landed.
 
+## Passkeys (WebAuthn Passwordless) — DECLARED IN GIT, NOT YET IN THE REALM (2026-10-17)
+
+`keycloak-realm-import.yaml` now declares passkey login: the realm's **WebAuthn Passwordless**
+policy (`webAuthnPolicyPasswordless*`), and a `browser-mfa` flow that offers a passkey as an
+**ALTERNATIVE** beside the password form. Design decisions and their reasoning are in the manifest
+comments, not repeated here. The headline choices:
+
+| Decision | Value | One-line reason |
+|---|---|---|
+| Additive or replacing? | **Additive** — passkey *or* password | A misfiring mandatory flow locks the single operator out of every SSO app *and* kubectl |
+| Enrolment | **Optional**, never a default action | Same reason; `webauthn-register-passwordless` is already enabled-not-forced by default |
+| Relying-party ID | `${BASE_DOMAIN}` (not `keycloak.${BASE_DOMAIN}`) | An unre-keyable credential must not be bound to a hostname that can move |
+| User verification | `required` | Without it, mere possession of an unlocked device is a complete login |
+| Resident key | `Yes` | A non-discoverable credential is a second factor, not a passkey |
+| OTP after a passkey? | **No** — conditional OTP stays on the password branch | A UV-required passkey is already two factors, and a phishing-resistant pair at that |
+
+### This change does NOT reach the live realm. At all.
+
+Same one-shot failure this file already documents twice above, third field over. Measured on prod
+2026-10-17:
+
+```
+$ kubectl --context k3s-prod -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
+{"conditions":[{"status":"True","type":"Done"},{"status":"False","type":"Started"},
+               {"status":"False","type":"HasErrors"}]}
+```
+
+`Done=True` / `Started=False` — the import has completed and **will not re-run**, so a policy block
+and three new sub-flows added to this file change nothing about how anyone logs in. The
+client-secret/client-URL reconciler does **not** close this gap and was never meant to: its hard
+endpoint allow-list makes `/authentication` unreachable **by design** (see the table above), and
+realm-level policy fields are not client fields. This is exactly the "a NEW client, a new protocol
+mapper or a changed scope still needs the realm import to actually re-run" case named above.
+
+**So the manifest is the declared source of truth and a DR-rebuild input, and nothing more, until
+an operator applies it by hand.** Two paths, both operator actions:
+
+1. **Admin console (expected path).** *Authentication → Policies → WebAuthn Passwordless Policy*
+   for the policy block, then *Authentication → Flows → browser-mfa* to rebuild the executions to
+   match the manifest. The flow edit is the fiddly half: split the existing Username Password Form
+   into Username Form + Password Form and add the two sub-flows exactly as declared.
+2. **A deliberate re-import.** Deleting and recreating the `KeycloakRealmImport` re-runs it against
+   the **existing** realm. Do not reach for this casually — it is the operation whose failure mode
+   is the 2026-09-06 total SSO outage documented above, and it has not been rehearsed here.
+
+Either way the realm is then configured by hand and git is only the record of intent, which is a
+live exception to R-IDP-03 (config as code) — not a new one, but one more instance of it.
+
+### Verifying it actually works (nobody has, as of 2026-10-17)
+
+No passkey has been enrolled and no browser has been pointed at this. The checks that would prove
+it, in order:
+
+```sh
+# 1. the policy is live (RP ID must read as the base domain, NOT the keycloak host, NOT a URL)
+#    Authentication -> Policies -> WebAuthn Passwordless Policy, in the admin console.
+# 2. enrol: account console (https://keycloak.${BASE_DOMAIN}/realms/homelab/account) -> Signing in
+#    -> Passwordless -> set up. A browser/OS passkey prompt must appear; if it does not, the RP ID
+#    is wrong for the origin and NOTHING will say so.
+# 3. log in: a fresh private window to any SSO-gated app. Expect username screen, then a passkey
+#    prompt. A platform-admin must NOT be asked for TOTP on this path.
+# 4. regression (the important one): a user with NO passkey must still get the password screen,
+#    and a platform-admin must still be asked for TOTP there.
+```
+
+A failed passkey ceremony is usually silent in the server log — the browser refuses before Keycloak
+is involved — so step 2 failing with no log entry is the expected signature of a wrong RP ID.
+
 ## Exceptions that keep native auth (do not force SSO) — ADR-0014
 
 Plex/Seerr (Plex auth), Minecraft (Mojang/Microsoft), UniFi (native).

platform/keycloak/keycloak-realm-import.yaml

@@ -74,6 +74,142 @@ spec:
     bruteForceProtected: true
     permanentLockout: false
     sslRequired: external
+    # ===========================================================================================
+    # PASSKEYS — the WebAuthn PASSWORDLESS policy (R-IDP-04). 2026-10-17.
+    # ===========================================================================================
+    # Read this together with the browser-mfa flow at the bottom of the file: THIS block decides
+    # what a passkey must BE, the FLOW decides when one is OFFERED. Keycloak keeps TWO independent
+    # WebAuthn policies per realm and they are not interchangeable:
+    #
+    #   webAuthnPolicy*             -> the TWO-FACTOR policy, consumed by the `webauthn-authenticator`
+    #                                  authenticator (a security key presented AFTER a password).
+    #   webAuthnPolicyPasswordless* -> the PASSWORDLESS policy, consumed by
+    #                                  `webauthn-authenticator-passwordless` (a passkey presented
+    #                                  INSTEAD of a password). This is the one passkeys use.
+    #
+    # Setting the wrong family is a silent no-op: the passwordless authenticator reads
+    # getWebAuthnPolicyPasswordless() only (WebAuthnPasswordlessAuthenticator.getWebAuthnPolicy),
+    # so a policy written into the two-factor fields is simply never consulted by a passkey login.
+    # Only the passwordless family is declared here; the two-factor family is deliberately left at
+    # Keycloak's defaults because nothing in this realm uses `webauthn-authenticator` — TOTP
+    # (auth-otp-form) is the second factor for platform-admins, see the flow section.
+    #
+    # ADDITIVE AND OPTIONAL — THE DECISION THIS BLOCK EXISTS TO MAKE
+    # -------------------------------------------------------------
+    # Passkeys are offered ALONGSIDE the password, never instead of it, and enrolment is NOT forced.
+    # This is a single-operator homelab whose IdP gates Grafana, Gitea, Nextcloud, Headlamp,
+    # Semaphore, PiHole, Crafty, the *arr stack AND kubectl itself (the `kubernetes` client below).
+    # A mandatory passkey flow that misfires is therefore not an inconvenience, it is a total
+    # lock-out from the whole platform AND from the cluster API in one step — and the only repair
+    # paths left would be the Keycloak bootstrap admin and the break-glass cert kubeconfig. The
+    # expected gain from forcing enrolment (one human stops typing a password) is small; the
+    # expected loss is the platform. So: additive, optional, and password login keeps working for
+    # every user who never enrols.
+    #
+    # Concretely that means: no `requiredActions` block is declared anywhere in this file (see the
+    # long note in the flow section for why declaring one would be actively DANGEROUS), and the
+    # passkey step sits as an ALTERNATIVE beside the password form rather than ahead of it.
+    #
+    # RELYING-PARTY ID — THE FIELD THAT SILENTLY BREAKS EVERY PASSKEY IF IT IS WRONG
+    # ------------------------------------------------------------------------------
+    # The RP ID is baked into every credential at enrolment and can never be changed afterwards:
+    # a passkey created under one RP ID is unusable under another, with no error the user can act
+    # on — the browser simply reports that no credential is available. It is also browser-enforced,
+    # so it cannot be "fixed" server-side: per WebAuthn, the RP ID must be equal to, or a
+    # registrable-domain suffix of, the origin's effective domain. Keycloak is served at
+    # keycloak.<BASE_DOMAIN> (httproute.yaml, and the Keycloak CR's hostname), so the legal choices
+    # are keycloak.<BASE_DOMAIN>, <BASE_DOMAIN>, or the registrable domain one level up. It is NOT
+    # the full URL and NOT a scheme — "https://..." here would fail every ceremony.
+    #
+    # We choose BASE_DOMAIN, the shared platform suffix, NOT the keycloak host, for one concrete
+    # reason drawn from this repo's own history: a credential that cannot be re-keyed must not be
+    # bound to a name that can move. The oauth2-proxy client below ALREADY declares a redirect URI
+    # on auth.<BASE_DOMAIN>, i.e. this platform already contemplates serving auth on a second
+    # hostname; and the client-URL reconciler exists precisely because moving a hostname in git
+    # silently fails to reach the realm (see README, "Client URLs: the same one-shot failure").
+    # Binding passkeys to the keycloak subdomain would mean that relocating Keycloak invalidates
+    # every enrolled passkey; binding them to BASE_DOMAIN survives it. The cost is scope: any
+    # future WebAuthn relying party under this base domain could assert the same RP ID. In a
+    # homelab where every such party is already behind THIS IdP that is not a trust boundary worth
+    # buying. Note the residual, unavoidable risk: changing BASE_DOMAIN itself invalidates every
+    # passkey, which is a re-enrolment, not a repair.
+    #
+    # Written as the cluster-var so it is substituted by the same Flux postBuild pass, with the same
+    # value, as every hostname in this file — there is no second place the domain is decided.
+    webAuthnPolicyPasswordlessRpId: ${BASE_DOMAIN}
+    # The human-readable name the browser/OS shows in the passkey prompt and stores beside the
+    # credential ("Save a passkey for Homelab SSO?"). Matches the realm displayName above so the
+    # OS keychain entry is recognisable. It is also load-bearing on the UPDATE path: Keycloak's
+    # updateRealm applies the passwordless policy ONLY when this field is non-empty
+    # (DefaultExportImportManager line ~842), so an admin-API realm update that omits it drops the
+    # whole policy silently.
+    webAuthnPolicyPasswordlessRpEntityName: "Homelab SSO"
+    # ES256 then RS256 — exactly Keycloak's own DEFAULT_POLICY, declared explicitly rather than
+    # inherited so the realm's crypto is auditable from git. ES256 is what every platform
+    # authenticator (Touch ID/Face ID, Windows Hello, Android) and every FIDO2 key produces;
+    # RS256 is the fallback for older Windows Hello TPM-backed credentials. EdDSA/Ed25519 is
+    # deliberately NOT offered: authenticator and browser support is uneven, and an authenticator
+    # that picks an algorithm the verifier handles badly fails at enrolment, which is the worst
+    # moment to discover it. Order is a preference list, not an allow-list ordering requirement.
+    webAuthnPolicyPasswordlessSignatureAlgorithms:
+      - ES256
+      - RS256
+    # DISCOVERABLE CREDENTIAL REQUIRED. "Yes" is a Keycloak sentinel, not a boolean, and the login
+    # theme compares it as the literal string: webauthnRegister.js maps 'Yes' to
+    # authenticatorSelection.requireResidentKey=true and ANY other non-sentinel value to false.
+    # This is not optional for passkeys — a non-discoverable credential cannot be found by the
+    # browser without the server first naming it, so it is a second factor, not a passkey.
+    # (Keycloak 26.0.6 emits only the LEGACY requireResidentKey member, not the newer
+    # residentKey:"required"; per WebAuthn L2 the client maps requireResidentKey=true onto
+    # residentKey="required", so discoverable credentials are genuinely requested — but it also
+    # means this policy cannot express residentKey:"preferred" at all.)
+    webAuthnPolicyPasswordlessRequireResidentKey: "Yes"
+    # USER VERIFICATION REQUIRED. The authenticator must verify the human (biometric or PIN) and
+    # say so in the assertion. This is the field that makes a passkey login at least as strong as
+    # the password it replaces: with verification merely "preferred", possession of an unlocked
+    # phone or an unattended security key would be a COMPLETE login to every SSO-gated app — a
+    # single-factor possession check, strictly weaker than the password+TOTP path beside it.
+    # Required makes the passkey branch inherently two-factor (something you have + something you
+    # are/know), which is the premise the flow section relies on when it does NOT ask a
+    # passkey-authenticated platform-admin for TOTP as well.
+    webAuthnPolicyPasswordlessUserVerificationRequirement: required
+    # "not specified" is Keycloak's sentinel for "either kind of authenticator"
+    # (Constants.DEFAULT_WEBAUTHN_POLICY_NOT_SPECIFIED; the theme JS omits the WebAuthn member
+    # entirely when it sees this exact string). Deliberately NOT narrowed: "platform" would reject
+    # every YubiKey and every phone used as a cross-device passkey, and "cross-platform" would
+    # reject Touch ID and Windows Hello — i.e. either choice silently halves the usable hardware.
+    # The operator should be able to enrol a laptop AND a phone AND a hardware key.
+    webAuthnPolicyPasswordlessAuthenticatorAttachment: "not specified"
+    # ATTESTATION: none. Requesting "direct" would hand Keycloak an attestation statement it has
+    # no trust anchors to validate here (no metadata service is configured) and nothing to compare
+    # it against, since the AAGUID allow-list below is deliberately empty — so the statement would
+    # be collected, unverifiable, and would add an extra "share model information?" consent prompt
+    # on Safari/iOS for no security gain. Attestation is how you ENFORCE an approved-hardware list;
+    # we do not have one, so asking for it is cost without benefit.
+    webAuthnPolicyPasswordlessAttestationConveyancePreference: none
+    # Empty = accept any authenticator model. An AAGUID allow-list is a hardware whitelist: it
+    # would have to be maintained by hand, and every new phone, laptop or key the operator buys
+    # would be rejected at enrolment with a message that does not explain itself. It is also moot
+    # while attestation is "none" (above) — with no attestation statement there is no trustworthy
+    # AAGUID to match against in the first place.
+    webAuthnPolicyPasswordlessAcceptableAaguids: []
+    # Ceremony timeout in SECONDS (the login theme multiplies by 1000; 0 means "send no timeout and
+    # let the browser decide"). 300s is chosen over a tighter value because the slowest LEGITIMATE
+    # path is the cross-device hybrid flow — Keycloak renders a QR code, the operator unlocks a
+    # phone, scans it, and completes a biometric check — which routinely takes over a minute. A
+    # 60-120s timeout looks tidy and converts that flow into an unexplained failure. Browsers clamp
+    # this value (Chrome to at most 10 minutes), so 300 is honoured rather than reinterpreted.
+    webAuthnPolicyPasswordlessCreateTimeout: 300
+    # Send the user's existing passkeys as excludeCredentials at enrolment, so re-enrolling an
+    # authenticator the user already registered is refused by the authenticator itself instead of
+    # producing a second, indistinguishable credential in the account console. Cheap, and it keeps
+    # "which passkey is which" answerable — which matters when the recovery story is "delete the
+    # lost one".
+    webAuthnPolicyPasswordlessAvoidSameAuthenticatorRegister: true
+    # webAuthnPolicyPasswordlessExtraOrigins is intentionally NOT set. It exists to admit non-web
+    # origins (native app facet IDs, e.g. android:apk-key-hash:...). Every ceremony here happens in
+    # a browser on Keycloak's own origin, which is already covered by the RP ID above; adding
+    # origins is only ever a widening of who may ask for these credentials.
     # --- Authorization model (R-IDP-06): groups drive access; platform-admins carry a realm role. ---
     roles:
       realm:
@@ -528,14 +664,105 @@ spec:
               "access.token.claim": "true"
               "userinfo.token.claim": "true"
               "claim.name": groups
-    # --- MFA for platform-admins (R-IDP-04): a browser flow whose OTP step is CONDITIONAL on the
-    #     platform-admin role. Non-admins log in without OTP; platform-admins must enroll/present TOTP.
-    #     Set as the realm browserFlow below. VERIFY at bring-up (see runbook): a malformed flow fails
-    #     the whole import — if so, the fallback is to comment this block + browserFlow and configure
-    #     the conditional-OTP-by-role flow via the admin console. ---
+    # ===========================================================================================
+    # MFA for platform-admins (R-IDP-04) + PASSKEY login (2026-10-17). The realm browserFlow is
+    # bound to browser-mfa at the bottom of this file.
+    # ===========================================================================================
+    # Non-admins log in without OTP; platform-admins must enroll/present TOTP. VERIFY at bring-up
+    # (see runbook): a malformed flow fails the whole import — if so, the fallback is to comment
+    # this block + browserFlow and configure the flow via the admin console.
+    #
+    # THE SHAPE, AND WHY IT IS THIS SHAPE
+    # -----------------------------------
+    #   browser-mfa                           (top level, bound as browserFlow)
+    #     auth-cookie                     ALTERNATIVE   already-authenticated SSO cookie
+    #     identity-provider-redirector    ALTERNATIVE   (no IdPs configured; kept for parity)
+    #     browser-mfa-forms               ALTERNATIVE   the interactive path:
+    #       auth-username-form              REQUIRED      WHO are you (no password yet)
+    #       browser-mfa-passkey-or-password REQUIRED      ...and PROVE it, one of two ways:
+    #         webauthn-authenticator-passwordless  ALTERNATIVE   a passkey, or
+    #         browser-mfa-password-and-otp         ALTERNATIVE   the password path:
+    #           auth-password-form                   REQUIRED      password, then
+    #           browser-mfa-conditional-otp          CONDITIONAL   TOTP iff platform-admin
+    #
+    # WHY THE USERNAME/PASSWORD SCREEN HAD TO BE SPLIT IN TWO. The previous flow used the combined
+    # `auth-username-password-form`, which collects both on one page. A passkey branch cannot live
+    # behind that: the server must know WHICH user is logging in before it can decide whether a
+    # passkey is even available, and the combined form has already taken the password by the time
+    # it knows. So it is replaced by the `auth-username-form` / `auth-password-form` pair — the
+    # same decomposition Keycloak's own passwordless recipe uses. THIS IS A VISIBLE UX CHANGE FOR
+    # EVERY USER, passkey or not: login becomes two screens (username, then password) instead of
+    # one. That cost is accepted deliberately — it is the enabling change, it is what Google and
+    # Microsoft logins already do, and the alternative (below) is worse.
+    #
+    # WHY NOT USERNAMELESS ("just tap your passkey", no username screen at all). Keycloak 26.0.6
+    # does support it: WebAuthnPasswordlessAuthenticator.requiresUser() returns FALSE, so the
+    # authenticator can legally run as a first step with no user in context. We deliberately do not
+    # do that, because of how the flow engine then behaves. In DefaultAuthenticationFlow the
+    # "does this user actually have this credential?" guard sits INSIDE the requiresUser() branch;
+    # with requiresUser() false and no user yet set, the engine skips that guard and simply invokes
+    # the authenticator. A passkey execution placed before the username form is therefore offered
+    # to EVERYONE, so every login — including every password-only user and every family member who
+    # will never own a passkey — would open on a passkey prompt they must dismiss. That is the
+    # opposite of additive. Keeping the passkey step AFTER auth-username-form means a user is in
+    # context, which is what makes the graceful degradation below actually work.
+    #
+    # WHY THE PASSKEY BRANCH DEGRADES CLEANLY TO A PASSWORD (the property the whole "additive"
+    # claim rests on). With a user in context, the engine takes the non-requiresUser branch and
+    # raises CREDENTIAL_SETUP_REQUIRED when the user has no credential of this type and the
+    # authenticator's factory does not allow self-setup — and WebAuthnPasswordlessAuthenticatorFactory
+    # .isUserSetupAllowed() is false, and the authenticator is a CredentialValidator, so both
+    # conditions hold. That exception is raised INSIDE the ALTERNATIVE loop, which catches it,
+    # marks the execution ATTEMPTED and proceeds to the next alternative. Net effect: a user with
+    # no passkey never sees a passkey screen and lands on the password form; a user WITH one gets
+    # the passkey screen and keeps the password as a "Try another way" option. Note that for a
+    # user holding BOTH, which screen appears FIRST is decided by Keycloak's authentication
+    # selection list (the user's own credential ordering), not by the priorities below.
+    #
+    # WHY A PASSKEY LOGIN IS NOT ALSO ASKED FOR TOTP. browser-mfa-conditional-otp hangs off the
+    # PASSWORD branch only, so a platform-admin who authenticates with a passkey is not challenged
+    # for TOTP. That is intentional, and it rests on the policy above: userVerification is
+    # `required`, so the authenticator has already proved possession of hardware AND verified the
+    # human (biometric or PIN) in one phishing-resistant, origin-bound ceremony. Stacking TOTP on
+    # top would add a replayable, phishable shared secret to a credential that is neither, for
+    # friction on every login. If a future operator disagrees, the change is small and local: move
+    # the browser-mfa-conditional-otp reference out of browser-mfa-password-and-otp and into
+    # browser-mfa-passkey-or-password's parent as a REQUIRED step, so it applies to both branches.
+    #
+    # WHY THERE IS NO `requiredActions` BLOCK IN THIS FILE — DO NOT ADD ONE
+    # --------------------------------------------------------------------
+    # Enrolment uses Keycloak's `webauthn-register-passwordless` required action, and the obvious
+    # move is to declare it here. DO NOT. In Keycloak 26.0.6 the realm import treats
+    # requiredActions as REPLACE, NOT MERGE: DefaultExportImportManager.importRealm branches on
+    # `if (rep.getRequiredActions() != null)` and, when the representation declares ANY, imports
+    # exactly those plus delete_account — the `else` branch that installs the full default set
+    # (DefaultRequiredActions.addActions) never runs. RealmManager.setupRequiredActions is no
+    # backstop either: it fires only when the realm has ZERO required-action providers.
+    #
+    # So declaring one action would create this realm with that action and delete_account and
+    # NOTHING ELSE — no CONFIGURE_TOTP, no UPDATE_PASSWORD, no VERIFY_EMAIL, no delete_credential,
+    # no UPDATE_PROFILE. The first casualty would be the MFA requirement this very flow enforces:
+    # with CONFIGURE_TOTP unregistered, a platform-admin who does not yet have TOTP could never
+    # enrol one, while auth-otp-form below still demands it — a hard lock-out of every
+    # platform-admin from every SSO-gated app, including Headlamp and kubectl, delivered by a
+    # change whose stated purpose was to ADD a login method.
+    #
+    # And the declaration buys nothing, because the default set ALREADY CONTAINS IT.
+    # DefaultRequiredActions registers `webauthn-register-passwordless` with enabled=true,
+    # defaultAction=false, priority 80, gated on Profile.Feature.WEB_AUTHN — which is Type.DEFAULT
+    # in 26.0.6, i.e. on unless explicitly disabled. enabled=true / defaultAction=false IS the
+    # additive-and-optional posture this change wants: the action exists and can be triggered for
+    # a user on demand, but it is not forced on anyone at next login. Declaring it would restate
+    # the default while deleting ten siblings.
+    #
+    # (The symmetric question for authenticationFlows has the opposite answer, which is why THIS
+    # block is safe: importAuthenticationFlows adds the declared flows and then backfills each
+    # unbound binding — browser, registration, direct grant, reset credentials, client auth, first
+    # broker login, docker auth — from the built-in defaults individually. Declaring flows does not
+    # suppress the built-ins; declaring required actions does suppress them.)
     authenticationFlows:
       - alias: browser-mfa
-        description: "Browser flow with conditional OTP required for platform-admins."
+        description: "Browser flow: passkey or password, with conditional OTP for platform-admins."
         providerId: basic-flow
         topLevel: true
         builtIn: false
@@ -555,13 +782,63 @@ spec:
             priority: 30
             flowAlias: browser-mfa-forms
             userSetupAllowed: false
+      # Identify the user FIRST (username only), then prove it. Both steps are REQUIRED: the
+      # username is never optional, and neither is proving possession of the account.
       - alias: browser-mfa-forms
-        description: "Username/password then conditional OTP."
+        description: "Username, then passkey or password (+ conditional OTP)."
         providerId: basic-flow
         topLevel: false
         builtIn: false
         authenticationExecutions:
-          - authenticator: auth-username-password-form
+          # auth-username-form = Keycloak's "Username Form" (UsernameFormFactory) — resolves the
+          # user WITHOUT taking a password, which is the prerequisite for offering a passkey.
+          # Replaces the previous auth-username-password-form; see the header for the UX cost.
+          - authenticator: auth-username-form
+            authenticatorFlow: false
+            requirement: REQUIRED
+            priority: 10
+            userSetupAllowed: false
+          - authenticatorFlow: true
+            requirement: REQUIRED
+            priority: 20
+            flowAlias: browser-mfa-passkey-or-password
+            userSetupAllowed: false
+      # THE FORK. Two ALTERNATIVEs, so satisfying either one completes the flow. Priority puts the
+      # passkey first for users who have one; users who do not fall through to the password branch
+      # automatically (CREDENTIAL_SETUP_REQUIRED is caught by the ALTERNATIVE loop — see header).
+      - alias: browser-mfa-passkey-or-password
+        description: "A passkey (WebAuthn passwordless), or password with conditional OTP."
+        providerId: basic-flow
+        topLevel: false
+        builtIn: false
+        authenticationExecutions:
+          # webauthn-authenticator-passwordless = "WebAuthn Passwordless Authenticator"
+          # (WebAuthnPasswordlessAuthenticatorFactory). It reads the realm's PASSWORDLESS policy
+          # declared above — NOT the two-factor webAuthnPolicy* family. Keep it ALTERNATIVE: as
+          # REQUIRED it would force a passkey on every user, which is the lock-out this change is
+          # explicitly designed not to create.
+          - authenticator: webauthn-authenticator-passwordless
+            authenticatorFlow: false
+            requirement: ALTERNATIVE
+            priority: 10
+            userSetupAllowed: false
+          - authenticatorFlow: true
+            requirement: ALTERNATIVE
+            priority: 20
+            flowAlias: browser-mfa-password-and-otp
+            userSetupAllowed: false
+      # The unchanged pre-passkey behaviour, relocated intact into the password branch: password,
+      # then TOTP iff the user holds the platform-admin role. Nothing about the OTP condition
+      # changed; it simply no longer sits on the path a passkey login takes.
+      - alias: browser-mfa-password-and-otp
+        description: "Password, then conditional OTP for platform-admins."
+        providerId: basic-flow
+        topLevel: false
+        builtIn: false
+        authenticationExecutions:
+          # auth-password-form = Keycloak's "Password Form" (PasswordFormFactory) — the password
+          # half of the old combined form, now that the username has already been collected.
+          - authenticator: auth-password-form
             authenticatorFlow: false
             requirement: REQUIRED
             priority: 10

platform/kured/README.md

@@ -0,0 +1,127 @@
+# platform/kured
+
+**kured (Kubernetes Reboot Daemon)** — coordinated, one-node-at-a-time OS reboots of the three prod
+control-plane nodes.
+
+- **Flux tier:** `clusters/base/kured.yaml` (its own Kustomization, `dependsOn: [platform]`,
+  `wait: true`)
+- **Chart:** `kured` 6.1.0 → appVersion 1.23.0, from `https://kubereboot.github.io/charts`
+- **The other half:** `infra/ansible/roles/unattended_upgrades` — the package upgrades. **Neither
+  half is useful alone.**
+- **Runbook:** `/runbooks/node-reboot-automation`
+- **Every tuned value and its reasoning:** `helmrelease.yaml`. That file is the argument; this one is
+  the orientation.
+- **Design record:** `decisions/2026-10-17-automated-upgrades.md`
+
+## What this component owns
+
+The *mechanics* of a reboot: take a cluster-wide lock, cordon, drain, reboot, uncordon, hold the
+lock a while longer. Nothing else.
+
+**It does not decide whether to reboot.** kured reboots a node if and only if
+`/var/run/reboot-required` exists there. On these nodes that file is written by a deliberate
+operator kernel upgrade (`playbooks/node-baseline.yml --tags kernel`) — unattended-upgrades is
+security-only with the kernel blacklisted, so it essentially never produces one. **The human gate is
+the sentinel, not the daemon.**
+
+## The problem this fixes
+
+Rebooting these three nodes was a manual, serialized, supervised procedure, and
+`roles/node_baseline` has to end its kernel run by shouting the rules at the operator: *"Reboot prod
+nodes ONE AT A TIME, verify each returns before touching the next, and NEVER node 2 first"*. That is
+a correct procedure and an error-prone one, and nothing enforced it.
+
+What it has to get right, measured on prod 2026-10-17:
+
+- **etcd quorum is 2 of 3** — all three nodes are control-plane+etcd. Two down is a lost cluster.
+- **Every drain degrades all 25 Longhorn volumes to 2/3**, because 3 nodes × 3 replicas ×
+  `replica-soft-anti-affinity: false` leaves nowhere to place a replacement, and rebuild is
+  serialized behind a 600s wait. Reboot the next node too early and volumes hit 1/3, where
+  `block-if-contains-last-replica` can block a drain forever.
+- **Six PodDisruptionBudgets allow zero voluntary disruptions** — 3 Longhorn `instance-manager-*`
+  (structurally 0 by design; this is how Longhorn gates the drain) and 3 CNPG `*-primary`.
+
+## Placement — why its own Kustomization
+
+`platform` is `wait: true` over twelve components and gates `platform-config` → every apps wave. A
+reboot daemon that failed to go healthy there would stall **every application in the cluster** on
+"dependency not ready". A component added to make maintenance safer must not be able to cause an
+outage. `wait: true` here is a health **signal** for something nothing depends on — and nothing
+should ever `dependsOn: [kured]`.
+
+The `dependsOn: [platform]` edge is load-bearing, not conventional: it supplies the
+`monitoring.coreos.com` CRDs this component's `PrometheusRule` needs (F-MON1 — a pass containing an
+unknown kind aborts), plus the Prometheus that kured **queries** before every reboot.
+
+## How concurrent reboots are made impossible
+
+| mechanism | value | why |
+|---|---|---|
+| cluster-wide lock | one annotation on kured's own DaemonSet | a node reboots only while holding it; enforced by API-server optimistic concurrency, not timing |
+| `concurrency` | `1` | one lock holder at a time |
+| **`lockTtl`** | **`0` (disabled)** | a TTL lets reboots continue after the holder dies — on 3 etcd nodes that is the catastrophe. A hung node holds the lock **forever** and nothing cascades. |
+| `lockReleaseDelay` | `30m` | the next node waits out Longhorn's 600s replenishment even if no alert latched |
+| alert gate | `alertFilterMatchOnly: true` + anchored allow-list | refuses to proceed while a volume is degraded/faulted or a node is down |
+| window | 03:00–06:00 daily, America/New_York | matches `renovate.json`'s window; daily (not weekly) is what makes `KuredRebootPending`'s `for: 36h` meaningful |
+
+## Deliberate non-choices
+
+- **No `--ignore-daemonsets` / `--delete-emptydir-data` / `--force`.** Those flags **do not exist**
+  in kured 1.23.0 — it hardcodes all three in the drain helper (`cmd/kured/main.go:438-449`).
+  Passing them would CrashLoop the DaemonSet. All three are genuinely required here (12 DaemonSets,
+  135 emptyDir pods).
+- **`rebootMethod: command`, not `signal`.** `signal` would allow `privileged: false` + `CAP_KILL`,
+  but needs a host-side `kured-reboot.service` that does not exist on these nodes. A real future
+  hardening step; explicitly not taken here.
+- **No tolerations.** No prod node is tainted (measured). A future tainted node would be silently
+  uncovered — hence `KuredNodeCoverageGap`.
+- **`NodeEtcdRequestsSlow` / `NodeIOPressure*` are NOT in the alert gate**, despite being exactly
+  what you would want to pause for. prod-1 has a documented etcd WAL I/O starvation history and
+  including them risks a permanent wedge that looks like safety.
+- **No Slack/shoutrrr notifications.** Would require a SOPS Secret and a decryption patch; the three
+  alerts cover the same ground through Alertmanager.
+
+## Versioning
+
+Chart pinned `>=6.1.0 <6.2.0`. The minor ceiling is **deliberate here**, unlike elsewhere in this
+repo where it is a main cause of pin rot: a kured minor may change drain or lock semantics, and this
+is the one component where "a human read the changelog" should be a precondition. `renovate.json`'s
+`blast-radius` rule additionally forbids auto-merging it, so the ceiling is a review prompt, not a
+wall.
+
+## What breaks silently, and what catches it
+
+Every failure mode of this component leaves a cluster that looks healthy — kured stays 1/1 Running
+while rebooting nothing. `alerts.yaml` ships three rules over kured's single metric, with a
+companion promtool suite:
+
+| alert | catches |
+|---|---|
+| `KuredNotReporting` | the scrape pipeline is dead (chart defaults `service.create`/`metrics.create` to **false**; a NetworkPolicy can time the scrape out) — and the other two are inert while this is true |
+| `KuredNodeCoverageGap` | kured missing on *some* node, which `absent()` cannot see |
+| `KuredRebootPending` | a reboot was offered and blocked for >36h |
+
+## Post-merge verification
+
+```bash
+infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds,pod,svc,servicemonitor -o wide'
+infra/devcontainer/dev.sh run bash -c 'flux --context k3s-prod get kustomization kured'
+infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured logs ds/kured --tail=40'
+```
+
+Expect `3/3` ready, the Kustomization `Ready=True`, and kured logging a reboot-check decision each
+period. Then confirm the scrape is real — this is the step that proves the alerts are not inert:
+
+```bash
+infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n monitoring exec \
+  sts/prometheus-kube-prometheus-stack-prometheus -c prometheus -- \
+  wget -qO- "http://localhost:9090/api/v1/query?query=count(kured_reboot_required)"'
+```
+
+**Expect `3`.** Anything else (and especially an empty result) means the Service, ServiceMonitor or
+NetworkPolicy is wrong and all three alerts are silently dead.
+
+**Before trusting the timer with a real kernel upgrade**, do one watched drain to settle the open
+question the design record flags: whether CNPG actually switches its primary away in response to a
+drain-induced eviction. If it does not, draining `k3s-prod-1` blocks on three 0-disruption PDBs at
+once.

platform/kured/alerts.promtool-test.yaml

@@ -0,0 +1,282 @@
+---
+# promtool unit tests for the three rules in platform/kured/alerts.yaml.
+#
+# WHY THIS FILE EXISTS. The rules it covers are the ONLY thing that can tell an operator that the
+# reboot automation has stopped working, and every failure mode they describe is silent — kured stays
+# 1/1 Running while rebooting nothing. An alert that cannot fire is therefore not a cosmetic defect
+# here, it is the removal of the only signal. This repo has shipped that bug in both directions
+# already (a permanently-firing VPNTunnelDown; a structurally inert VeleroNoRecentBackup over an
+# absent series), which is why the harness exists at all. Each rule below is pinned BOTH ways: quiet
+# on the healthy state, firing on the constructed bad one, with the `for:` boundary asserted rather
+# than assumed.
+#
+# HOW TO RUN. `platform/monitoring/tests/run.sh` — the one repo-wide rule-unit harness. It finds this
+# file by the COMPANION convention (`<manifest>.promtool-test.yaml` beside `<manifest>.yaml`), so the
+# manifest under test is exactly platform/kured/alerts.yaml and nothing else. Run just this file with
+#   platform/monitoring/tests/run.sh platform/kured/alerts.promtool-test.yaml
+# The harness flattens the PrometheusRule CR's `.spec.groups` into the bare rules file promtool wants,
+# `promtool check rules` it first so a non-parsing file cannot pass vacuously, and runs it all in a
+# temp dir. It leaves Flux's `${BASE_DOMAIN}` in the runbook_url annotations verbatim, which is safe
+# because the assertions below query the `ALERTS` series and never compare annotations.
+#
+# WHY THE ASSERTIONS QUERY `ALERTS` RATHER THAN USING `exp_alerts`. promtool's alert_rule_test
+# compares annotations VERBATIM, which would mean pasting each rule's description prose in here and
+# re-pasting it on every wording change. Querying the synthetic `ALERTS` series proves what actually
+# matters — the alert reaches `pending` and then `firing` across its `for:` boundary, with the right
+# labels — and stays decoupled from annotation text. Same choice as
+# platform/cnpg/alerts.promtool-test.yaml.
+#
+# THE SERIES SHAPES ARE REAL. `kured_reboot_required{node=...}` is a GaugeVec with exactly one label,
+# `node` — read from kured's cmd/kured/main.go:92-96 (the only registered metric in the project).
+# `kube_node_info{node=...}` was measured on prod Prometheus on 2026-10-17 returning
+# `count(kube_node_info) == 3`, matching the three nodes. The extra kube_node_info labels prod
+# carries are omitted deliberately: every rule aggregates with `count()`, so they cannot affect the
+# result, and inventing plausible-but-wrong values would be worse than leaving them out.
+#
+# evaluation_interval is 5m, not the usual 1m: KuredRebootPending has `for: 36h`, and at a 1m
+# interval promtool would evaluate that test over 2000+ steps for no added coverage.
+rule_files:
+  - alerts.rules.yaml
+evaluation_interval: 5m
+
+tests:
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  # 1. THE HEALTHY STEADY STATE. Three nodes, kured running on all three, nothing pending.
+  #    ALL THREE RULES MUST BE SILENT. This is the case that a badly-written `absent()` or a
+  #    `count() <` comparison with an off-by-one gets wrong, and it is also the state the cluster
+  #    is in ~100% of the time, so a false positive here would train the operator to ignore the
+  #    whole group.
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  - interval: 5m
+    name: healthy cluster - no reboots pending, kured on every node, all three rules quiet
+    input_series:
+      - series: 'kured_reboot_required{node="k3s-prod-1"}'
+        values: '0x60'
+      - series: 'kured_reboot_required{node="k3s-prod-2"}'
+        values: '0x60'
+      - series: 'kured_reboot_required{node="k3s-prod-3"}'
+        values: '0x60'
+      - series: 'kube_node_info{node="k3s-prod-1"}'
+        values: '1x60'
+      - series: 'kube_node_info{node="k3s-prod-2"}'
+        values: '1x60'
+      - series: 'kube_node_info{node="k3s-prod-3"}'
+        values: '1x60'
+    promql_expr_test:
+      # Nothing from this group may be pending or firing, at any point in the window.
+      - expr: 'count(ALERTS{alertname=~"Kured.*"})'
+        eval_time: 4h
+        exp_samples: []
+      # And the underlying expressions must each be genuinely false, not merely unevaluated:
+      # absent() yields no series because the vector is populated...
+      - expr: 'absent(kured_reboot_required)'
+        eval_time: 4h
+        exp_samples: []
+      # ...coverage is exact (3 == 3), so the `<` comparison drops the sample...
+      - expr: 'count(kured_reboot_required) < count(kube_node_info)'
+        eval_time: 4h
+        exp_samples: []
+      # ...and no node reports a pending reboot.
+      - expr: 'max by (node) (kured_reboot_required) == 1'
+        eval_time: 4h
+        exp_samples: []
+
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  # 2. KuredNotReporting — THE WHOLE PIPELINE IS DEAD. No kured series at all, which is what the
+  #    cluster looks like if the chart's `service.create`/`metrics.create` regress to their FALSE
+  #    defaults, if the ServiceMonitor is dropped, or if a NetworkPolicy silently times out the
+  #    scrape (the Longhorn precedent). The nodes still exist, so kube_node_info is present —
+  #    proving the rule keys on kured's absence and not on a dead cluster.
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  - interval: 5m
+    name: no kured series anywhere - KuredNotReporting fires after 30m
+    input_series:
+      - series: 'kube_node_info{node="k3s-prod-1"}'
+        values: '1x60'
+      - series: 'kube_node_info{node="k3s-prod-2"}'
+        values: '1x60'
+      - series: 'kube_node_info{node="k3s-prod-3"}'
+        values: '1x60'
+    promql_expr_test:
+      # The expression itself yields 1 immediately.
+      - expr: 'absent(kured_reboot_required)'
+        eval_time: 10m
+        exp_samples:
+          - labels: '{}'
+            value: 1
+      # `for: 30m` is honoured — still only pending at 25m.
+      - expr: 'ALERTS{alertname="KuredNotReporting"}'
+        eval_time: 25m
+        exp_samples:
+          - labels: 'ALERTS{alertname="KuredNotReporting", alertstate="pending", severity="warning"}'
+            value: 1
+      # ...and firing once past it.
+      - expr: 'ALERTS{alertname="KuredNotReporting", alertstate="firing"}'
+        eval_time: 35m
+        exp_samples:
+          - labels: 'ALERTS{alertname="KuredNotReporting", alertstate="firing", severity="warning"}'
+            value: 1
+      # THE GUARD CLAUSE: with kured absent, the other two rules are inert — exactly the reason
+      # this rule has to exist. KuredRebootPending cannot fire (no series to be 1) and
+      # KuredNodeCoverageGap cannot either, because `count()` over an empty vector returns NO
+      # SAMPLE rather than 0, so the `<` comparison has nothing to compare. Asserting that here
+      # pins the dependency: if a future edit made coverage-gap fire in this state, the overlap
+      # would be deliberate rather than accidental.
+      - expr: 'ALERTS{alertname="KuredRebootPending"}'
+        eval_time: 35m
+        exp_samples: []
+      - expr: 'ALERTS{alertname="KuredNodeCoverageGap"}'
+        eval_time: 35m
+        exp_samples: []
+      - expr: 'count(kured_reboot_required) < count(kube_node_info)'
+        eval_time: 35m
+        exp_samples: []
+
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  # 3. KuredNodeCoverageGap — kured COVERS ONLY TWO OF THREE NODES. k3s-prod-3's series stops at
+  #    1h and never returns (a node taint this DaemonSet does not tolerate, a node that never came
+  #    back from a reboot, a DaemonSet nodeSelector edit). This also pins the `for: 2h` choice,
+  #    which exists SO THAT AN ORDINARY REBOOT DOES NOT TRIP IT: the gap is real from ~1h05 but
+  #    the alert is only pending an hour later, and a healthy reboot is minutes.
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  - interval: 5m
+    name: kured missing on one node - KuredNodeCoverageGap fires after 2h, not before
+    input_series:
+      - series: 'kured_reboot_required{node="k3s-prod-1"}'
+        values: '0x60'
+      - series: 'kured_reboot_required{node="k3s-prod-2"}'
+        values: '0x60'
+      # Ends after 1h (13 samples at 5m) and is stale thereafter.
+      - series: 'kured_reboot_required{node="k3s-prod-3"}'
+        values: '0x12'
+      - series: 'kube_node_info{node="k3s-prod-1"}'
+        values: '1x60'
+      - series: 'kube_node_info{node="k3s-prod-2"}'
+        values: '1x60'
+      - series: 'kube_node_info{node="k3s-prod-3"}'
+        values: '1x60'
+    promql_expr_test:
+      # Before the gap opens, coverage is exact and nothing is pending.
+      - expr: 'ALERTS{alertname="KuredNodeCoverageGap"}'
+        eval_time: 30m
+        exp_samples: []
+      # The shortfall is real well before the alert fires — 2 < 3.
+      - expr: 'count(kured_reboot_required)'
+        eval_time: 2h
+        exp_samples:
+          - labels: '{}'
+            value: 2
+      # A SHORT OUTAGE DOES NOT ALERT: still only pending at 2h, an hour after the gap opened.
+      - expr: 'ALERTS{alertname="KuredNodeCoverageGap"}'
+        eval_time: 2h
+        exp_samples:
+          - labels: 'ALERTS{alertname="KuredNodeCoverageGap", alertstate="pending", severity="warning"}'
+            value: 1
+      # Firing once the gap has persisted past `for: 2h`.
+      - expr: 'ALERTS{alertname="KuredNodeCoverageGap", alertstate="firing"}'
+        eval_time: 3h15m
+        exp_samples:
+          - labels: 'ALERTS{alertname="KuredNodeCoverageGap", alertstate="firing", severity="warning"}'
+            value: 1
+      # KuredNotReporting must stay QUIET: two nodes are still publishing, so the vector is not
+      # absent. This is the distinction between "kured is gone" and "kured is incomplete", and
+      # getting it wrong in either direction would collapse two different diagnoses into one.
+      - expr: 'ALERTS{alertname="KuredNotReporting"}'
+        eval_time: 3h15m
+        exp_samples: []
+
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  # 4. KuredRebootPending — A REBOOT IS BLOCKED. k3s-prod-1 reports a pending reboot continuously:
+  #    the sentinel was written (an operator kernel upgrade) and the reboot has not happened across
+  #    more than one 03:00-06:00 window. Asserts the 36h boundary in both directions, which is the
+  #    assertion that makes the DAILY-window decision load-bearing rather than cosmetic: on a
+  #    weekly window this threshold could not distinguish "blocked" from "waiting for Sunday".
+  #    Series run to 38h20m (456 samples at 5m) so the post-boundary evaluation has real data.
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  - interval: 5m
+    name: reboot pending on one node for over 36h - KuredRebootPending fires, per-node label kept
+    input_series:
+      - series: 'kured_reboot_required{node="k3s-prod-1"}'
+        values: '1x456'
+      - series: 'kured_reboot_required{node="k3s-prod-2"}'
+        values: '0x456'
+      - series: 'kured_reboot_required{node="k3s-prod-3"}'
+        values: '0x456'
+      - series: 'kube_node_info{node="k3s-prod-1"}'
+        values: '1x456'
+      - series: 'kube_node_info{node="k3s-prod-2"}'
+        values: '1x456'
+      - series: 'kube_node_info{node="k3s-prod-3"}'
+        values: '1x456'
+    promql_expr_test:
+      # The expression selects ONLY the pending node — the operator needs to know which box.
+      - expr: 'max by (node) (kured_reboot_required) == 1'
+        eval_time: 1h
+        exp_samples:
+          - labels: '{node="k3s-prod-1"}'
+            value: 1
+      # A NORMAL WAIT IS NOT AN ALERT. At 24h the node is still inside its legitimate
+      # once-per-day window wait, so the alert may be pending but must NOT be firing.
+      - expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}'
+        eval_time: 24h
+        exp_samples: []
+      # Still not firing just short of the boundary.
+      - expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}'
+        eval_time: 35h30m
+        exp_samples: []
+      # Firing past it, carrying the node label through to the notification.
+      - expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}'
+        eval_time: 36h30m
+        exp_samples:
+          - labels: 'ALERTS{alertname="KuredRebootPending", alertstate="firing", node="k3s-prod-1", severity="warning"}'
+            value: 1
+      # The two healthy nodes must NOT be alerted on. A rule written with `kured_reboot_required`
+      # unaggregated, or compared with `>= 0`, would raise all three and bury the real one.
+      - expr: 'count(ALERTS{alertname="KuredRebootPending", alertstate="firing"})'
+        eval_time: 36h30m
+        exp_samples:
+          - labels: '{}'
+            value: 1
+      # Liveness rules stay quiet: kured is present and covering all three nodes. A pending reboot
+      # is NOT a wiring fault and must not be reported as one.
+      - expr: 'ALERTS{alertname="KuredNotReporting"}'
+        eval_time: 36h30m
+        exp_samples: []
+      - expr: 'ALERTS{alertname="KuredNodeCoverageGap"}'
+        eval_time: 36h30m
+        exp_samples: []
+
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  # 5. THE RULE MUST STOP. A reboot that SUCCEEDS clears the sentinel, and the alert must clear
+  #    with it — the other half of the failure space this harness exists for (a permanently-firing
+  #    rule is how VPNTunnelDown taught this repo the lesson). k3s-prod-1 is pending for ~40h,
+  #    long enough to fire, then the reboot happens and the gauge returns to 0.
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  - interval: 5m
+    name: reboot completes after firing - KuredRebootPending resolves
+    input_series:
+      # 1 for 480 samples (40h), then 0 for the remainder.
+      - series: 'kured_reboot_required{node="k3s-prod-1"}'
+        values: '1x480 0x120'
+      - series: 'kured_reboot_required{node="k3s-prod-2"}'
+        values: '0x600'
+      - series: 'kured_reboot_required{node="k3s-prod-3"}'
+        values: '0x600'
+      - series: 'kube_node_info{node="k3s-prod-1"}'
+        values: '1x600'
+      - series: 'kube_node_info{node="k3s-prod-2"}'
+        values: '1x600'
+      - series: 'kube_node_info{node="k3s-prod-3"}'
+        values: '1x600'
+    promql_expr_test:
+      # Firing while the reboot is still outstanding.
+      - expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}'
+        eval_time: 39h
+        exp_samples:
+          - labels: 'ALERTS{alertname="KuredRebootPending", alertstate="firing", node="k3s-prod-1", severity="warning"}'
+            value: 1
+      # Cleared once the sentinel is gone. If this assertion ever fails, the rule has become
+      # self-sustaining and the operator will start ignoring it.
+      - expr: 'ALERTS{alertname=~"Kured.*"}'
+        eval_time: 45h
+        exp_samples: []

platform/kured/alerts.yaml

@@ -0,0 +1,124 @@
+---
+# Alert rules for the kured reboot automation.
+#
+# WHY THESE THREE, AND WHY THEY ARE NOT OPTIONAL. Reboot automation's characteristic failure is
+# SILENCE, not noise. Every way this component can break leaves a cluster that looks perfectly
+# healthy: a sentinel nothing writes, a Service the chart did not create, a NetworkPolicy that drops
+# the scrape, a taint that excludes a node, a drain wedged on a PodDisruptionBudget, a cluster lock
+# held forever by a node that hung on warm reboot. In none of those cases does anything turn red on
+# its own. kured itself stays 1/1 Running throughout.
+#
+# kured EXPOSES EXACTLY ONE METRIC. Read from cmd/kured/main.go:92-96 and :113 on 2026-10-17 — the
+# only `prometheus.MustRegister` in the tree is a GaugeVec `Subsystem: "kured"`, `Name:
+# "reboot_required"`, labelled `{node}`. So the full vocabulary available is
+# `kured_reboot_required{node="..."}`, 1 when the sentinel exists and 0 when it does not. Each rule
+# below is built from that one series plus `kube_node_info` (kube-state-metrics), and nothing else.
+# There is deliberately no rule over an invented metric name: this repo has shipped a
+# structurally-inert alert before (VeleroNoRecentBackup over an absent series, which is why
+# platform/monitoring/tests/run.sh exists), and an alert that cannot fire is worse than no alert
+# because it also removes the suspicion that would have caught it.
+#
+# THESE RULES LIVE HERE, NOT IN platform/monitoring/alerts.yaml, for two reasons. (1) The unit-test
+# harness treats a `<manifest>.promtool-test.yaml` COMPANION as covering just its sibling, whereas a
+# `tests/*.test.yaml` SUITE covers every PrometheusRule in its directory — so shipping them beside
+# the component keeps the test obligation local and legible (the platform/cnpg precedent). (2)
+# platform/monitoring/alerts.yaml is concurrently edited by other open branches; adding 3 rules to
+# its tail would create a merge conflict for no benefit.
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: kured
+  namespace: kured
+  labels:
+    app.kubernetes.io/name: kured
+spec:
+  groups:
+    - name: platform-health.node-reboots
+      rules:
+        # ── 1. IS THE MECHANISM ALIVE AT ALL? ────────────────────────────────────────────────
+        - alert: KuredNotReporting
+          # `absent()` yields a series only when the ENTIRE vector is empty, so this fires when no
+          # node at all is publishing — kured gone, or its Service/ServiceMonitor/NetworkPolicy
+          # regressed so nothing is scraped. It stays quiet while a single node is mid-reboot
+          # (the other two still report), which is what makes it specific to a wiring failure
+          # rather than to a reboot in progress.
+          #
+          # THIS IS THE RULE THAT GUARDS THE OTHER TWO. Both of them read `kured_reboot_required`,
+          # so if that series stops existing they go quiet and look like good news. This one turns
+          # the absence itself into the signal. The specific regression it is aimed at: the kured
+          # chart defaults `service.create` and `metrics.create` to FALSE, and Longhorn has already
+          # demonstrated on this cluster (see the companion note in
+          # conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh) that a
+          # NetworkPolicy can let targets be DISCOVERED while every scrape times out and the TSDB
+          # stays empty.
+          expr: absent(kured_reboot_required)
+          # 30m absorbs a HelmRelease upgrade rolling the DaemonSet and a Prometheus restart,
+          # without tolerating a real wiring break for more than one quiet night's worth of time.
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: kured is publishing no reboot_required series on any node
+            description: >-
+              No node is reporting kured_reboot_required, so the reboot automation is either not
+              running or not being scraped. While this is true, OS reboots are not happening and
+              the two alerts that would say so (KuredRebootPending, KuredNodeCoverageGap) are
+              themselves inert. Nodes will silently accumulate pending kernel upgrades.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-reboot-automation"
+
+        # ── 2. IS IT ALIVE ON *EVERY* NODE? ──────────────────────────────────────────────────
+        - alert: KuredNodeCoverageGap
+          # A per-node agent that silently covers only some nodes is the subtler half of the
+          # liveness question, and `absent()` cannot see it. kured publishes one series per node it
+          # runs on; kube-state-metrics publishes one `kube_node_info` per node that exists
+          # (measured 2026-10-17: 3, matching the 3 nodes). A shortfall means at least one node has
+          # no reboot automation — most likely a new or re-provisioned node carrying a taint this
+          # DaemonSet does not tolerate (it declares none, because no prod node is tainted today).
+          # That node would then never reboot, with nothing else to indicate it.
+          expr: count(kured_reboot_required) < count(kube_node_info)
+          # 2h, chosen to sit comfortably ABOVE the duration of a healthy reboot. A node being
+          # rebooted stops publishing for a few minutes and legitimately creates a shortfall; a
+          # node that is still missing after two hours is either genuinely uncovered or has hung on
+          # reboot, and both of those want a human.
+          for: 2h
+          labels:
+            severity: warning
+          annotations:
+            summary: kured is not running on every node
+            description: >-
+              Fewer nodes are reporting kured_reboot_required than exist in the cluster, so at
+              least one node has no reboot automation and will never be rebooted by it. Most
+              likely a node carries a taint this DaemonSet does not tolerate, or has hung during
+              a reboot.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-reboot-automation"
+
+        # ── 3. IS A REBOOT STUCK? ────────────────────────────────────────────────────────────
+        - alert: KuredRebootPending
+          # A node has wanted a reboot for longer than a full maintenance window plus margin, so
+          # the reboot was OFFERED and did not happen. The causes are all real and all invisible
+          # otherwise: the Prometheus alert gate is blocking on a degraded Longhorn volume that is
+          # not recovering; the drain hit `drainTimeout` against a 0-disruption PDB; another node
+          # holds the cluster lock and never released it (`lockTtl: 0` means a lock held by a hung
+          # node is held FOREVER — that is deliberate, see helmrelease.yaml, and this is the alert
+          # that makes the resulting stall visible).
+          expr: max by (node) (kured_reboot_required) == 1
+          # 36h, and the window being DAILY is what makes this number meaningful — see the window
+          # block in helmrelease.yaml. With a 03:00-06:00 window every day, a pending reboot waits
+          # at most ~24h legitimately, so 36h means at least one full window elapsed without the
+          # reboot completing. On a weekly window this threshold would have to exceed 7 days, which
+          # is longer than this cluster's 7d Prometheus retention and is reset by any Prometheus
+          # restart past the 1h for-outage-tolerance — i.e. unreliable in the one direction that
+          # matters. Note this alert can still be LATE for that reason, but it cannot be spurious.
+          for: 36h
+          labels:
+            severity: warning
+          annotations:
+            summary: Node {{ $labels.node }} has needed a reboot for over 36h
+            description: >-
+              kured reports a pending reboot on {{ $labels.node }} that has survived at least one
+              full 03:00-06:00 maintenance window. The reboot is being blocked, not merely
+              waiting: likely a degraded Longhorn volume holding the Prometheus alert gate, a
+              drain timing out against a PodDisruptionBudget, or a stale cluster lock held by a
+              node that did not come back. The kernel or security update that needed the reboot is
+              installed but NOT running.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-reboot-automation"

platform/kured/helmrelease.yaml

@@ -0,0 +1,289 @@
+---
+# kured (Kubernetes Reboot Daemon) — coordinated, one-node-at-a-time reboots of the three prod
+# control-plane nodes.
+#
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# READ THIS FIRST: kured DOES NOT DECIDE *WHETHER* TO REBOOT. IT ONLY DECIDES *WHEN* AND *HOW*.
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# kured reboots a node if and only if the file `/var/run/reboot-required` exists on it. On these
+# nodes exactly one thing creates that file: a deliberate operator run of
+# `playbooks/node-baseline.yml --tags kernel` (which advances the backports kernel). The
+# unattended-upgrades role this PR also adds installs SECURITY updates only and BLACKLISTS
+# `linux-image-*`/`linux-headers-*`, so it can never produce a reboot-required sentinel on its own —
+# see infra/ansible/roles/unattended_upgrades/defaults/main.yml for that argument in full.
+#
+# That is the whole answer to this repo's two standing objections to reboot automation:
+#
+#   * infra/ansible/roles/node_baseline/tasks/main.yml:61-67 — "IT DOES NOT REBOOT." k3s-prod-2 hung
+#     on a warm reboot on 2026-09-03 (powered, off the LAN ~38 minutes, physical power button
+#     required), so rebooting was made "a SERIALIZED, SUPERVISED operator step ... and a role cannot
+#     enforce that." STILL TRUE, and nothing here changes it: node_baseline continues not to reboot.
+#   * docs/plans/cluster-upgrade-plan.md:291-299 — system-upgrade-controller was rejected because it
+#     "automates away the human gate between nodes", and the operator's hard requirement is an
+#     explicit PASS/STOP verification BETWEEN every node.
+#
+# THE HUMAN GATE IS PRESERVED, AND IT IS THE SENTINEL. The operator decides whether a kernel moves
+# and on which hosts (`--limit k3s-prod-1`), which is what writes the sentinel on those hosts and
+# only those hosts. kured then performs the cordon/drain/reboot/uncordon that the operator would
+# otherwise have typed by hand — the error-prone mechanical half, including the ordering rule
+# node_baseline's own message has to shout ("NEVER node 2 first"). Run the kernel play one host at a
+# time and the operator keeps per-node PASS/STOP control exactly as before; run it on all three and
+# kured still does them strictly one at a time, just in an order it chooses. Both are safe for
+# quorum; only the first preserves the ordering preference. See runbooks/node-reboot-automation.md.
+#
+# cluster-upgrade-plan.md:329-330 and :545-548 name the condition under which its own rejection
+# reverses — "SUC becomes viable once the cluster is on fast disks ... where drain-driven rebuild is
+# cheap" — and prod is that cluster. This component takes that clause up, but NOT its drain
+# assumption: see the Longhorn section below, where drain-driven rebuild is measurably NOT free even
+# on SSD, and is gated on explicitly rather than waved through.
+#
+# WHAT IS STILL UNMITIGATED, STATED PLAINLY: kured cannot press a power button. If a node hangs on
+# warm reboot the way k3s-prod-2 did in September, kured will not recover it. The blast radius is
+# bounded by design rather than by luck — `concurrency: 1` plus `lockTtl: 0` means the hung node
+# keeps the cluster lock forever, so NO second node is ever rebooted behind it; etcd keeps 2/3 quorum
+# and the cluster stays up, degraded, until a human attends. The KuredRebootPending alert in
+# alerts.yaml is what tells them. That is strictly better than the previous automation story (none)
+# but it is NOT the same as "safe to ignore at 3am on a Sunday".
+apiVersion: helm.toolkit.fluxcd.io/v2
+kind: HelmRelease
+metadata:
+  name: kured
+  namespace: kured
+spec:
+  interval: 30m
+  chart:
+    spec:
+      chart: kured
+      # Chart 6.1.0 -> appVersion 1.23.0 (published 2026-06-30; newest of the 80 entries in
+      # https://kubereboot.github.io/charts/index.yaml, read 2026-10-17).
+      #
+      # THE `<6.2.0` CEILING IS A DELIBERATE CHOICE HERE, NOT THE REFLEX IT IS ELSEWHERE. This PR's
+      # own audit found that 9 of 12 HelmReleases in this repo pin a minor-locking range, and that
+      # this is a MAIN CAUSE of pin rot: Renovate can then only ever offer patches inside a line it
+      # has been told not to leave (kube-prometheus-stack is stuck 4 chart majors back that way).
+      # For a component whose job is to reboot control-plane nodes, that ceiling is worth paying on
+      # purpose: a kured MINOR may change drain or lock semantics, and this is the one component
+      # where "a human looked at the changelog" should be a precondition, not a nice-to-have.
+      # Renovate will raise the minor as a non-automerged PR (see the kured packageRule in
+      # renovate.json), so the ceiling is a review prompt, not a wall.
+      version: ">=6.1.0 <6.2.0"
+      sourceRef:
+        kind: HelmRepository
+        name: kured
+        namespace: kured
+  install:
+    remediation:
+      retries: 3
+  upgrade:
+    remediation:
+      retries: 3
+  values:
+    # ── IDENTITY / PLACEMENT ───────────────────────────────────────────────────────────────────
+    # No tolerations. Measured 2026-10-17: all three prod nodes carry `spec.taints` EMPTY (k3s does
+    # not taint its servers), so a DaemonSet reaches every node untainted. If a future node IS
+    # tainted, kured silently stops covering it and that node then never reboots — invisible,
+    # because the other two keep reporting. KuredNodeCoverageGap in alerts.yaml is the detector: it
+    # compares kured's per-node series count against the node count rather than merely asking
+    # whether kured exists.
+    priorityClassName: system-node-critical
+    # The chart's defaults are kept: `hostPID: true` and `containerSecurityContext.privileged: true`
+    # with `readOnlyRootFilesystem: true`. Both are load-bearing and neither is reducible:
+    # `rebootMethod: command` runs the reboot by nsenter'ing /proc/1/ns/mnt, which needs the host PID
+    # namespace to see PID 1 and the privilege to enter its mount namespace. The documented
+    # lower-privilege alternative (privileged:false + CAP_KILL) applies ONLY to
+    # `rebootMethod: signal`, which signals a host-side kured-reboot.service that does not exist on
+    # these nodes. Choosing `signal` would mean shipping and maintaining that unit via Ansible for a
+    # capability reduction on a DaemonSet that is already cluster-admin-adjacent by function; it is a
+    # reasonable future hardening step and is explicitly NOT taken here.
+    resources:
+      requests:
+        cpu: 10m
+        memory: 32Mi
+      limits:
+        # No CPU limit on purpose (the repo's node-pressure work treats CPU limits on tiny system
+        # agents as a throttling hazard). Memory is capped: kured holds no state beyond one
+        # annotation and a gauge, and 128Mi is ~4x its idle RSS upstream reports.
+        memory: 128Mi
+
+    configuration:
+      # ── THE QUORUM CONSTRAINT: TWO NODES MUST NEVER BE DOWN AT ONCE ──────────────────────────
+      # All three prod nodes are control-plane + etcd (measured 2026-10-17: every node reports
+      # `control-plane,etcd`). etcd quorum is 2 of 3, so one node down is degraded-but-up and two
+      # nodes down is a LOST CLUSTER that does not self-heal. Everything in this block exists to
+      # make the second case structurally unreachable rather than merely unlikely.
+      #
+      # concurrency: 1 — kured's cluster-wide lock is a single annotation
+      # (`weave.works/kured-node-lock`) on kured's OWN DaemonSet object, and a node reboots only
+      # while it holds that lock. One lock + concurrency 1 = one reboot, enforced by the API server's
+      # optimistic concurrency rather than by timing. This is the chart default; it is written out
+      # explicitly because it is the single most important value in this file.
+      concurrency: 1
+      # lockTtl: 0 (DISABLED) — THIS IS A SAFETY DECISION, NOT A DEFAULT WE FORGOT TO TUNE.
+      # A TTL makes the lock EXPIRE, which exists for the case "the lock holder died and we want
+      # reboots to continue". On a 3-node etcd cluster that is precisely the catastrophe: if node A
+      # has hung on a warm reboot (which has happened here — see the header) and its lock expires,
+      # node B becomes eligible and takes the cluster from 2/3 to 1/3. We would rather the reboot
+      # cycle STOP FOREVER and page a human than have it step over a corpse. The cost is that a
+      # genuinely stale lock needs manual clearing; runbooks/node-reboot-automation.md has the
+      # procedure, and KuredRebootPending is the alert that surfaces it.
+      lockTtl: 0
+      # lockReleaseDelay: 30m — hold the lock for 30 minutes AFTER this node is back and uncordoned,
+      # before any other node may take it. See the Longhorn section: this is the STRUCTURAL half of
+      # the inter-node gate, deliberately not dependent on an alert's `for:` timer having latched.
+      lockReleaseDelay: 30m
+
+      # ── THE MAINTENANCE WINDOW: 03:00-06:00 EVERY DAY, America/New_York ──────────────────────
+      # The timezone and the 03:00-06:00 band match renovate.json's existing
+      # `"schedule": ["after 3am and before 6am"]` / `"timezone": "America/New_York"`, so every
+      # unattended change in this repo lands in the same quiet 3-hour window.
+      #
+      # DAILY, NOT WEEKLY, AND THE REASON IS THE ALERT — not impatience. A weekly (`- sun`) window
+      # was the first draft and it breaks the only alert that can tell you this mechanism has
+      # stalled. With a Sunday-only window, a kernel upgraded on a Monday legitimately waits six
+      # days, so "a reboot has been pending too long" cannot be distinguished from "it is Tuesday"
+      # until the threshold exceeds a full 7-day cycle. That forces `for: 8d`+, which is longer than
+      # this cluster's Prometheus retention (MONITORING_PROMETHEUS_RETENTION is 7d) and is reset by
+      # any Prometheus restart beyond the 1h default for-outage-tolerance — i.e. the alert would be
+      # unreliable in exactly the direction that matters. A daily window caps the legitimate wait at
+      # under 24h, which makes KuredRebootPending's `for: 36h` both robust and meaningful: 36 hours
+      # with a sentinel still set means at least one full window passed and SOMETHING BLOCKED IT.
+      #
+      # Daily does NOT weaken the quorum guarantee — that is `concurrency: 1` plus the lock, not the
+      # calendar. It does mean a 3-node kernel rollout can finish in one to three nights instead of
+      # three weeks, with each node still gated on the previous one's Longhorn rebuilds (below).
+      rebootDays:
+        - su
+        - mo
+        - tu
+        - we
+        - th
+        - fr
+        - sa
+      startTime: "03:00"
+      endTime: "06:00"
+      timeZone: America/New_York
+      # How often to re-evaluate (sentinel + window + alerts + lock). 1h is the chart default; at a
+      # 3h window that is three evaluation opportunities per node per night.
+      period: 1h
+
+      # ── THE SENTINEL ─────────────────────────────────────────────────────────────────────────
+      # Written ONLY by a deliberate operator kernel upgrade (see the header). `update-notifier-common`
+      # — the package that owns /etc/kernel/postinst.d and actually creates this file on Debian — was
+      # measured ABSENT on all three nodes on 2026-10-17, which is why the unattended_upgrades role
+      # installs it. Without that package nothing ever writes this path, kured correctly never
+      # reboots, every probe looks healthy, and the entire mechanism is a silent no-op. That failure
+      # mode is the reason KuredNotReporting exists.
+      #
+      # THE PATH kured LOGS IS NOT THIS PATH, and that trips people debugging it. The chart
+      # defaults `useRebootSentinelHostPath: true` (kept), which bind-mounts the DIRNAME of this
+      # value onto /sentinel read-only and renders the flag as
+      # `--reboot-sentinel=/sentinel/reboot-required` (daemonset.yaml:134). So the container watches
+      # /sentinel/reboot-required and the host file is /var/run/reboot-required. Same file, two
+      # names. Set this value as the HOST path, as below.
+      rebootSentinel: /var/run/reboot-required
+
+      # ── THE LONGHORN GATE: WHY POD READINESS IS NOT A SUFFICIENT INTER-NODE SIGNAL ────────────
+      # Measured on prod 2026-10-17, and this is the finding that shaped this file:
+      #   * 25 Longhorn volumes, ALL `numberOfReplicas: 3`, all healthy, with exactly 25 replicas on
+      #     each of the 3 nodes. So there is no single-replica volume to lose today.
+      #   * BUT `replica-soft-anti-affinity: false` (replicas may not share a node) and
+      #     `disable-scheduling-on-cordoned-node: true`. With 3 nodes and 3 replicas, cordoning ONE
+      #     node leaves nowhere to place a replacement — so every drain takes ALL 25 volumes to
+      #     degraded 2/3 for the whole reboot.
+      #   * Recovery is slow and serialized: `replica-replenishment-wait-interval: 600` (10 minutes
+      #     before a rebuild even starts) and `concurrent-replica-rebuild-per-node-limit: 1`.
+      # Therefore: if the next node is rebooted before rebuilds finish, those volumes go to 1/3, and
+      # the drain after that genuinely CAN hit Longhorn's `node-drain-policy:
+      # block-if-contains-last-replica` and block forever. The inter-node gate must be VOLUME
+      # ROBUSTNESS, not "are the pods Ready again".
+      #
+      # It is gated twice, on purpose, because each half covers the other's blind spot:
+      #   1. lockReleaseDelay: 30m (above) — unconditional. Covers the 600s replenishment wait with
+      #      margin and does not depend on any alert having fired.
+      #   2. The Prometheus alert gate below — semantic. Covers a rebuild that takes LONGER than 30
+      #      minutes (25 serialized rebuilds on one node can), which a fixed delay cannot.
+      prometheusUrl: http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090
+      # Only consider alerts that are FIRING, not merely pending. A `pending` alert is a 10-minute
+      # guess; blocking a weekly reboot on one would make the gate fire on noise.
+      alertFiringOnly: true
+      # ── alertFilterMatchOnly: true IS LOAD-BEARING. DO NOT DROP IT. ──────────────────────────
+      # kured's two alert modes are near-opposites and the flag name does not advertise it:
+      #   * false (the chart default): alertFilterRegexp is an IGNORE list — kured blocks on ANY
+      #     firing alert except those matching.
+      #   * true: alertFilterRegexp is an ALLOW list — kured blocks ONLY on alerts that match.
+      # We need `true`, because this cluster always has something firing. Measured via the Prometheus
+      # API on 2026-10-17, these five were firing with no reboot pending and no end in sight:
+      # CNPGPrimariesCoLocated, KubeDeploymentReplicasMismatch, KubeDeploymentRolloutStuck,
+      # KubePodNotReady, OAuth2ProxyGateUnobserved (the last three all trace to the pm-agent pod,
+      # ImagePullBackOff on a placeholder digest for 12 days). Under the default `false`, kured would
+      # have been blocked by those on day one and would never have rebooted anything — a silent
+      # no-op indistinguishable, from outside, from "no reboots were needed".
+      alertFilterMatchOnly: true
+      # The ALLOW list: the conditions under which rebooting ANOTHER node risks data or quorum.
+      # Anchored and explicit — a prefix match here would silently re-admit the chronic five above.
+      #   LonghornVolumeDegraded  — the inter-node gate (see above). `longhorn_volume_robustness
+      #                             {state="degraded"} == 1`, for: 10m, in platform/monitoring/alerts.yaml.
+      #   LonghornVolumeFaulted   — data already at risk; never reboot into this.
+      #   KubeNodeNotReady        — a node is already down. Rebooting a second one loses quorum. This
+      #                             is the single most important entry for the etcd constraint.
+      #   NodeExporterDown        — a node we cannot see the health of. Treated as "assume the worst".
+      # DELIBERATELY NOT INCLUDED: NodeEtcdRequestsSlow and the NodeIOPressure* family. They are
+      # exactly the conditions you would want to pause for, but prod-1 has a documented history of
+      # etcd WAL I/O starvation, so including them risks a permanent wedge that looks like safety.
+      # Revisit once those are quiet for a sustained period; KuredRebootPending makes a wedge visible
+      # either way.
+      alertFilterRegexp: ^(LonghornVolumeDegraded|LonghornVolumeFaulted|KubeNodeNotReady|NodeExporterDown)$
+
+      # ── THE DRAIN ────────────────────────────────────────────────────────────────────────────
+      # NOTE ON FLAGS THAT DO NOT EXIST: there is no `--ignore-daemonsets`, no
+      # `--delete-emptydir-data` and no `--force` in kured 1.23.0. Read from
+      # cmd/kured/main.go:438-449 on 2026-10-17, kured HARDCODES
+      # `IgnoreAllDaemonSets: true`, `DeleteEmptyDirData: true` and `Force: true` in the drain
+      # helper. Passing them as extraArgs would CrashLoop the DaemonSet on an unknown flag. This
+      # matters because all three are genuinely REQUIRED here and it is easy to assume they must be
+      # configured: 12 DaemonSets and 135 emptyDir-carrying pods were measured on 2026-10-17, and a
+      # drain without those three settings refuses to start.
+      #
+      # drainTimeout: 15m — the chart/kured default is 0, meaning INFINITE, and infinite is the wrong
+      # answer here. Measured 2026-10-17, six PodDisruptionBudgets currently allow ZERO voluntary
+      # disruptions, and a plain drain loops on them indefinitely:
+      #   * 3x longhorn-system/instance-manager-* — one per node, `minAvailable: 1` against a
+      #     selector pinned to a single node's single pod, so disruptionsAllowed is STRUCTURALLY 0
+      #     forever. This is the mechanism by which Longhorn's node-drain-policy blocks: the
+      #     longhorn-manager deletes the node's instance-manager PDB only once it judges the drain
+      #     safe. The drain is SUPPOSED to wait on this one.
+      #   * 3x CNPG `<cluster>-primary` (keycloak-db, nextcloud-db, semaphore-db) — `minAvailable: 1`
+      #     against a selector matching exactly one pod (the primary), so again structurally 0 until
+      #     the CNPG operator switches the primary away. All three primaries were on k3s-prod-1.
+      # With a timeout, a drain that is genuinely stuck ABORTS, the lock is released, and kured
+      # retries next period instead of holding the cluster lock forever on a wedged node. 15m is
+      # chosen to be longer than a CNPG switchover and longer than the `unifi` VM's eviction (below)
+      # usually needs, while still being well inside the 3h window.
+      drainTimeout: 15m
+      # Taint a node PreferNoSchedule while its reboot is pending, so the scheduler stops placing new
+      # pods on a node that is about to go away. With concurrency 1 this is a small win (there is no
+      # second rebooting node to shed pods onto) but it is free and it shortens the drain.
+      preferNoScheduleTaint: weave.works/kured-node-reboot
+      # Record reboot state on the Node objects (`weave.works/kured-reboot-in-progress`,
+      # `weave.works/kured-most-recent-reboot-needed`). Off by default. Turned on because the only
+      # other observability kured offers is ONE gauge, and during an incident `kubectl get node -o
+      # yaml` is the thing an operator can reach without Prometheus.
+      annotateNodes: true
+      logFormat: json
+
+    # ── METRICS: BOTH OF THESE MUST BE true OR THE ALERTS IN alerts.yaml CANNOT FIRE ───────────
+    # The chart ships `service.create: false` AND `metrics.create: false`. Leaving either at its
+    # default means no Service, or a ServiceMonitor with nothing to select, so `kured_reboot_required`
+    # is never scraped — and a PrometheusRule over a series that no target produces is INERT: quiet
+    # forever, indistinguishable from healthy. This repo has shipped that exact bug twice
+    # (VeleroNoRecentBackup over an absent series; the inert gotk_* rules in
+    # runbooks/image-automation.md), which is why platform/monitoring/tests/run.sh exists at all.
+    # KuredNotReporting (alerts.yaml) is the standing detector for this pair regressing.
+    # Prometheus selects ServiceMonitors cluster-wide: measured 2026-10-17, `serviceMonitorSelector`
+    # and `serviceMonitorNamespaceSelector` are both `{}`, so no label plumbing is needed here.
+    service:
+      create: true
+      port: 8080
+    metrics:
+      create: true
+      interval: 60s

platform/kured/helmrepository.yaml

@@ -0,0 +1,12 @@
+---
+# HelmRepository for the official kured chart (github.com/kubereboot/charts). Renovate maintains the
+# pinned chart version in helmrelease.yaml via its `flux` manager, whose managerFilePatterns already
+# cover `platform/**` (renovate.json). Same Flux source pattern as platform/headlamp.
+apiVersion: source.toolkit.fluxcd.io/v1
+kind: HelmRepository
+metadata:
+  name: kured
+  namespace: kured
+spec:
+  interval: 1h
+  url: https://kubereboot.github.io/charts

platform/kured/kustomization.yaml

@@ -0,0 +1,36 @@
+---
+# platform/kured — the Kubernetes Reboot Daemon.
+#
+# Rendered by its OWN Flux Kustomization (clusters/base/kured.yaml, dependsOn platform), NOT by a
+# line in platform/kustomization.yaml. The reasoning is the same one the Cilium split-out wrote down
+# and intel-gpu-plugin reused: inside the shared `wait: true` platform tier, this component's health
+# would be indistinguishable from eleven others, the only emergency valve would be suspending the
+# whole tier, and that tier's 5m timeout is wrong for a 3-node DaemonSet rollout. It matters more
+# than usual here, because `platform` gates platform-config which gates every apps wave — so a wedged
+# reboot daemon placed there could stall every application in the cluster. A component whose entire
+# purpose is to make maintenance safer must not be able to cause an outage.
+#
+# `namespace: kured` applies to every resource below, so none of them repeats it. Keep it: the
+# CiliumNetworkPolicy in networkpolicy.yaml is a CR whose namespace kustomize would otherwise leave
+# to chance.
+#
+# NO SECRET OF ANY KIND. kured authenticates to the API server with its own ServiceAccount token and
+# talks to nothing off-cluster — no registry credential, no webhook URL, no notification token. So
+# neither clusters/dev nor clusters/prod adds a SOPS decryption patch for this tier (same as
+# homepage/runbooks/intel-gpu-plugin). If Slack or shoutrrr notifications are ever wanted, the chart
+# takes them via `configuration.notifyUrl`, and THAT would need a SOPS-encrypted Secret plus a
+# decryption patch — do not inline a webhook URL here.
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: kured
+resources:
+  - namespace.yaml
+  - helmrepository.yaml
+  - helmrelease.yaml     # chart installs the DaemonSet / RBAC / SA / Service / ServiceMonitor
+  # default-deny baseline + the Prometheus scrape ingress + the apiserver/Prometheus egress.
+  # The scrape ingress is NOT optional decoration: without it the ServiceMonitor's targets are
+  # discovered and every scrape times out, leaving the TSDB empty and all three rules in alerts.yaml
+  # permanently quiet. See the header in that file for the measured Longhorn precedent.
+  - networkpolicy.yaml
+  # Three rules over kured's single metric, with a companion promtool unit test beside them.
+  - alerts.yaml

platform/kured/namespace.yaml

@@ -0,0 +1,34 @@
+---
+# kured — namespace for the Kubernetes Reboot Daemon DaemonSet.
+#
+# PLATFORM INFRA, NOT AN APPLICATION NAMESPACE, and here that distinction is the difference between
+# a working component and an admission rejection. kured runs `hostPID: true` with a `privileged: true`
+# container (it `nsenter`s /proc/1/ns/mnt to run `/bin/systemctl reboot` on the host). Both are
+# forbidden by platform/kyverno/policies/restrict-host-namespaces.yaml — but that policy, like every
+# other policy in this repo, is scoped by
+#   `namespaceSelector: matchLabels: homelab.keithjackson.dev/app-namespace: "true"`
+# and its own header states the carve-out: "platform components that do [need these] live in platform
+# namespaces that do NOT carry the app-namespace label, so they are out of scope."
+# Verified by reading the policies on this branch, not assumed:
+#   * restrict-host-namespaces.yaml rule `disallow-host-namespaces`     (hostPID)
+#   * restrict-host-namespaces.yaml rule `disallow-privileged-containers` (privileged)
+#
+# We set the label EXPLICITLY to "false" rather than leaving it unset, following
+# platform/intel-gpu-plugin/namespace.yaml and platform/vpn-egress: an unset label and a deliberate
+# "false" are identical to the admission engine but not to a reviewer, and this component's
+# correctness depends entirely on the answer.
+#
+# NO PodSecurity labels. There is no cluster-wide PSA baseline in this repo — `pod-security.*`
+# appears only where a namespace opts IN (conformance/incluster, platform/kyverno/admission,
+# platform/pm-agent), all at `enforce: restricted`. Adding them here would reject this DaemonSet for
+# the same reasons Kyverno would. Leaving them off is the convention for a privileged platform
+# namespace (platform/vpn-ingress/namespace.yaml sets none either).
+#
+# NO `gateway-access` label. kured serves no HTTP to users and attaches no HTTPRoute; its only
+# listener is the :8080 metrics endpoint, reached in-cluster by Prometheus. See networkpolicy.yaml.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: kured
+  labels:
+    homelab.keithjackson.dev/app-namespace: "false"

platform/kured/networkpolicy.yaml

@@ -0,0 +1,125 @@
+---
+# Default-deny baseline (R-NET-01): deny all ingress + egress, allow only DNS. `kured` is a PLATFORM
+# namespace, so Kyverno's generator — scoped to `app-namespace=true` — does not produce this; we ship
+# it explicitly, as crafty/headlamp/homepage/runbooks all do.
+#
+# THESE POLICIES DO APPLY TO kured, UNLIKE platform/vpn-ingress'. That component's identical-looking
+# baseline is honest-but-inert because its DaemonSet runs `hostNetwork: true` and therefore has no
+# Cilium endpoint for any policy to select. kured's chart defaults to `hostNetwork: false` and we
+# keep that default (it needs the host PID namespace to reboot, not the host network stack), so it
+# has a normal pod identity and every rule here is enforced.
+#
+# Cilium enforces the UNION of these policies (allow if ANY policy allows), so the CiliumNetworkPolicy
+# below adds the kube-apiserver path on top of this deny-all floor.
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: default-deny
+  namespace: kured
+  labels:
+    homelab.keithjackson.dev/baseline: default-deny
+spec:
+  podSelector: {}
+  policyTypes:
+    - Ingress
+    - Egress
+  ingress: []
+  egress:
+    - to:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: kube-system
+      ports:
+        - protocol: UDP
+          port: 53
+        - protocol: TCP
+          port: 53
+---
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# PROMETHEUS MUST REACH :8080, OR EVERY RULE IN alerts.yaml IS INERT.
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# This is not a theoretical hazard in this repo, it is a measured one. From the Longhorn entry in
+# conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh: with the ServiceMonitor
+# applied and no NetworkPolicy allowance, all three targets were DISCOVERED and every scrape failed
+# with "context deadline exceeded" after 10.0s — "a drop, not a refusal" — and the TSDB stayed
+# empty. Target discovery succeeding is therefore NOT evidence the scrape works, and the alerts
+# would have been quiet rather than broken. KuredNotReporting is the standing detector for exactly
+# this regression, and it can only work if this policy is correct.
+#
+# THE WHOLE `monitoring` NAMESPACE, NOT THE PROMETHEUS POD LABELS — following the same choice
+# longhorn/networkpolicy-metrics.yaml made and for the same reason: the prometheus-operator owns
+# those pod labels and rewrites them on upgrade (measured 2026-10-17, the pod carries
+# `app.kubernetes.io/name: prometheus` plus five operator-managed labels including a
+# controller-revision-hash), so selecting on them makes the scrape silently fail on a chart bump.
+# The namespace holds only the monitoring stack.
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: kured-allow-prometheus-scrape
+  namespace: kured
+  labels:
+    app.kubernetes.io/name: kured
+spec:
+  podSelector:
+    matchLabels:
+      # Set by the chart's `kured.matchLabels` helper (_helpers.tpl:70-71) with release name `kured`.
+      app.kubernetes.io/name: kured
+  policyTypes:
+    - Ingress
+  ingress:
+    - from:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: monitoring
+      ports:
+        - protocol: TCP
+          # The container port the chart names `metrics` (daemonset.yaml:211-213), from
+          # `configuration.metricsPort`, which we leave at the chart default of 8080 and mirror in
+          # the HelmRelease's `service.port`.
+          port: 8080
+---
+# Egress allows. kured talks to exactly two things and nothing else; both are required and the
+# component is useless without either.
+#
+#   1. THE KUBE-APISERVER — the lock annotation on its own DaemonSet (`weave.works/kured-node-lock`),
+#      cordon/uncordon (`nodes` patch), the node annotations, listing pods, and creating
+#      `pods/eviction` objects during the drain. Reached via the `kube-apiserver` Cilium entity
+#      rather than an IP: on this cluster the API is fronted by a kube-vip VIP that moves between
+#      nodes on every reboot, so any CIDR-based rule here would break during the exact operation
+#      this component exists to perform. Same construct as platform/headlamp/networkpolicy.yaml.
+#
+#   2. PROMETHEUS, on 9090 — the alert gate. The HelmRelease sets
+#      `prometheusUrl: http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090`
+#      and `alertFilterMatchOnly: true`, so before taking the cluster lock kured asks Prometheus
+#      whether any of LonghornVolumeDegraded / LonghornVolumeFaulted / KubeNodeNotReady /
+#      NodeExporterDown is firing. IF THIS EGRESS IS BLOCKED, kured cannot evaluate the gate.
+#      It treats the query error as "do not proceed" and skips the reboot — which is the correct
+#      fail-safe direction, but it fails SILENTLY and indefinitely: reboots simply stop happening
+#      and nothing goes red except, eventually, KuredRebootPending. Do not remove this rule
+#      thinking it is only for metrics; metrics ingress is the policy above, this is the gate.
+#
+# NOTHING ELSE. No egress to the internet (kured pulls no content at runtime), no DNS beyond the
+# baseline above, no Longhorn or etcd access — kured reads Longhorn's health THROUGH Prometheus
+# alerts, never from Longhorn's own API, which is why that allow is absent here.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: kured-allow-egress
+  namespace: kured
+  labels:
+    app.kubernetes.io/name: kured
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: kured
+  egress:
+    - toEntities:
+        - kube-apiserver
+    - toEndpoints:
+        - matchLabels:
+            io.kubernetes.pod.namespace: monitoring
+            app.kubernetes.io/name: prometheus
+      toPorts:
+        - ports:
+            - port: "9090"
+              protocol: TCP

platform/monitoring/node-pressure-dashboard.yaml

@@ -199,7 +199,7 @@ data:
           "id": 15,
           "type": "timeseries",
           "title": "CAUSE 4 — NFS RPC retransmission ratio, per node",
-          "description": "node_nfs_rpc_retransmissions_total / node_nfs_rpcs_total, from /proc/net/rpc/nfs. Alert line at 1% (NodeNFSRetransmissionRateHigh). MEASURED BASELINE IS EXACTLY ZERO on all three nodes over 48h, including throughout the 2026-09-14 spike — so any visible line here means the export at 192.168.2.102, or the path to it, is dropping requests. A flat zero does NOT clear NFS: it only clears the TRANSPORT. A merely SLOW export needs per-mount RPC round-trip time, which arrives with the mountstats collector (see kube-prometheus-stack.yaml) and is the known gap recorded in runbooks/node-io.md.",
+          "description": "node_nfs_rpc_retransmissions_total / node_nfs_rpcs_total, from /proc/net/rpc/nfs. Alert line at 1% (NodeNFSRetransmissionRateHigh). MEASURED BASELINE IS EXACTLY ZERO on all three nodes over 48h, including throughout the 2026-09-14 spike — so any visible line here means the export at 192.168.2.30, or the path to it, is dropping requests. A flat zero does NOT clear NFS: it only clears the TRANSPORT. A merely SLOW export needs per-mount RPC round-trip time, which arrives with the mountstats collector (see kube-prometheus-stack.yaml) and is the known gap recorded in runbooks/node-io.md.",
           "gridPos": {"h": 8, "w": 12, "x": 12, "y": 39},
           "fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [
             {"color": "green", "value": null}, {"color": "red", "value": 0.01}

platform/oauth2-proxy/helmrelease.yaml

@@ -63,6 +63,75 @@ spec:
       # gets oauth2-proxy's HTML sign-in page (HTTP 403) instead of an immediate redirect (docs/15 UX).
       skip-provider-button: "true"
       scope: "openid email profile"
+    # ── Cold-start gate: do not start the proxy until Keycloak's discovery doc answers ──────────
+    # MEASURED on prod 2026-10-17, during the UPS move: oauth2-proxy performs OIDC discovery ONCE at
+    # startup and EXITS 1 if it fails. On a whole-cluster cold boot Keycloak is not serving yet, so
+    # the Gateway answers 503 and the proxy dies:
+    #
+    #   [provider.go:55] Performing OIDC Discovery...
+    #   [main.go:73] ERROR: Failed to initialise OAuth2 Proxy: ... failed to discover OIDC
+    #                configuration: unexpected status "503": no healthy upstream
+    #
+    # It then sits in exponential CrashLoopBackOff long after Keycloak is healthy -- one replica was
+    # still down minutes later and only a manual pod delete cleared it. There is no retry flag; the
+    # process simply does not survive a cold start. A readinessProbe cannot help (the container is
+    # already dead), and CrashLoopBackOff backoff is not tunable per-pod.
+    #
+    # `extraInitContainers` is the chart's OWN documented hook for this -- its commented example in
+    # values.yaml is literally `- name: wait-for-idp` -- and the chart already uses an alpine init
+    # container for its built-in waitForRedis gate, so this adds no new base image to the platform.
+    # A pod in Init does not accrue CrashLoopBackOff backoff and retries promptly, which is the whole
+    # point.
+    #
+    # BOUNDED, AND FAILS OPEN. After ~5 minutes it gives up and starts the proxy anyway: the worst
+    # case is then exactly today's behaviour, never something new. An unbounded wait would trade a
+    # CrashLoop for a pod wedged in Init forever, which is not an improvement.
+    #
+    # SHELL VARIABLES ARE WRITTEN BARE ($i, $url, $(( ))), WITH NO BRACED FORM ANYWHERE. This file is
+    # rendered through Flux envsubst (platform declares postBuild.substituteFrom cluster-vars), which
+    # CONSUMES braced expansions -- so a braced shell variable renders to the empty string and
+    # silently breaks the loop. BASE_DOMAIN below is braced ON PURPOSE: it is a real cluster-vars
+    # knob and must be substituted. Same hazard, and the same convention, as the gitea and nextcloud
+    # OIDC bootstrap hooks. (ci: flux-postbuild-substitution enforces it.)
+    extraInitContainers:
+      - name: wait-for-idp
+        image: docker.io/library/alpine:3.22
+        imagePullPolicy: IfNotPresent
+        command: ["/bin/sh", "-c"]
+        args:
+          - |
+            set -u
+            url=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
+            i=0
+            while [ "$i" -lt 60 ]; do
+              # busybox wget validates against alpine's ca-certificates-bundle, so a real TLS
+              # failure is a non-zero exit here rather than a false pass.
+              if wget -q -O /dev/null --timeout=5 "$url" 2>/dev/null; then
+                echo "[wait-for-idp] Keycloak discovery reachable; starting oauth2-proxy"
+                exit 0
+              fi
+              i=$(( i + 1 ))
+              sleep 5
+            done
+            echo "[wait-for-idp] discovery still unreachable after ~5m; starting anyway so this" >&2
+            echo "[wait-for-idp] gate can never be worse than the un-gated behaviour it replaces" >&2
+            exit 0
+        securityContext:
+          allowPrivilegeEscalation: false
+          runAsNonRoot: true
+          runAsUser: 65534
+          runAsGroup: 65534
+          readOnlyRootFilesystem: true
+          capabilities:
+            drop:
+              - ALL
+        resources:
+          requests:
+            cpu: 10m
+            memory: 16Mi
+          limits:
+            memory: 32Mi
+
     # Metrics for the ServiceMonitor (../oauth2-proxy servicemonitor in platform-config).
     metrics:
       enabled: true

platform/vpn-ingress/README.md

@@ -13,7 +13,7 @@ egress out the house WAN.
 | Transport | WireGuard, **udp/51820**, kernel implementation |
 | Where it runs | `hostNetwork` **DaemonSet** — one warm listener on every node |
 | Client subnet | `${WG_CLIENT_CIDR}` = `10.90.0.0/24`; `.1` is the server, `.2`–`.254` devices |
-| Authorisation | a device's **public key** is in [`peers.yaml`](./peers.yaml), or it cannot connect |
+| Authorisation | a device's **public key** is in [`peers.conf`](./peers.conf), or it cannot connect |
 | The one secret | the server's private key — **SOPS-encrypted in this tree**, `wg-server-key.sops.yaml` |
 | Exposure | one hand-made WAN port-forward on the SOHO router, off-repo |
 | Enrolling a device | [`infra/vpn-ingress/new-peer.sh`](../../infra/vpn-ingress/new-peer.sh) |
@@ -129,7 +129,7 @@ translations win here.
 
 | half | where | why |
 |---|---|---|
-| peer **public** keys + addresses | [`peers.yaml`](./peers.yaml), **plaintext in git** | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
+| peer **public** keys + addresses | [`peers.conf`](./peers.conf), **plaintext in git**, as a content-hashed `configMapGenerator` so an edit rolls the pods | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
 | the server's **private** key | `wg-server-key.sops.yaml`, **SOPS-encrypted in git** | the only real secret (R-SEC-01, spec docs/07). Ships with a `REPLACE-ME` placeholder: an agent must not mint a long-lived private key (CLAUDE.md §6), so the operator `sops set`s it in |
 
 The two are joined only at pod start, by the init container, into a **`medium: Memory` emptyDir** —
@@ -166,13 +166,26 @@ NetworkPolicy applies.
 ## Operating it
 
 * **Enrol a device:** `./infra/vpn-ingress/new-peer.sh <name> <octet>` → paste the printed `[Peer]`
-  stanza into `peers.yaml` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
-* **Revoke a device:** delete its stanza → PR → merge. No other step; the key stops working.
+  stanza into `peers.conf` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
+* **Revoke a device:** delete its stanza from `peers.conf` → PR → merge. No other step; the key
+  stops working as soon as the pods roll (the generator's content hash guarantees they do).
 * **Is it up?** `kubectl --context k3s-prod -n vpn-ingress get ds wireguard` — three Ready.
 * **Who is connected?** `kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show`
   (`latest handshake` under ~3 min means live). Never dump `wg showconf` — it prints the private key.
 * **Full bring-up and verification:** `runbooks/vpn-remote-access.md`.
 
+## Measured on prod, 2026-10-06 — what is now known rather than assumed
+
+The first real reconcile settled four things this component's design rested on. Three went the
+right way; one was a defect in my own manifest.
+
+| question | answer |
+|---|---|
+| Does the **kernel module** reach the pod? | **Yes.** `ip link add dev wg0 type wireguard` succeeds, and `/proc/modules` lists `wireguard`, after the `node_baseline` run. `CAP_SYS_MODULE` is correctly not needed. |
+| Does `allowPrivilegeEscalation: false` break the image's **s6 init**? | **No.** The container reaches `wg-quick` normally. The flag stays. |
+| Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
+| Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |
+
 ## Prerequisites outside this directory
 
 1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`
@@ -199,7 +212,7 @@ NetworkPolicy applies.
   covered generically by `KubeDeploymentReplicasMismatch`/`KubePodNotReady` from
   kube-prometheus-stack. **This is the top follow-up.**
 * **No preshared keys.** They would add a symmetric post-quantum hedge on top of the Noise
-  handshake — and would be a real secret, forcing `peers.yaml` into SOPS and costing the
+  handshake — and would be a real secret, forcing `peers.conf` into SOPS and costing the
   reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the
   `wg-server-key` Secret.
 * **No IPv6.** The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6

platform/vpn-ingress/daemonset.yaml

@@ -37,6 +37,10 @@
 #     /etc/sysctl.d/99-k3s-cilium.conf), so no pod sysctl is needed — and `net.ipv4.ip_forward`
 #     is NOT in the kubelet's safe-sysctl allowlist, so a pod-netns version of this would have
 #     required `--allowed-unsafe-sysctls` on all three kubelets (a node-restart change).
+#     COROLLARY LEARNED THE HARD WAY (prod, 2026-10-06): this pod must not merely avoid NEEDING
+#     the sysctl, it must not ATTEMPT it. hostNetwork implies /proc/sys is mounted read-only, so
+#     `sysctl -w` fails with EPERM regardless of capabilities, and wg-quick treats a failed PostUp
+#     as fatal and deletes the interface. Verify the value; never write it.
 #   * no LoadBalancer VIP, so no CiliumL2AnnouncementPolicy lease lottery (the defect that cost
 #     this LAN an 8-day Minecraft outage) and no `externalTrafficPolicy` trade-off at all.
 #   * no double encapsulation. WireGuard's UDP lands on the node's own NIC, so the underlay is a
@@ -104,7 +108,7 @@ spec:
       dnsPolicy: ClusterFirstWithHostNet
       automountServiceAccountToken: false
       # ── Assemble the running config: PUBLIC peer register (git) + PRIVATE server key (Secret) ──
-      # The two halves are deliberately stored apart (peers.yaml explains why) and are only ever
+      # The two halves are deliberately stored apart (peers.conf explains why) and are only ever
       # joined here, at pod start, into a tmpfs that is destroyed with the pod. The server's private
       # key is therefore never written to any node's disk, never committed, and never logged.
       initContainers:
@@ -171,6 +175,19 @@ spec:
                 exit 1
               fi
 
+              # The pod shares the HOST netns, so this reads the node's real setting. Reading
+              # /proc/sys is permitted even though writing is not (see the PostUp note below).
+              # Fail closed: a tunnel that comes up but cannot forward would look healthy to every
+              # probe here and simply drop every client's traffic.
+              if ! grep -qx 1 /proc/sys/net/ipv4/ip_forward; then
+                echo "FATAL: net.ipv4.ip_forward is 0 on this node, so forwarded VPN traffic would" >&2
+                echo "       be dropped. This pod CANNOT set it: with hostNetwork the runtime mounts" >&2
+                echo "       /proc/sys read-only, because it is the host's global setting." >&2
+                echo "       Fix on the NODE: run the node_baseline role (it writes" >&2
+                echo "       /etc/sysctl.d/99-k3s-cilium.conf) — runbooks/vpn-remote-access.md step 1." >&2
+                exit 1
+              fi
+
               {
                 echo '[Interface]'
                 echo "Address = $base.1/$mask"
@@ -184,9 +201,20 @@ spec:
                 echo 'MTU = 1420'
                 printf 'PrivateKey = %s\n' "$key"
                 # ── Forwarding + SNAT, applied in the HOST netns when wg0 comes up ──
-                # ip_forward is already 1 from node_baseline; re-asserted so this component does not
-                # silently depend on another role's sysctl file surviving.
-                echo 'PostUp = sysctl -q -w net.ipv4.ip_forward=1'
+                # NO `sysctl -w net.ipv4.ip_forward=1` HERE. It is IMPOSSIBLE from this pod and it was
+                # the only thing stopping the tunnel coming up — MEASURED on prod 2026-10-06:
+                #     [#] sysctl -q -w net.ipv4.ip_forward=1
+                #     sysctl: permission denied on key "net.ipv4.ip_forward"
+                #     [#] ip link delete dev wg0
+                # and wg-quick then tore the whole tunnel down. The cause is not a missing
+                # capability: with `hostNetwork: true` the runtime mounts /proc/sys READ-ONLY in the
+                # container (confirmed in /proc/mounts: `proc /proc/sys proc ro,...`), precisely
+                # because those sysctls are the HOST's globals and a container must not rewrite
+                # them. CAP_NET_ADMIN does not and cannot lift a read-only mount.
+                # It was never needed anyway: node_baseline writes net.ipv4.ip_forward=1 to
+                # /etc/sysctl.d/99-k3s-cilium.conf, and reading it from this pod returns 1. The
+                # init container now VERIFIES that (a read, which /proc/sys does allow) and fails
+                # closed with a pointed message, which is what the write was pretending to achieve.
                 # SNAT client traffic to the node's address. REQUIRED: without it a LAN host would
                 # reply to 10.90.0.x, find no route, hand the packet to the router, and the router
                 # would drop it — so every connection would hang rather than fail. Matching on
@@ -246,8 +274,10 @@ spec:
             # `kubectl get pod -o wide` shows what this listener owns.
             - { name: wireguard, containerPort: 51820, hostPort: 51820, protocol: UDP }
           securityContext:
-            # NET_ADMIN is the whole privilege budget: create the wg0 link, set the host sysctl, and
-            # write the two netfilter rules above. Everything else is dropped, and the container is
+            # NET_ADMIN is the whole privilege budget: create the wg0 link and write the netfilter
+            # rules above. (It does NOT let the pod set a host sysctl — /proc/sys is read-only under
+            # hostNetwork — which is why nothing here tries.) Everything else is dropped, and the
+            # container is
             # NOT privileged — which also keeps it clear of restrict-host-namespaces' second rule.
             # SYS_MODULE is deliberately NOT granted: the `wireguard` module is loaded by the node
             # (infra/ansible/roles/node_baseline, node_baseline_kernel_modules), so this container
@@ -261,6 +291,16 @@ spec:
                 - NET_ADMIN
             # readOnlyRootFilesystem intentionally UNSET: the image's s6 init writes under /run and
             # /var during startup. Same deferral as crafty/headlamp.
+            #
+            # EXPECTED, HARMLESS STARTUP NOISE — do not "fix" it. Because capabilities are dropped
+            # to NET_ADMIN only, the image's s6 init cannot chown /config (no CAP_CHOWN) and logs a
+            # run of `chown: ... Operation not permitted` plus "Permissions could not be set. This
+            # is probably because your volume mounts are remote or read-only" and a scary "we will
+            # not provide support for it". It is wrong about the cause and it does not matter:
+            # /config is a writable tmpfs emptyDir, everything here runs as root (uid 0) anyway, and
+            # wg-quick proceeds normally — verified on prod 2026-10-06, where the tunnel's only
+            # failure was the sysctl above, well past these warnings. Granting CAP_CHOWN to silence
+            # log noise would widen the privilege set of an internet-facing pod for nothing.
           resources:
             requests:
               cpu: 25m

platform/vpn-ingress/kustomization.yaml

@@ -11,10 +11,16 @@
 # NOT in platform/kustomization.yaml — see clusters/base/vpn-ingress.yaml.
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
+# REQUIRED for the configMapGenerator below to be wired up at all, not merely tidy. A generated
+# ConfigMap inherits NO namespace, every other object here declares `vpn-ingress`, and kustomize's
+# nameReference transformer will not rewrite a reference across a namespace boundary — so without
+# this the DaemonSet keeps mounting the unhashed name `wg-peers`, which does not exist, and the
+# pods wedge on a missing volume. Verified by building it both ways (kustomize v5.8.1).
+# Same placement as runbooks/kustomization.yaml, which uses `namespace: runbooks` for its own
+# content generator.
+namespace: vpn-ingress
 resources:
   - namespace.yaml
-  # The peer register: which devices may dial in. PLAINTEXT and reviewable on purpose (see the file).
-  - peers.yaml
   # The listener itself: hostNetwork DaemonSet, udp/51820 on every node.
   - daemonset.yaml
   # Namespace default-deny baseline. NOTE: does NOT constrain the hostNetwork DaemonSet — the file
@@ -38,3 +44,26 @@ resources:
 # hands Flux the raw ciphertext, the Secret APPLIES CLEANLY full of ENC[...], and nothing in any
 # Flux condition looks wrong. The init container's `wg pubkey` guard exists partly to turn exactly
 # that into a named error rather than a silently non-working tunnel.
+
+# ── THE PEER REGISTER, as a CONTENT-HASHED ConfigMap ─────────────────────────────────────────
+# peers.conf (plaintext, reviewable — see the file) becomes the `wg-peers` ConfigMap. It is a
+# GENERATOR, not a plain resource, and that distinction is load-bearing rather than stylistic:
+# kustomize appends a hash of the content to the generated name and rewrites the DaemonSet's
+# volume reference to match, so adding or revoking a peer changes the pod template and ROLLS THE
+# PODS automatically.
+#
+# A fixed-name ConfigMap (which is what this component shipped first) does not do that. Flux
+# updates it in place, the pod template is byte-identical, no rollout is triggered — and because
+# the init container assembles wg0.conf ONCE at pod start into a tmpfs, the running tunnel keeps
+# the old peer list indefinitely. Measured on prod 2026-10-06: a merged peer reached the
+# ConfigMap while `wg show` listed no peers and the DaemonSet still read generation=1, having
+# never rolled. The runbook's "PR -> merge -> Flux rolls the DaemonSet" was false until this.
+#
+# DO NOT set `disableNameSuffixHash` here. The hash IS the mechanism.
+configMapGenerator:
+  - name: wg-peers
+    files:
+      - peers.conf
+    options:
+      labels:
+        app.kubernetes.io/name: vpn-ingress

platform/vpn-ingress/networkpolicy.yaml

@@ -19,7 +19,7 @@
 # protects this port, and it would be enforcing nothing. So there isn't one.
 #
 # WHAT ACTUALLY PROTECTS THE EXPOSED PORT, then, since the CNI does not:
-#   1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.yaml gets
+#   1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.conf gets
 #      NO REPLY — see README "Why this is safe to expose". This is the real control.
 #   2. The SOHO router's forward: only udp/51820 is forwarded inbound, to one node.
 #   3. The blast radius once authenticated is deliberately the whole LAN — that IS the requirement

platform/vpn-ingress/peers.conf

@@ -1,4 +1,16 @@
----
+# THIS FILE IS THE MOUNTED CONTENT. It is turned into the `wg-peers` ConfigMap by
+# `configMapGenerator` in kustomization.yaml, whose generated name carries a CONTENT HASH — so
+# editing this file changes the ConfigMap name, which changes the DaemonSet's pod template, which
+# ROLLS THE PODS. That is the whole reason it is a generator and not an inline ConfigMap manifest:
+# a fixed-name ConfigMap updates in place, the pod template never changes, no rollout happens, and
+# the init container's wg0.conf (built once at pod start, in tmpfs) keeps the OLD peer list
+# forever. Measured 2026-10-06: a merged peer sat in the ConfigMap while `wg show` listed none and
+# the DaemonSet reported generation=1 — it had never rolled. Same mechanism and same fix as
+# runbooks/kustomization.yaml.
+#
+# Every line here is WireGuard config. `#` is a legal comment in it, so the documentation below
+# travels inside the file that is actually mounted, rather than in a wrapper that gets stripped.
+#
 # ============================================================================================
 # THE PEER REGISTER — every device allowed to dial in, in PLAINTEXT, in git, on purpose.
 # ============================================================================================
@@ -41,33 +53,40 @@
 # the Noise handshake and would be a genuine secret — which would force this register into SOPS
 # and cost the reviewability above. Revisit only if the threat model changes; if it does, the PSK
 # belongs in the `wg-server-key` Secret, not here. See README "Hardening not taken".
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: wg-peers
-  namespace: vpn-ingress
-  labels:
-    app.kubernetes.io/name: vpn-ingress
-data:
-  # Appended VERBATIM to the generated [Interface] block, in this order, at pod start.
-  #
-  # *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. ***
-  # An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of
-  # authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the
-  # DaemonSet will run and listen, and every handshake will be refused because no key is trusted.
-  # The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible
-  # (`wg show` reports an interface with no peers).
-  #
-  # COPY THIS TEMPLATE, uncomment, one block per device:
-  #
-  #   # pixel-8 — Keith's phone, enrolled 2026-10-05
-  #   [Peer]
-  #   PublicKey = <the public key new-peer.sh printed>
-  #   AllowedIPs = 10.90.0.2/32
-  #
-  # Keep the `# <name> — <owner>, enrolled <date>` comment line. It is the only place a key is
-  # tied to a human-recognisable device; `wg show` can only ever print the key itself, so without
-  # it an incident responder cannot tell which stanza to delete.
-  peers.conf: |
-    # No peers enrolled yet. See the template in this file's comments and
-    # ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device.
+#
+# name: wg-peers
+# namespace: vpn-ingress
+# labels:
+# app.kubernetes.io/name: vpn-ingress
+# Appended VERBATIM to the generated [Interface] block, in this order, at pod start.
+#
+# *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. ***
+# An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of
+# authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the
+# DaemonSet will run and listen, and every handshake will be refused because no key is trusted.
+# The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible
+# (`wg show` reports an interface with no peers).
+#
+# COPY THIS TEMPLATE, uncomment, one block per device:
+#
+#   # <device-name> — <owner>, enrolled <YYYY-MM-DD>
+#   [Peer]
+#   PublicKey = <the public key new-peer.sh printed>
+#   AllowedIPs = 10.90.0.<octet>/32
+#
+# The placeholders above are deliberately NOT valid values. An earlier version of this template
+# used a real-looking `10.90.0.2/32`, and new-peer.sh's duplicate-address check — which grepped
+# the whole file — matched it, so `.2` could never be allocated and the script insisted it was
+# already taken. The check now ignores comments; keeping the example unmatchable too means a
+# future grep cannot be fooled either.
+#
+# Keep the `# <name> — <owner>, enrolled <date>` comment line. It is the only place a key is
+# tied to a human-recognisable device; `wg show` can only ever print the key itself, so without
+# it an incident responder cannot tell which stanza to delete.
+#
+# No peers enrolled yet. See the template in this file's comments and
+# ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device.
+# keith-phone — <owner>, enrolled 2026-10-06
+[Peer]
+PublicKey = pBRSHrRyG2hnsuZ6FvHfEzF8hOVTF5stHUb/7fW2Km8=
+AllowedIPs = 10.90.0.2/32

platform/vpn-ingress/wg-server-key.sops.yaml.example

@@ -5,7 +5,7 @@
 # WHAT THIS IS: the WireGuard SERVER's long-term private key for the remote-access VPN
 # (platform/vpn-ingress). It is the ONE secret in the whole component. The init container in
 # daemonset.yaml reads it at `/run/wg-server-key/privatekey` and joins it to the PLAINTEXT peer
-# register (peers.yaml) in a tmpfs, so the assembled wg0.conf never touches a node's disk.
+# register (peers.conf) in a tmpfs, so the assembled wg0.conf never touches a node's disk.
 #
 # WHY IT IS IN GIT, SOPS-ENCRYPTED, AND NOT HAND-CREATED. R-SEC-01 / spec docs/07: secrets live in
 # source, encrypted, and reach the cluster only through Flux. An out-of-band `kubectl create secret`

renovate.json

@@ -1,6 +1,6 @@
 {
   "$schema": "https://docs.renovatebot.com/renovate-schema.json",
-  "_comment": "Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). Renovate owns HELM CHART + DEPENDENCY bumps (opens PRs; allowed classes auto-merge on green CI). Flux image-automation owns per-service container IMAGE TAGS (the $imagepolicy setters). The two do NOT overlap: this config DISABLES image-tag updates so Renovate never fights Flux over a running version. Self-hosted against the bootstrap Gitea; platform: gitea set at runtime by the operator's renovate runner.",
+  "_comment": "Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). AS OF 2026-10-17 RENOVATE OWNS BOTH HALVES: Helm chart/dependency bumps AND per-service container image tags. It opens PRs for everything; allowed low-risk classes auto-merge on green CI. The ADR-0009 split that gave image tags to Flux image-automation is formally closed — see .gitea/workflows/renovate.yaml for the three measured reasons (it pushes commits rather than opening PRs, so it bypasses the CLAUDE.md §1 gate; `main` is protected with user_can_push:false and the flux-system GitRepository has no secretRef, so it cannot write anyway; and enabling it would need an in-cluster git WRITE credential for the repo that defines the cluster). image-automation has been commented out at platform/kustomization.yaml:52 since it was authored and the cluster holds zero ImagePolicy/ImageRepository/ImageUpdateAutomation objects, so nothing is being torn down. CONSEQUENCE: the `docker` and `kubernetes` managers, previously disabled ONLY to avoid fighting that controller, are now ENABLED — without them every plain `image: repo/name:tag` line in this repo was maintained by nothing at all, which is how the oauth2-proxy auth sidecars reached 7 minors behind the oauth2-proxy running in the same cluster. RUNNER: .gitea/workflows/renovate.yaml (scheduled daily); platform/endpoint are set there, not here.",
   "extends": [
     "config:recommended",
     ":dependencyDashboard",
@@ -15,13 +15,24 @@
   "labels": ["dependencies", "automated-update"],
   "ignoreUnstable": true,
   "respectLatest": true,
-  "enabledManagers": ["flux", "kustomize", "helmv3", "regex", "github-actions"],
+  "enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "regex", "github-actions"],
   "flux": {
     "managerFilePatterns": ["/(^|/)platform/.+\\.ya?ml$/", "/(^|/)apps/.+\\.ya?ml$/"]
   },
   "kustomize": {
     "managerFilePatterns": ["/(^|/)kustomization\\.ya?ml$/"]
   },
+  "_kubernetes_comment": "ADDED 2026-10-17. The `kubernetes` manager reads plain `image:` fields out of Kubernetes manifests, and it is NOT auto-detected — it only looks at files matched here, which is why this block is as broad as it is. Scope covers every place this repo writes a container image: platform components, the rendered app waves, the ServiceDefinition catalog that generates them, the runbooks web server and the pm-agent job template. `catalog/` matters as much as `apps/`: apps/wave-*/*.yaml are GENERATED by catalog/render.py, so a bump applied only to the rendered output is reverted by the next render and would also fail ci/render-golden.sh. Renovate will raise BOTH, and both must land together.",
+  "kubernetes": {
+    "managerFilePatterns": [
+      "/(^|/)platform/.+\\.ya?ml$/",
+      "/(^|/)platform-config/.+\\.ya?ml$/",
+      "/(^|/)apps/.+\\.ya?ml$/",
+      "/(^|/)catalog/.+\\.ya?ml$/",
+      "/(^|/)runbooks/serving/.+\\.ya?ml$/",
+      "/(^|/)infra/agent/.+\\.ya?ml$/"
+    ]
+  },
   "customManagers": [
     {
       "customType": "regex",
@@ -60,14 +71,29 @@
       "matchFileNames": ["ci/versions.env", "infra/devcontainer/versions.env"],
       "automerge": false,
       "addLabels": ["ci-tooling"]
+    },
+    {
+      "description": "ADDED 2026-10-17. NEVER AUTO-MERGE THE BLAST-RADIUS COMPONENTS, not even a patch. The default rule above auto-merges minor/patch/digest on a green gate, which is right for an app image: a bad Sonarr bump inconveniences one pod. It is wrong for these five, because the gate is a STATIC check — it renders manifests and lints them, it does not boot a node, drain a volume or form an etcd quorum (vars.CLUSTER_AVAILABLE is unset, so L2/L3 conformance does not run in CI at all). So for anything whose failure mode is the CLUSTER rather than a service, a green gate is not evidence and a human must read the changelog. kured is here because it reboots control-plane nodes; longhorn because a storage-engine bump migrates live volume data; cilium because it is the CNI and a bad bump is a total partition (and its version is DUAL-PINNED in infra/ansible/roles/cilium/defaults/main.yml, which Renovate does not see, so the two would silently desync); cloudnative-pg because it drives Postgres switchovers; kube-prometheus-stack because ci/versions.env asserts an exact promtool/Prometheus lockstep and a chart bump that moves Prometheus fails the gate until that file moves with it.",
+      "matchPackageNames": [
+        "kured",
+        "longhorn",
+        "cilium",
+        "cloudnative-pg",
+        "kube-prometheus-stack"
+      ],
+      "automerge": false,
+      "addLabels": ["needs-review", "blast-radius"]
     }
   ],
+  "_unmanageable_comment": "WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A FLOATING tag has no version to compare, so Renovate skips it silently — it does not pin it and does not warn. As of 2026-10-17 that is: plexinc/pms-docker:latest, lscr.io/linuxserver/jackett:latest, pihole/pihole (NO TAG AT ALL), quay.io/keycloak/keycloak (no tag), nextcloud:31-apache and docker:27-dind. Those must be pinned BY HAND to a concrete version before any automation can maintain them, and until then they drift on every pull with nothing recording what is running. Separately, the HelmRelease minor-locking ranges (9 of 12, e.g. `>=88.0.0 <89.0.0` on kube-prometheus-stack) cap Renovate at patches inside a line it has been told not to leave; raising those ceilings is deliberate per-component work and is NOT part of enabling the bot. Both lists are in decisions/2026-10-17-automated-upgrades.md.",
   "ignorePaths": [
     "**/*.example",
     "**/*.sops.yaml",
     "conformance/**",
-    "platform/image-automation/per-service/**"
+    "platform/image-automation/per-service/**",
+    "platform/kyverno/tests/**"
   ],
-  "docker": { "enabled": false },
-  "kubernetes": { "enabled": false }
+  "_ignorePaths_comment": "platform/kyverno/tests/** ADDED 2026-10-17: those Kyverno policy fixtures pin `alpine:latest` and a `conformance-runner:latest`, which are test inputs whose whole point is to be a fixed shape, not software to keep current. `conformance/**` was already ignored for the same reason; this closes the matching hole that enabling the `kubernetes` manager would otherwise have opened.",
+  "_docker_comment": "The `docker` manager (Dockerfiles, docker-compose) and the `kubernetes` manager above were BOTH previously `enabled: false`, solely so Renovate would not fight Flux image-automation over a running tag. That controller was never enabled and never will be here (see _comment), so the only thing those two flags were achieving was leaving every plain `image:` tag in the repo unmanaged. `docker` is left out of enabledManagers because this repo contains no Dockerfile that ships to the cluster; `kubernetes` is now ON and is the one that matters.",
+  "docker": { "enabled": true }
 }

runbooks/cluster-maintenance-shutdown.md

@@ -0,0 +1,162 @@
+# cluster-maintenance-shutdown — powering the whole cluster off, and back on
+
+**Operator procedure, not an alert page.** Nothing links here from an alert.
+**Script:** `infra/maintenance/quiesce.sh` (`quiesce` | `resume` | `status`)
+**Related:** `/runbooks/update-rollback` is for rolling node work where the cluster stays up. This
+page is for taking **all three nodes down at once**.
+
+First written after the 2026-10-17 UPS move, which was improvised live because no procedure
+existed — `infra/cluster.sh stop` is `virsh`-based and only ever applied to the decommissioned dev
+cluster, so it does nothing for these three baremetal nodes.
+
+## Why you cannot just run `shutdown -h`
+
+You mostly can now, and could not before.
+
+Until 2026-10-17 **kubelet graceful node shutdown was unconfigured on every prod node**, so systemd
+stopped k3s, containerd went with it, and every pod was SIGKILLed where it stood — 9 Postgres
+instances mid-checkpoint, the UniFi VM, and 25 attached Longhorn volumes.
+`roles/k3s_server` now sets `shutdown-grace-period` (and, critically, the
+`InhibitDelayMaxSec` logind cap that actually enforces it — without that systemd stops waiting
+after 5s and the drain is silently truncated). A plain reboot of one node is therefore safe.
+
+**A planned whole-cluster shutdown still wants `quiesce`**, for three things the kubelet's drain
+does not do:
+
+1. **The UniFi VM needs an ACPI guest shutdown**, not an eviction. KubeVirt does that on
+   `running: false`; an eviction is a power cut to the guest.
+2. **CloudNativePG wants hibernation**, not pod deletion — and hibernation is **imperative** state
+   that Flux will neither set nor clear.
+3. **Draining three nodes one at a time makes pods reschedule onto the survivors**, churning
+   Longhorn replicas for no reason when everything is going down anyway. Scaling in place avoids it.
+
+## The hazard that bit us — read this before scaling anything by hand
+
+**A StatefulSet can carry `persistentVolumeClaimRetentionPolicy.whenScaled: Delete`, and scaling it
+to 0 then DELETES its PVCs and their Longhorn volumes.**
+
+On 2026-10-17 the hand-rolled quiesce scaled `monitoring/loki` (which has that policy) to 0, and
+`storage-loki-0` was deleted outright. Loki stores with `type: filesystem`, so that PVC was the only
+copy. Its 72h retention bounded the loss to a rolling window — nothing about the *mechanism* is
+bounded. The only visible symptom was the Longhorn volume count going 25 → 24, which was first
+misread as a miscount.
+
+`quiesce.sh` now refuses to scale any such StatefulSet, leaves it running, and tells you which
+volumes to expect `degraded` on restart. **If you quiesce by hand, check the policy first:**
+
+```bash
+kubectl --context k3s-prod get statefulset -A \
+  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,WHEN_SCALED:.spec.persistentVolumeClaimRetentionPolicy.whenScaled'
+```
+
+A hard-detached volume comes back `degraded` and rebuilds itself automatically. That is a vastly
+better outcome than a deleted one — take the rebuild.
+
+## Shutting down
+
+Run everything through the pinned toolbox; it supplies `kubectl` and the context.
+
+```bash
+# 0. Look before you leap. Refuses to proceed if any Longhorn volume is unhealthy —
+#    powering off on top of a degraded volume is how one replica becomes zero.
+./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh status'
+
+# 1. Quiesce: suspend Flux, ACPI-stop the VM, hibernate Postgres, scale down every
+#    PVC-holding workload (NFS consumers first, PiHole last), wait for detach.
+./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh quiesce'
+```
+
+**Expect DNS to stop** when PiHole scales down — it is the LAN resolver. `kubectl` is unaffected
+because the kubeconfig points at the kube-vip IP `192.168.2.9`, not a name.
+
+```bash
+# 2. Power the nodes off. Inside the devcontainer, which holds the become password.
+./infra/devcontainer/dev.sh run bash -c \
+  'cd /work/infra/ansible && ansible -i inventory/hosts.ini k3s_prod -b \
+     -m shell -a "shutdown -h +1 \"maintenance\""'
+```
+
+`+1` gives Ansible time to disconnect cleanly. The group resolves to all three nodes; confirm with
+`--list-hosts` if you want to see them first.
+
+```bash
+# 3. Confirm they are actually down before touching anything else.
+for h in 192.168.2.10 192.168.2.11 192.168.2.12; do ping -c1 -W2 $h >/dev/null && echo "$h UP" || echo "$h down"; done
+```
+
+**Then, in this order and not before:**
+
+| # | Machine | How |
+|---|---|---|
+| 4 | **TrueNAS** `192.168.2.30` | **Web UI only.** Not in the ansible inventory, and SSH is **refused** (service disabled). `https://192.168.2.30` → power icon → Shut Down. Safe once the nodes are down, because every NFS consumer (`plex-media`, `*-media`, `qbittorrent-downloads`, `nextcloud-legacy-archive`) is already scaled to zero and nothing holds a hard mount. |
+| 5 | **Router / switch** | Last. |
+
+TrueNAS must go **after** the k3s nodes: a hard NFS mount on a live node wedges kubelet in
+D-state (`/runbooks/nfs-export-latency`, and the load-average symptom in the 2026-10-?? note).
+
+## Bringing it back
+
+**Power on in the reverse order: router/switch → TrueNAS → all three nodes together.** etcd needs
+2 of 3 for quorum, so start the nodes as a group rather than one at a time.
+
+```bash
+# 1. Nodes Ready, etcd quorum, storage reachable.
+./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get --raw=/healthz/etcd'
+
+# 2. Resume: un-hibernate Postgres, then resume Flux.
+./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh resume'
+```
+
+**Un-hibernating Postgres is the step everyone forgets.** The annotation is imperative and survives
+a Flux resume, so resuming Flux alone leaves all three databases down and Keycloak, Nextcloud and
+Semaphore looking broken for reasons that point nowhere near Postgres. `resume` does it first, for
+exactly that reason.
+
+Flux restores every scaled-down replica count **and** the VM's `running: true` from git, so nothing
+needs undoing by hand.
+
+### What "still broken" looks like when it is actually fine
+
+* **`4/19` Kustomizations Ready for several minutes.** The dependency graph serialises behind
+  `platform-config`; 15 of them report `dependency ... is not ready` until it finishes. Check
+  `platform-config` itself is `Ready=True` and its `lastAppliedRevision` matches the GitRepository
+  artifact — if it is, wait.
+* **One or more Longhorn volumes `degraded`.** Any volume that was still attached at poweroff
+  rebuilds its lagging replica automatically. On 2026-10-17 it was Prometheus's 50Gi TSDB; it went
+  healthy on its own. Worry only if it is still degraded after ~30 minutes.
+* **`keycloak-operator` crashlooping, then recovering.** It is the CRD reconciler, not in the
+  request path; Keycloak's StatefulSet stays 2/2 and auth never goes down. See
+  `/runbooks/keycloak-*` and the known OOM-at-450Mi behaviour.
+
+### What is actually broken and needs you
+
+* **An `oauth2-proxy` replica stuck in CrashLoopBackOff.** It performs OIDC discovery at startup and
+  **exits 1** on the `503 no healthy upstream` it gets while Keycloak is still coming up, then sits
+  in exponential backoff long after Keycloak is fine. It has no startup dependency on Keycloak.
+  Delete the pod once and it comes up clean:
+
+  ```bash
+  ./infra/devcontainer/dev.sh run bash -c \
+    'kubectl --context k3s-prod -n oauth2-proxy delete pod <name>'
+  ```
+
+* **Orphaned `Unknown`/`Error` Job pods** from CronJobs that were mid-run at poweroff. They never
+  recover; delete them.
+
+## Verifying you are actually back
+
+```bash
+./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh status'
+# nodes 3 Ready | flux 19/19 | longhorn 0 not healthy | postgres 9 pods | unifi 1 VMI
+
+# DNS, through PiHole's VIP
+./infra/devcontainer/dev.sh run bash -c 'nslookup home.keithjackson.dev 192.168.2.23'
+
+# The Gateway actually serving (401 on an auth-gated host is CORRECT)
+for h in home.keithjackson.dev gitea.home.keithjackson.dev plex.home.keithjackson.dev; do
+  curl -s -o /dev/null -w "$h %{http_code}\n" -k --resolve $h:443:192.168.2.20 https://$h/
+done
+```
+
+Expect `200`, `200`, `401`.

runbooks/kustomization.yaml

@@ -52,6 +52,13 @@ configMapGenerator:
       - backup-failure=backup-failure.md
       - backup-missed=backup-missed.md
       - backup-partial=backup-partial.md
+      # Operator procedure, not an alert page — nothing links here from an alert. Served because
+      # the whole-cluster shutdown had NO written procedure until 2026-10-17 (infra/cluster.sh is
+      # virsh-based and only ever applied to the decommissioned dev cluster), so the UPS move was
+      # improvised live, and the one hazard it exposed -- a StatefulSet with
+      # persistentVolumeClaimRetentionPolicy whenScaled: Delete loses its PVC when scaled to 0 --
+      # destroyed monitoring/loki's volume before anyone noticed.
+      - cluster-maintenance-shutdown=cluster-maintenance-shutdown.md
       - cdi-import-stuck=cdi-import-stuck.md
       - cert-expiry=cert-expiry.md
       - cert-issuance=cert-issuance.md
@@ -85,6 +92,9 @@ configMapGenerator:
       - node-io=node-io.md
       - node-memory=node-memory.md
       - node-memory-swap=node-memory-swap.md
+      # Covers all three platform-health.node-reboots rules (KuredRebootPending,
+      # KuredNodeCoverageGap, KuredNotReporting) from platform/kured/alerts.yaml.
+      - node-reboot-automation=node-reboot-automation.md
       # Linked from OAuth2ProxyGateUnobserved, and the diagnostic procedure for any gated app
       # that "is up but does not work" — the gate answers an unauthenticated XHR with a 302,
       # not a 401, so the failure arrives as an HTTP 200 carrying the wrong body.

runbooks/node-reboot-automation.md

@@ -0,0 +1,254 @@
+# Node reboot automation (kured)
+
+**Alerts:** `KuredRebootPending`, `KuredNodeCoverageGap`, `KuredNotReporting`
+**File:** `platform/kured/alerts.yaml`
+**Expressions:** `max by (node) (kured_reboot_required) == 1` / `count(kured_reboot_required) < count(kube_node_info)` / `absent(kured_reboot_required)`
+
+## What this means here
+
+These three alerts cover the OS reboot half of the automated upgrade process. The design is two
+pieces that only work together:
+
+- **`roles/unattended_upgrades`** (Ansible, on the host) installs Debian **security** updates
+  nightly. It never upgrades the kernel, never restarts k3s, and never reboots. When something it
+  installed needs a reboot, it leaves `/var/run/reboot-required` behind.
+- **kured** (`platform/kured`, a DaemonSet) watches that file. When it exists, kured takes a
+  **cluster-wide lock**, cordons the node, drains it, reboots it, and uncordons it — then holds the
+  lock for a further 30 minutes before any other node may go.
+
+**kured does not decide whether to reboot.** On these nodes the sentinel is written almost
+exclusively by a deliberate operator kernel upgrade
+(`playbooks/node-baseline.yml --tags kernel`), because the kernel is blacklisted from unattended
+upgrades. So a pending reboot normally means *a human already asked for this*.
+
+Which alert you have tells you which half is wrong:
+
+| alert | meaning |
+|---|---|
+| `KuredRebootPending` | The mechanism is alive and a reboot was offered, but it did not happen for >36h. **Something is blocking it.** |
+| `KuredNodeCoverageGap` | kured is not running on every node. The uncovered node will never be rebooted, silently. |
+| `KuredNotReporting` | kured is publishing nothing at all. Reboots are not happening, **and the other two alerts are inert.** Treat this as the most serious of the three. |
+
+**All three nodes are control-plane + etcd, so quorum is 2 of 3.** One node down is degraded but
+up; two is a lost cluster. Everything below is written to protect that. `concurrency: 1` plus
+`lockTtl: 0` means a node that hangs on reboot keeps the lock **forever** and no second node is ever
+rebooted behind it. That is deliberate: the cycle stops and pages a human rather than stepping over
+a corpse.
+
+**A reboot is not seamless.** Most workloads here are single-replica, so each reboot cycle takes a
+brief outage of whatever runs on that node — including `coredns` (1 replica), which means a
+cluster-wide DNS gap while it reschedules. See "Expected outage per cycle" below.
+
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
+picks a current-context per invocation, and that default has already changed once. Do not run two
+`dev.sh` invocations at once; they share one container and interfere.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `uncordon`, `rollout`) and `flux reconcile` are
+operator actions, not agent actions. Where this runbook gives a mutating command it is for an
+operator, and it says so.
+
+## First commands
+
+Start here, in order. The first question is always *which half is broken*.
+
+    # 1. Is kured running at all, and on how many nodes?
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds,pod -o wide'
+
+    # 2. Which node is asking for a reboot, and who holds the lock?
+    #    The lock is an annotation on kured's OWN DaemonSet. An empty/absent value means no holder.
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds kured \
+      -o jsonpath="{.metadata.annotations}"; echo'
+
+    # 3. Is a node cordoned, and is a reboot in progress?
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes \
+      -o custom-columns=NAME:.metadata.name,SCHEDULABLE:.spec.unschedulable,TAINTS:.spec.taints'
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes \
+      -o jsonpath="{range .items[*]}{.metadata.name}{\" \"}{.metadata.annotations.weave\.works/kured-reboot-in-progress}{\"\n\"}{end}"'
+
+    # 4. What is kured itself saying? It logs its decision every period (1h).
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured logs ds/kured --tail=80'
+
+    # 5. Is the gate blocking it? These are the ONLY alerts kured blocks on.
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n longhorn-system \
+      get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness' \
+      | grep -v healthy
+
+    # 6. Does the node actually want a reboot? (read-only; SSH as k3s, no sudo needed)
+    ssh -o BatchMode=yes 192.168.2.10 'ls -la /var/run/reboot-required; cat /var/run/reboot-required.pkgs 2>/dev/null'
+
+## Likely causes, most likely first
+
+### 1. The Longhorn gate is holding it (the common `KuredRebootPending` cause)
+
+kured queries Prometheus before taking the lock and refuses to proceed while
+`LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `KubeNodeNotReady` or `NodeExporterDown` is
+firing. This is working as designed and it is the **most likely** cause.
+
+Why it lingers: there are 3 nodes, every volume has 3 replicas, and
+`replica-soft-anti-affinity: false` means replicas may not share a node. So cordoning one node
+leaves nowhere to put a replacement and **every volume goes degraded 2/3** for the duration.
+Recovery is slow on purpose: `replica-replenishment-wait-interval` is 600s before a rebuild even
+starts, and `concurrent-replica-rebuild-per-node-limit` is 1, so ~25 volumes rebuild one at a time.
+
+Check progress, then wait:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n longhorn-system \
+      get volumes.longhorn.io -o json' | python3 -c 'import json,sys,collections; \
+      v=json.load(sys.stdin)["items"]; print(collections.Counter(x["status"]["robustness"] for x in v))'
+
+If volumes are rebuilding, **do nothing**. If a volume is stuck `degraded` or `faulted` with no
+rebuild progressing, that is a storage problem to fix first — see `/runbooks/longhorn-degraded`.
+Rebooting another node while a volume is at 1/3 is how you lose data.
+
+### 2. The drain timed out against a PodDisruptionBudget
+
+Six PDBs currently allow **zero** voluntary disruptions, and two kinds matter:
+
+- **`longhorn-system/instance-manager-*`** (one per node). `minAvailable: 1` against a selector
+  pinned to a single node's single pod, so `disruptionsAllowed` is structurally **always 0**.
+  Longhorn's `longhorn-manager` *deletes* the PDB when its `node-drain-policy`
+  (`block-if-contains-last-replica`) judges the drain safe. The drain is **supposed** to wait here.
+- **CNPG `<cluster>-primary`** (`keycloak-db`, `nextcloud-db`, `semaphore-db`). Also structurally 0
+  until the CNPG operator switches the primary to another instance. If all three primaries are on
+  the node being drained, three switchovers must happen.
+
+kured's `drainTimeout` is 15m: a genuinely stuck drain **aborts**, releases the lock, and is retried
+next period rather than wedging forever. Repeated timeouts show in kured's logs.
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pdb -A -o wide' | awk '$NF==0 || NR==1'
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
+      -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PRIMARY:.status.currentPrimary'
+
+If CNPG will not switch over, an **operator** can move the primary deliberately before the next
+window rather than fighting the drain.
+
+### 3. A node hung on reboot and still holds the lock
+
+`k3s-prod-2` has done this before (2026-09-03: powered, off the LAN ~38 minutes, physical power
+button required). `lockTtl: 0` means the lock does **not** expire, so the cycle stops here —
+correctly, since the cluster is at 2/3 and must not go to 1/3.
+
+Symptoms: one node `NotReady` or absent, `KuredNodeCoverageGap` firing alongside
+`KuredRebootPending`, and the lock annotation naming the missing node.
+
+**Operator recovery:** attend to the node physically. Once it is back and `Ready`, kured clears its
+own lock on the next period. If the node is gone for good and the lock must be cleared by hand,
+that is an operator action:
+
+    # OPERATOR ONLY. Only when you have confirmed the named node is NOT mid-reboot.
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured annotate ds kured \
+      weave.works/kured-node-lock- --overwrite'
+
+If the node was pruned mid-reboot and is left **cordoned** with no daemon to uncordon it:
+
+    # OPERATOR ONLY.
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod uncordon <node>'
+
+### 4. Nothing can write the sentinel (so nothing ever reboots)
+
+If `KuredRebootPending` never fires and reboots never happen even after a kernel upgrade, suspect
+the sentinel writer. On Debian, `/var/run/reboot-required` is created by a hook shipped in
+**`update-notifier-common`** — and it was absent on all three nodes before this automation landed.
+Without it kured is permanently, invisibly idle.
+
+    ssh -o BatchMode=yes 192.168.2.10 'dpkg -l update-notifier-common needrestart unattended-upgrades | tail -4'
+    ssh -o BatchMode=yes 192.168.2.10 'ls /etc/kernel/postinst.d/ /etc/apt/apt.conf.d/ | sort'
+
+Fix by re-running the role, which asserts this explicitly:
+
+    cd infra/ansible && ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify
+
+### 5. kured is not being scraped (`KuredNotReporting`)
+
+The chart defaults `service.create` and `metrics.create` to **false**, and the `kured` namespace
+ships a default-deny NetworkPolicy. If either enablement regressed, or the Prometheus ingress allow
+was dropped, targets are *discovered* and every scrape times out — the TSDB stays empty and all
+three rules go quiet. Longhorn hit exactly this failure on this cluster.
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get svc,servicemonitor,networkpolicy,ciliumnetworkpolicy'
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n monitoring exec \
+      sts/prometheus-kube-prometheus-stack-prometheus -c prometheus -- \
+      wget -qO- "http://localhost:9090/api/v1/targets?state=active" ' | grep -o '"job":"kured[^,]*' | head
+
+### 6. A node is uncovered by the DaemonSet (`KuredNodeCoverageGap`)
+
+kured declares **no tolerations**, because no prod node is tainted. A newly provisioned or
+re-provisioned node carrying a taint gets no kured pod and will never be rebooted.
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes \
+      -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints'
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds kured \
+      -o custom-columns=DESIRED:.status.desiredNumberScheduled,READY:.status.numberReady'
+
+The fix is a PR adding the needed toleration to `platform/kured/helmrelease.yaml` — not a `kubectl
+patch`, which Flux reverts.
+
+## Expected outage per reboot cycle
+
+Stated plainly, because this process is **not** seamless. Every workload at 1 replica on the
+draining node is down until it reschedules and becomes Ready elsewhere. Measured 2026-10-17: **45
+Deployments and 3 StatefulSets run a single replica.** The ones worth knowing about:
+
+- **`kube-system/coredns` — 1 replica.** A cluster-wide in-cluster DNS gap while it moves. This is
+  the single most impactful item and it is **not** fixed by this PR; raising coredns to 2–3 replicas
+  is recommended follow-up work.
+- **`monitoring/prometheus-…-0`, `monitoring/loki-0`, `monitoring/alertmanager-…-0`** — all
+  single-replica StatefulSets on Longhorn RWO volumes, so they additionally wait for volume
+  detach/reattach. **You lose monitoring and alerting for part of the cycle**, including these very
+  alerts.
+- **All three CNPG Postgres primaries** blip during switchover, which means Keycloak, Nextcloud and
+  Semaphore briefly lose their database.
+- **`gitea`, `nextcloud`, `plex`, `jackett`, `qbittorrent`, `radarr`, `sonarr`, `seerr`, `crafty`,
+  `homepage`, `headlamp`, `semaphore`** — each down for its reschedule.
+- **`unifi` (a KubeVirt VM)** has `evictionStrategy: LiveMigrate` but
+  `migrationMethod: BlockMigration`, so evicting it copies the whole root disk over the network.
+  Expect the drain of its node to be slow.
+- **`kube-vip`** fails the control-plane VIP over on every node reboot.
+
+Multi-replica and therefore fine: `cilium-operator`, `oauth2-proxy`, `kyverno-admission-controller`,
+the Longhorn CSI sidecars, `keycloak`, `pihole`, `runbooks`.
+
+## When to escalate or stop
+
+**STOP and do not intervene** if:
+
+- Longhorn volumes are `degraded` **and rebuilding**. This is the gate doing its job. Wait.
+- A single node is mid-reboot and under 15 minutes have passed.
+
+**STOP and get a human to the hardware** if:
+
+- Any node is `NotReady` while another reboot is pending. **Never clear the kured lock in this
+  state** — that is the one action that can take etcd from 2/3 to 1/3 and lose the cluster.
+- Two nodes are down. The cluster has no quorum; this runbook cannot help. Recover hardware first.
+
+**Escalate to a deliberate maintenance decision** if:
+
+- `KuredRebootPending` has been firing for over a week. The reboot is not going to happen on its
+  own; find the blocker above, or take the reboot manually during a watched window.
+- A volume is `faulted`. Fix storage before any further reboot.
+
+**Turn the automation off** if it is misbehaving and you need the cluster stable. The honest way is
+a PR, but for an emergency an operator may suspend the tier — this stops future reboots and leaves
+running nodes untouched:
+
+    # OPERATOR ONLY. Stops kured being reconciled; does NOT uncordon a node mid-cycle.
+    infra/devcontainer/dev.sh run bash -c 'flux --context k3s-prod suspend kustomization kured'
+
+Check afterwards whether a node was left cordoned (First commands, step 3) and uncordon it if so.
+Resume with `flux resume kustomization kured`. Nothing in the graph depends on `kured`, so
+suspending it stalls nothing else.
+
+---
+
+Source of truth: `runbooks/node-reboot-automation.md`. Index: `/runbooks/`
+
+Records: `decisions/2026-10-17-automated-upgrades.md` (design + what was not verified),
+`platform/kured/helmrelease.yaml` (every tuned value and why),
+`infra/ansible/roles/unattended_upgrades/README.md` (the package half, and the kernel exclusion).

runbooks/vpn-remote-access.md

@@ -21,7 +21,7 @@ behaves like a machine on the LAN: full reach to `192.168.2.0/24`, the Gateway's
 DNS, and internet egress out the house WAN.
 
 Authorisation is one thing only: the device's **public key is in
-`platform/vpn-ingress/peers.yaml`**. An unlisted key gets no reply at all.
+`platform/vpn-ingress/peers.conf`**. An unlisted key gets no reply at all.
 
 > **An enrolled device is as privileged as a machine sitting in the house** — including the three
 > k3s nodes (`192.168.2.10/.11/.12`: apiserver 6443, kubelet 10250, SSH 22). That is what "reach
@@ -134,6 +134,37 @@ specific failure in its log — so you never get a server that silently cannot c
 | `It is EMPTY` | the value was blanked |
 | `still SOPS CIPHERTEXT (starts with ENC[)` | the Secret reached the cluster undecrypted — this environment's overlay is missing its `vpn-ingress` decryption patch. **This is the dangerous one:** the Secret applies cleanly and every Flux condition stays green |
 | `wg pubkey rejected it` with none of the above | a mangled value, or a *public* key pasted by mistake |
+| `net.ipv4.ip_forward is 0 on this node` | the node_baseline role has not run here (step 1). The pod **cannot** set this itself — under `hostNetwork` the runtime mounts `/proc/sys` read-only, because it is the host's global setting |
+| `0 peer(s)` after you merged a peer | the DaemonSet did not roll. Check its `generation` (above). Pre-2026-10-06 this was guaranteed, because the register was a fixed-name ConfigMap; if it recurs, something has reverted `peers.conf` to a plain resource instead of a `configMapGenerator` |
+
+### What a healthy start looks like
+
+Measured on prod 2026-10-06. The init container logs one line, naming the shape and never the key:
+
+```
+wg0.conf assembled: 0 peer(s), server 10.90.0.1/24, udp/51820.
+```
+
+then the `wireguard` container brings the tunnel up:
+
+```
+**** Found WG conf /config/wg_confs/wg0.conf, adding to list ****
+**** Activating tunnel /config/wg_confs/wg0.conf ****
+[#] ip link add dev wg0 type wireguard
+[#] wg addconf wg0 /dev/fd/63
+[#] ip -4 address add 10.90.0.1/24 dev wg0
+[#] ip link set mtu 1420 up dev wg0
+```
+
+**Ignore the `chown: ... Operation not permitted` run before that**, and the image's own
+"Permissions could not be set ... we will not provide support for it". Capabilities are dropped to
+`NET_ADMIN` only, so its s6 init has no `CAP_CHOWN`; `/config` is a writable tmpfs, everything runs
+as root, and `wg-quick` proceeds. It is wrong about the cause and it does not matter.
+
+**A pod stuck `0/1 Running` with `Startup probe failed: Unable to access interface: No such device`
+means `wg-quick` tore the tunnel down** — read the `wireguard` container log for the `[#]` line it
+died on. That is the shape of a real failure; the probe is doing its job of refusing to call a
+server without a `wg0` "ready".
 
 ## Step 3 — forward udp/51820 on the router (once)
 
@@ -188,10 +219,23 @@ It generates the device's keypair, writes a `0600` client profile, prints a QR c
 
 Then:
 
-1. Paste the stanza into `platform/vpn-ingress/peers.yaml` under `data.peers.conf`, keeping the
+1. Append the stanza to `platform/vpn-ingress/peers.conf`, keeping the
    `# <name> — <owner>, enrolled <date>` comment. `wg show` can only ever print the key, so that
    comment is the only record of which stanza belongs to which device.
-2. PR → merge. Flux rolls the DaemonSet (~1 min).
+2. PR → merge. Flux rolls the DaemonSet (~1 min), because `peers.conf` is a
+   **content-hashed** `configMapGenerator`: editing it changes the ConfigMap's generated name,
+   which changes the pod template, which forces the roll. (It did **not** work this way at first —
+   a fixed-name ConfigMap updates in place, the pod template never changes, and the init container
+   keeps the old peer list forever. Measured on prod 2026-10-06.) Confirm it actually rolled:
+
+   ```bash
+   ./infra/devcontainer/dev.sh run bash -c \
+     'kubectl --context k3s-prod -n vpn-ingress get ds wireguard \
+        -o jsonpath="{.metadata.generation}{\"/\"}{.status.observedGeneration}{\"\n\"}"'
+   ```
+
+   The generation must have **increased** since before the merge. If it is still `1`, the roll did
+   not happen and `wg show` will list no peer no matter how long you wait.
 3. Get the profile onto the device (QR scan for phones; "Import tunnel from file" on
    Windows/macOS; `/etc/wireguard/<name>.conf` + `wg-quick@<name>` on Linux).
 4. **Delete the profile from your machine**: `shred -u <path>`. It is the only copy of that
@@ -231,7 +275,7 @@ if the router is not delivering the packets, and a `handshake` with no completed
 
 ## Revoking a device
 
-Delete its `[Peer]` stanza from `peers.yaml`, PR, merge. Nothing else — the key stops working on the
+Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the
 next reconcile. **Never re-use that address octet**: an old profile would silently start working
 again as a different device.