fix(unattended-upgrades): Debian has no reboot-sentinel package, so ship the writer

The first real run failed on all three nodes:

  fatal: [k3s-prod-1]: "No package matching 'update-notifier-common' is available"

update-notifier-common is an UBUNTU package and does not exist in Debian --
`apt-cache policy` returns Candidate: (none) on trixie. The role asserted it
"owns the hook that creates /var/run/reboot-required", which is true on Ubuntu
and false here, so the role could never have worked.

Three further measurements settled the replacement: notify-reboot-required is
absent; NO linux-image-* postinst references it (invoking it is an Ubuntu patch,
so dropping a script at that path would never be called); and no hook under
apt.conf.d or /etc/kernel mentions reboot-required. Debian trixie ships no
mechanism at all.

So the role now provides it:
* /usr/local/sbin/k3s-notify-reboot-required -- touches the sentinel when the
  running kernel is not the newest installed, using `linux-version` (from
  linux-base, a dependency of every linux-image, measured present) because
  kernel version ordering is not string ordering.
* /etc/apt/apt.conf.d/99-k3s-reboot-required -- DPkg::Post-Invoke, `|| true` so
  it can never abort an apt transaction.
/run is tmpfs (measured), so the sentinel self-clears on boot; no cleanup task.

The verify assert was ALSO weaker than it looked: a `find` glob for
*update-notifier* / *unattended-upgrades* under apt.conf.d would have matched
Debian's own unattended-upgrades conffile and reported "writer present" while
none existed. It now stats the two exact paths this role owns and requires the
script to be executable.

Newly verified: the script ran on prod-1 as unprivileged k3s under sh -x --
running=newest=7.1.8+deb13-amd64, took the equal branch, exited 0, wrote
nothing. Correct. sh -n and bash -n clean.

Still unverified: the WRITING half never executes while running == newest, so
the sentinel has not been seen created and kured has not been seen react. The
honest test is the next kernel upgrade.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

.gitea/workflows/renovate.yaml

@@ -0,0 +1,173 @@
+# Renovate — the dependency/image update automation. SCHEDULED, self-hosted against this Gitea.
+#
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# WHY THIS FILE EXISTS: renovate.json HAS BEEN IN THIS REPO AND HAS NEVER ONCE RUN.
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# Measured 2026-10-17: `renovate.json` is present, complete and thoughtfully written — it declares
+# managers, schedules, automerge classes and grouping. Nothing executes it. There is no Renovate
+# workflow (this is the first), no Renovate CronJob or Deployment in the cluster
+# (`kubectl get cronjob,deploy,job -A | grep -i renovate` -> nothing), and no bot account doing it
+# from outside. Meanwhile SEVEN files carry comments asserting "Renovate maintains this pin"
+# (platform/homepage/deployment.yaml:40, runbooks/serving/deployment.yaml:87,
+# platform/crafty/deployment.yaml:63, platform/vpn-ingress/daemonset.yaml:259,
+# platform/keycloak/postgres-cluster.yaml:75, apps/wave-0/nextcloud-db.yaml:302, and
+# platform/headlamp/helmrepository.yaml:2). Those comments have been false for the life of the file.
+#
+# The cost is measurable. With nothing bumping anything: the six per-app oauth2-proxy auth sidecars
+# sit on v7.8.1 while the platform oauth2-proxy in the SAME cluster runs v7.15.5; Keycloak (the IdP)
+# runs 26.0.6, a November-2024 release, 8 minors behind; Velero runs a RELEASE CANDIDATE
+# (velero-plugin-for-aws v1.14.3-rc.2) with a stable available; the CNPG Postgres image is 16.4,
+# missing every 16.5+ security release. A pin nothing maintains is not a pin, it is a fossil.
+#
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# WHY RENOVATE AND NOT FLUX IMAGE-AUTOMATION (the decision, and it reverses ADR-0009's split)
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# ADR-0009 divided the work: Renovate owns Helm chart/dependency bumps, Flux image-automation owns
+# per-service container image TAGS, deliberately non-overlapping. The second half of that split
+# cannot be delivered on this repo's terms, for three measured reasons:
+#
+#   1. IT WOULD BYPASS THE MERGE GATE, WHICH CLAUDE.md §1 FORBIDS. ImageUpdateAutomation does not
+#      open pull requests; it COMMITS AND PUSHES. Its default target is the source's own branch —
+#      `main` — so it would write straight past the PR flow. It can be pointed at a side branch
+#      instead, but then nothing opens the PR and bumps silently accumulate on a branch nobody
+#      merges. Either way a human or a second bot has to close the loop.
+#   2. ON THIS CLUSTER IT CANNOT EVEN DO THAT. Measured 2026-10-17: `main` is protected with
+#      `user_can_push: false`, and the `flux-system` GitRepository has NO `spec.secretRef` at all —
+#      the source is anonymous and read-only. Enabling write-back means minting an in-cluster git
+#      WRITE credential for the repository that defines the cluster, which is a materially larger
+#      blast radius than a CI-scoped token.
+#   3. RENOVATE ALREADY OPENS PRs NATIVELY, so every change it proposes goes through the gate with
+#      no extra machinery.
+#
+# So: Renovate owns BOTH halves, and Flux image-automation stays disabled. Note this is a change of
+# plan, not a change of state — image-automation has been commented out at
+# platform/kustomization.yaml:52 since it was authored, and the cluster has ZERO
+# ImageRepository/ImagePolicy/ImageUpdateAutomation objects (measured). Its two controllers ARE
+# installed and have idled for 43 days. Nothing is being torn down here; a path that was never
+# taken is being formally closed, and renovate.json's `docker`/`kubernetes` managers — disabled
+# purely to avoid fighting a controller that never ran — are switched on to fill the gap.
+#
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# THIS WORKFLOW MUST NOT BE ABLE TO SUCCEED SILENTLY
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# A SKIPPED Gitea job reports `success` (CLAUDE.md §2), so "green" is not evidence this ran. Two
+# guards:
+#   * The preflight step FAILS HARD (exit 1) if RENOVATE_TOKEN is absent, rather than skipping.
+#     A missing credential on a dependency bot is indistinguishable from "nothing to update", and
+#     that is precisely how renovate.json stayed inert for months.
+#   * The summary step records what ran into the job summary unconditionally.
+# What this still does NOT give you is an alert when the SCHEDULE stops firing. See the handoff
+# entry's "What I did NOT verify" — that gap is stated, not papered over.
+#
+# OPERATOR PREREQUISITE — ONE SECRET, NOT COMMITTED HERE:
+#   `RENOVATE_TOKEN` — a Gitea access token for the bot account, scopes `write:repository` and
+#   `read:user`. Set it as an Actions secret on this repository. Nothing in this repo mints,
+#   stores or references its value; it is consumed by name only.
+#   Until it exists, this workflow FAILS LOUDLY on every scheduled run. That is intended: a
+#   visible red is the correct state for "automation configured but not credentialed".
+#
+# THE FIRST RUN WILL WANT TO OPEN A LOT OF PRs. ~30 dependencies are behind. renovate.json caps the
+# blast radius (`prConcurrentLimit: 5`, `prHourlyLimit: 2`), but run it manually with
+# `dry_run: true` FIRST and read the log before letting the schedule loose. That is what
+# workflow_dispatch's default is for.
+name: renovate
+
+on:
+  schedule:
+    # 03:15 UTC daily. renovate.json ALSO carries `"schedule": ["after 3am and before 6am"]` in
+    # America/New_York, and that inner schedule is the authoritative one — Renovate re-checks it and
+    # will decline to open PRs outside it. The two are deliberately both present: this cron decides
+    # when the process RUNS, renovate.json decides when it may ACT. Keeping the cron slightly ahead
+    # of the repo window means a run is always in flight when the window opens.
+    - cron: "15 3 * * *"
+  workflow_dispatch:
+    inputs:
+      dry_run:
+        description: "Dry run (no PRs, no branches). Leave true for the first run."
+        required: false
+        default: "true"
+      log_level:
+        description: "Renovate LOG_LEVEL (info | debug)"
+        required: false
+        default: "info"
+
+concurrency:
+  # Never two Renovate runs at once: concurrent runs race on the same branches and can open
+  # duplicate PRs. Not cancel-in-progress — a half-finished Renovate run can leave a branch pushed
+  # with no PR, so let the first one finish.
+  group: renovate
+  cancel-in-progress: false
+
+jobs:
+  renovate:
+    runs-on: ubuntu-latest
+    # Renovate scans every manager across the whole repo; 40m is generous for this repo's size and
+    # well under a runner's patience. Without a timeout a wedged run holds the `renovate`
+    # concurrency group and silently blocks every subsequent scheduled run.
+    timeout-minutes: 40
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Preflight - the token must exist, and its absence must be LOUD
+        env:
+          RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
+        run: |
+          set -euo pipefail
+          if [ -z "${RENOVATE_TOKEN:-}" ]; then
+            echo "::error::RENOVATE_TOKEN is not set on this repository." >&2
+            cat >&2 <<'MSG'
+
+          Renovate cannot run without a Gitea access token, and this step fails rather than
+          skipping ON PURPOSE: a skipped Gitea job reports `success`, so skipping here would make
+          "dependency automation is not running" look exactly like "everything is up to date".
+          That is the failure mode that let renovate.json sit inert in this repo for months while
+          seven files claimed "Renovate maintains this pin".
+
+          TO FIX: create a Gitea access token for the bot account with scopes
+          `write:repository` and `read:user`, then add it to this repository as the Actions
+          secret RENOVATE_TOKEN. Do not commit it anywhere.
+          MSG
+            exit 1
+          fi
+          echo "RENOVATE_TOKEN is present (value not printed)."
+
+      - name: Run Renovate
+        uses: renovatebot/[email protected]
+        with:
+          # Pinned, and Renovate maintains this pin itself: renovate.json's enabledManagers
+          # includes `github-actions`, and `helpers:pinGitHubActionDigests` is in its `extends`.
+          # This is the one dependency in the repo whose updater is its own updater.
+          configurationFile: renovate.json
+          token: ${{ secrets.RENOVATE_TOKEN }}
+        env:
+          # Self-hosted Gitea. `platform` + `endpoint` were the two values renovate.json's own
+          # header said were "set at runtime by the operator's renovate runner" — this is that
+          # runner, so they are set here rather than in the repo config, which keeps renovate.json
+          # portable and free of this instance's URL.
+          RENOVATE_PLATFORM: gitea
+          RENOVATE_ENDPOINT: https://gitea.home.keithjackson.dev/api/v1
+          RENOVATE_REPOSITORIES: Home-Network/home-network-k3s-impl
+          # Renovate must never push to a protected branch; it works on its own branches and opens
+          # PRs against this base. Stated explicitly rather than inherited, because this is the
+          # property that keeps the whole mechanism inside CLAUDE.md §1's branch -> PR -> gate path.
+          RENOVATE_BASE_BRANCHES: main
+          RENOVATE_ONBOARDING: "false"
+          RENOVATE_REQUIRE_CONFIG: "required"
+          RENOVATE_DRY_RUN: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run == 'true' && 'full' || '' }}
+          LOG_LEVEL: ${{ github.event_name == 'workflow_dispatch' && inputs.log_level || 'info' }}
+
+      - name: Summary
+        if: always()
+        run: |
+          {
+            echo "## Renovate"
+            echo
+            echo "- trigger: \`${{ github.event_name }}\`"
+            echo "- dry run: \`${{ github.event_name == 'workflow_dispatch' && inputs.dry_run || 'false' }}\`"
+            echo "- outcome: \`${{ job.status }}\`"
+            echo
+            echo "Renovate owns BOTH Helm chart bumps and container image tags here; Flux"
+            echo "image-automation is deliberately disabled (platform/kustomization.yaml)."
+            echo "Every change it proposes arrives as a pull request and must pass the merge gate."
+          } >> "$GITHUB_STEP_SUMMARY"

.gitignore

@@ -37,6 +37,14 @@ coverage/
 infra/tofu/backend.hcl
 infra/tofu/.r2.env
 
+# Operator scratch, writable from inside the dev container (the repo is its only persistent
+# read-write mount). Holds things that must never be committed: a staged copy of the SOPS age or
+# WireGuard key, rendered VPN client profiles (infra/vpn-ingress/new-peer.sh writes here by
+# default). `infra/ansible/.gitignore` already does this for its own .cache/; this is the same
+# convention at the root. Do NOT rely on `*.key` alone to protect a staged key — that only covers
+# one filename shape, which is why this rule exists.
+.cache/
+
 # Claude Code local scratch: agent worktrees + session state. Nothing here is tracked;
 # ignoring it keeps `git add -A` from staging embedded worktree repos and keeps the lint
 # stage (which scans untracked files) from tripping on .claude/worktrees/.

apps/wave-0/plex.yaml

@@ -77,7 +77,7 @@ spec:
   storageClassName: longhorn
   resources:
     requests:
-      storage: 8Gi
+      storage: 16Gi
 ---
 # PVC plex-media (R-STATE)
 apiVersion: v1

catalog/plex.yaml

@@ -351,7 +351,7 @@ exposure: public              # edge port-forward + public DNS; clients + Seerr
 egress: internet              # normal WAN egress (docs/03; R-NET-05)
 storage:
   - name: config
-    size: 8Gi
+    size: 16Gi
     class: longhorn           # Plex DB/metadata: replicated app state on the 3-replica tier (docs/04; R-STATE-01/02)
     mountPath: /config        # plexinc/pms-docker writes its identity/claim + "Library" DB under /config, NOT the
                               # renderer default /data/config. Without this the PVC sat unused and Plex wrote to the

catalog/qbittorrent.yaml

@@ -359,7 +359,7 @@ storage:
                                 # The live /config contents were copied onto each PVC before this landed,
                                 # so API keys and credentials other apps reference were preserved.
   # THE DOWNLOADS AREA — ONE mount of the SAME export sonarr/radarr/plex already consume
-  # (192.168.2.102:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
+  # (192.168.2.30:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
   # /downloads, which is precisely where this app is ALREADY configured to write.
   #
   # WHY subPath: downloads AND NOT THE EXPORT ROOT AT /media (the shape sonarr/radarr use):
@@ -434,7 +434,7 @@ dependencies: []                # default-deny: reaches only DNS; arr reach it n
                                 # for weeks and neither has any CNP naming the NFS server or port 2049
                                 # (grep apps/wave-1/sonarr.yaml: none). The SAME fact is what keeps the VPN
                                 # egress gateway off this traffic (its selector is a podSelector), and
-                                # 192.168.2.102 is inside the 192.168.0.0/16 excludedCIDR besides — see
+                                # 192.168.2.30 is inside the 192.168.0.0/16 excludedCIDR besides — see
                                 # platform/vpn-egress/egress-steering.yaml, "BULK NFS".
 ingressRoutes:
   - host: qbittorrent.${BASE_DOMAIN}

clusters/base/cluster-vars.yaml

@@ -234,26 +234,28 @@ data:
   WG_CLIENT_CIDR: "10.90.0.0/24"             # remote VPN clients; .1 = server. NOT a LAN range
 
   # --- Bulk (external NFS) storage host (docs/04 bulk tier) ---------------------------------------
-  # THE address of the Proxmox host whose ZFS datasets back every `class: bulk` volume: the media
-  # library (Storage/plex, 8.33 T — plex, sonarr, radarr) and the legacy Nextcloud archive
+  # THE address of the NAS whose ZFS datasets back every `class: bulk` volume: the media library
+  # (Storage/plex, 8.33 T — plex, sonarr, radarr, qbittorrent) and the legacy Nextcloud archive
   # (Storage/nextcloud, 270 G). The renderer emits it into each static NFS PersistentVolume's
   # `spec.nfs.server` (catalog/render.py, BULK_NFS_SERVER); the export PATH is per-dataset and is
   # declared in the ServiceDefinition, because the path is identical in every environment and only
   # the ADDRESS is not.
   #
-  # WHY THIS IS A KNOB AND NOT A LITERAL — the two clusters are on OPPOSITE SIDES of pfSense.
-  # The storage host has exactly one interface, vmbr0 192.168.1.9/24, on pfSense's LAN.
-  #   dev  nodes are 192.168.1.40/.41/.42 — the SAME L2 segment. They reach it directly, and one
-  #        NFS export to them already exists on that host (/mnt/Storage/k8s-volumes in
-  #        /etc/exports), so this path is proven, not theoretical. Base default = the dev value,
-  #        per the convention of the LAN block above.
-  #   prod nodes are 192.168.2.10/.11/.12 — pfSense's WAN. tcp/2049 does NOT traverse that today;
-  #        22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. Opening
-  #        the storage path is a PENDING OPERATOR DECISION with two candidate outcomes that give
-  #        DIFFERENT values here: 192.168.2.102 (another port-forward) or 192.168.1.9 (a route).
-  # So prod holds a loud TBD-PROD-* placeholder — see clusters/prod/kustomization.yaml for the
-  # blast radius of shipping it unset, which is NOT the usual "fails to apply".
-  BULK_NFS_SERVER: "192.168.1.9"
+  # IT IS NO LONGER A CROSS-NETWORK KNOB (2026-10-07). This used to read "the two clusters are on
+  # OPPOSITE SIDES of pfSense": the datasets lived on a Proxmox host whose only interface was
+  # vmbr0 192.168.1.9/24 on pfSense's LAN, which prod (192.168.2.0/24, pfSense's WAN) could reach
+  # only through a DNAT at 192.168.2.102. Both pools were physically moved onto a standalone
+  # TrueNAS at 192.168.2.30 — prod's OWN /24 — so there is no crossing to arrange and no candidate
+  # values to choose between. Measured 2026-10-07: 192.168.2.102:2049 no longer answers;
+  # 192.168.2.30:2049 and :111 do. There is now exactly ONE storage host and one correct value;
+  # this stays a knob only because the renderer reads it per-environment.
+  #
+  # *** CHANGING THIS VALUE IS NOT A PLAIN RECONCILE. *** A bound PersistentVolume's spec.nfs is
+  # immutable — measured by server dry-run 2026-10-07:
+  #     spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation
+  # so Flux's apply FAILS rather than updating, and the pods keep their old mount. The PVs must be
+  # deleted and recreated; the operator sequence is docs/BULK-NFS-STORAGE.md §10.
+  BULK_NFS_SERVER: "192.168.2.30"
 
   # Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style.
   # Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db,

clusters/base/kured.yaml

@@ -0,0 +1,98 @@
+---
+# Flux Kustomization: kured (Kubernetes Reboot Daemon) — reconciles ./platform/kured.
+# Performs the cordon / drain / reboot / uncordon of a node that has a pending OS reboot, strictly
+# one node at a time, inside a nightly window, gated on Longhorn volume health. Design rationale,
+# every tuned value, and the honest list of what is NOT mitigated: platform/kured/helmrelease.yaml.
+#
+# WHY ITS OWN KUSTOMIZATION, AND NOT ONE LINE IN platform/kustomization.yaml
+#
+# The same argument the Cilium split-out recorded in platform/kustomization.yaml and intel-gpu-plugin
+# reused, except the consequence is worse here than for either of them:
+#
+#   * `platform` is `wait: true` over twelve-plus components, so its Ready condition cannot tell you
+#     "the reboot daemon is wedged" from "longhorn is slow".
+#   * The only emergency valve would be `flux suspend kustomization platform` — freezing storage,
+#     TLS, DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a DaemonSet.
+#   * That tier's 10m interval / 5m timeout are called out there as WRONG for a 3-node DaemonSet
+#     rollout. This is a 3-node DaemonSet rollout.
+#   * And `platform` gates platform-config, which gates apps-wave-0 (wait:true), which gates
+#     apps-wave-1. A reboot daemon that failed to go healthy inside that tier would stall EVERY
+#     APPLICATION in the cluster on "dependency not ready". A component added to make maintenance
+#     safer must not acquire the ability to cause an outage; that trade is the whole argument.
+#
+# dependsOn: [platform] — AND THIS ONE IS A REAL EDGE, NOT CONVENTION
+#
+# Not `cilium` (what intel-gpu-plugin uses) and not `platform-config` (what the leaf UIs use). It is
+# `platform`, for two independent reasons:
+#
+#   1. CRD ORDERING (F-MON1). platform/kured ships a PrometheusRule, and the chart generates a
+#      ServiceMonitor. Both are CRs whose CRDs (monitoring.coreos.com) are installed by
+#      kube-prometheus-stack INSIDE the `platform` tier. Flux dry-runs every object in a pass before
+#      applying any of them, so a pass containing an unknown kind ABORTS — the failure that made the
+#      CDI StorageProfiles a true deadlock on prod on 2026-09-05 (see
+#      clusters/base/platform-config-cdi.yaml). Depending on `platform` guarantees the CRDs exist
+#      before this pass is attempted.
+#   2. FUNCTIONAL PREREQUISITE. kured's reboot gate QUERIES Prometheus
+#      (`prometheusUrl` in the HelmRelease). Prometheus lives in platform/monitoring. If kured ran
+#      first it would get connection errors, treat the gate as unevaluable and skip every reboot —
+#      the correct fail-safe direction, but silent. Longhorn, whose volume health IS the gate, is
+#      also in that tier.
+#
+# It does NOT need platform-config: no Keycloak, no TLS, no split-horizon DNS, no Gateway, no
+# HTTPRoute. kured serves no HTTP to users.
+#
+# wait: true — A HEALTH SIGNAL, NOT A GATE. Following intel-gpu-plugin rather than the leaf-UI
+# template. A leaf UI is legitimately not-Ready for long stretches of a cold start while it waits on
+# Keycloak and DNS, so its health is uninformative; kured has no such dependency — it is a DaemonSet
+# that opens a socket and watches a file, so on healthy nodes it is Ready in seconds and not-Ready
+# genuinely means BROKEN. This turns `flux get kustomization kured` into a one-line answer to "is
+# reboot automation live on all three nodes", which is the question the operator must be able to
+# answer before trusting a kernel upgrade to it.
+#
+# NOTHING IN THE GRAPH dependsOn THIS, so wait:true gates nothing — the blast radius of an unhealthy
+# kured is this Kustomization's own Ready condition. Do not add `dependsOn: [kured]` anywhere: that
+# would hand a node-rebooting DaemonSet the power to stall whatever depends on it, which is precisely
+# what the placement above exists to prevent.
+#
+# prune: true is safe and deliberate. If this Kustomization is deleted, kured's DaemonSet goes with
+# it and reboots simply stop happening — nodes are left uncordoned and running. The one caveat is
+# that pruning mid-reboot would leave a node CORDONED with no daemon to uncordon it; that is a
+# `kubectl uncordon` away and is covered in runbooks/node-reboot-automation.md.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: kured
+  namespace: flux-system
+spec:
+  # A HelmRelease plus five static manifests, nothing to re-render; kustomize-controller is
+  # source-triggered, so this is only the drift-correction floor.
+  interval: 10m
+  retryInterval: 2m
+  # 10m, not the 5m the `platform` tier uses: a 3-node DaemonSet rollout under `wait: true` can
+  # exceed 5m on a cold start (image pull on three nodes), which is one of the reasons the Cilium
+  # split-out gave for leaving that tier.
+  timeout: 10m
+  prune: true
+  # See the header: a health SIGNAL for a component nothing depends on, not a gate.
+  wait: true
+  dependsOn:
+    - name: platform   # monitoring CRDs + Prometheus + Longhorn; see the header, this is load-bearing
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/kured
+  # REQUIRED, not boilerplate. platform/kured/alerts.yaml renders `${BASE_DOMAIN}` into the
+  # `runbook_url` annotation of all three rules. Without this block that token becomes the EMPTY
+  # STRING and the Kustomization still reports Ready — Flux's substitution fails OPEN — so every
+  # alert would ship a dead link to `https:///runbooks/node-reboot-automation`. The gate stage
+  # `flux-postbuild-substitution` asserts the converse (a rendered token implies this block), and
+  # BASE_DOMAIN is declared in clusters/base/cluster-vars.yaml.
+  #
+  # Nothing else in this path contains `${...}` or a bare `$`: the HelmRelease's alertFilterRegexp is
+  # a plain anchored alternation, and there are no shell snippets anywhere in the component — which
+  # is deliberate, because envsubst would eat a braced shell variable and `$$` would trip the
+  # dollar-collapse assertion in the same check.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars

clusters/base/kustomization.yaml

@@ -85,4 +85,17 @@ resources:
   # pm-agent-credentials.sops.yaml, so both overlays add a SOPS decryption patch for it (by
   # name). See clusters/base/pm-agent.yaml and platform/pm-agent/README.md.
   - pm-agent.yaml        # Flux Kustomization -> ./platform/pm-agent (dependsOn apps-wave-0)
+  # kured (Kubernetes Reboot Daemon) — its OWN Kustomization (dependsOn platform, wait:true), NOT a
+  # line in platform/kustomization.yaml. Same reasoning as the Cilium and intel-gpu-plugin
+  # split-outs, with a sharper consequence: `platform` gates platform-config and every apps wave, so
+  # a wedged reboot DaemonSet placed in that wait:true tier could stall every application in the
+  # cluster — a component added to make maintenance SAFER must not be able to cause an outage.
+  # wait:true here is a HEALTH SIGNAL, not a gate: nothing dependsOn it, and nothing should.
+  # The `platform` edge is load-bearing rather than conventional — it supplies the
+  # monitoring.coreos.com CRDs this component's PrometheusRule/ServiceMonitor need (F-MON1), and
+  # Prometheus itself, which kured QUERIES before every reboot to gate on Longhorn volume health.
+  # It reboots a node only when /var/run/reboot-required exists, which on these nodes only a
+  # deliberate operator kernel upgrade creates — the human gate is the sentinel, not the daemon.
+  # See clusters/base/kured.yaml and platform/kured/helmrelease.yaml.
+  - kured.yaml           # Flux Kustomization -> ./platform/kured (dependsOn platform)
   - cluster-vars.yaml    # overlay knob surface (base defaults)

clusters/base/vpn-ingress.yaml

@@ -8,8 +8,8 @@
 #     NOTHING downstream reconciles — the incident behind the headlamp/oauth2-proxy/homepage moves
 #     (DECISIONS-LOG: "Headlamp gating stall").
 #   * SPECIFIC, AND THE REASON THIS IS NOT A JUDGEMENT CALL: this DaemonSet cannot start until the
-#     operator creates the out-of-band `wg-server-key` Secret (runbooks/vpn-remote-access.md
-#     step 1). That is the correct fail-closed state — an unenrolled VPN must not come up — but in
+#     operator replaces the `REPLACE-ME` placeholder in platform/vpn-ingress/wg-server-key.sops.yaml
+#     with a real key (runbooks/vpn-remote-access.md step 2). That is the correct fail-closed state — an unenrolled VPN must not come up — but in
 #     the gating tier it would have been a self-inflicted platform-wide outage ON MERGE, with the
 #     cause (a missing manual step) invisible from any Flux condition.
 #
@@ -51,6 +51,9 @@ spec:
     substituteFrom:
       - kind: ConfigMap
         name: cluster-vars
-  # NO `decryption:` stanza is needed in the overlays: this path carries no *.sops.yaml. The server
-  # key is an out-of-band Secret (platform/vpn-ingress/kustomization.yaml says what to do if that
-  # ever changes).
+  # SOPS decryption is layered per-environment by the overlay (clusters/dev and clusters/prod each
+  # patch THIS Kustomization by name), identical to headlamp/crafty/velero — this path carries
+  # platform/vpn-ingress/wg-server-key.sops.yaml, the WireGuard server private key. The `sops-age`
+  # Secret is created out of band and referenced by NAME only. Without the overlay patch the Secret
+  # applies cleanly full of ENC[...] and only the tunnel is broken, so the patch is as load-bearing
+  # as the encrypted file itself.

clusters/dev/kustomization.yaml

@@ -122,6 +122,24 @@ patches:
           provider: sops
           secretRef:
             name: sops-age
+  # vpn-ingress (the WireGuard remote-access VPN, clusters/base/vpn-ingress.yaml) carries
+  # platform/vpn-ingress/wg-server-key.sops.yaml — the WireGuard SERVER private key, the one
+  # secret in that component. Without this stanza kustomize hands Flux the raw ciphertext and the
+  # Secret applies full of ENC[...]: the APPLY SUCCEEDS and every Flux condition stays green, and
+  # the only symptom is a VPN that no client can handshake with. The DaemonSet's init container
+  # validates the value with `wg pubkey` specifically so that state is a named CrashLoop instead.
+  # The `sops-age` Secret is out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: vpn-ingress
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
   # pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries
   # pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot
   # account and the Anthropic subscription token from `claude setup-token`. Without this
@@ -261,15 +279,15 @@ patches:
       - op: replace
         path: /data/CRAFTY_L4_VIP
         value: "192.168.1.55"
-      # Bulk (external NFS) storage host: the Proxmox host itself, on the SAME LAN as dev's nodes
-      # (192.168.1.40/.41/.42), which already hold an NFS export from it (/mnt/Storage/k8s-volumes).
-      # Restated verbatim rather than inherited, per this overlay's "supply every knob" convention.
-      # Dev is therefore the environment in which the bulk manifests can actually be VALIDATED
-      # end-to-end before prod's network path exists — provided the additive `zfs set sharenfs=`
-      # change described in docs/BULK-NFS-STORAGE.md is applied for the dev node addresses.
+      # Bulk (external NFS) storage host. THIS VALUE IS INERT — dev is decommissioned, its nodes
+      # (192.168.1.40/.41/.42) no longer exist, and the Proxmox host this used to name (192.168.1.9)
+      # went with the pools it held. It is restated per this overlay's "supply every knob"
+      # convention, not because anything consumes it. It carries the real address so that no file
+      # in the tree renders a host that is gone; it has NOT been proven from a dev node, because
+      # there is no dev node to prove it from.
       - op: replace
         path: /data/BULK_NFS_SERVER
-        value: "192.168.1.9"
+        value: "192.168.2.30"
       # Off-cluster backup target: the on-Proxmox MinIO (plain HTTP, path-style) that every CNPG
       # barmanObjectStore and Velero's backupStorageLocation write to.
       #

clusters/prod/kustomization.yaml

@@ -113,6 +113,24 @@ patches:
           provider: sops
           secretRef:
             name: sops-age
+  # vpn-ingress (the WireGuard remote-access VPN, clusters/base/vpn-ingress.yaml) carries
+  # platform/vpn-ingress/wg-server-key.sops.yaml — the WireGuard SERVER private key, the one
+  # secret in that component. Without this stanza kustomize hands Flux the raw ciphertext and the
+  # Secret applies full of ENC[...]: the APPLY SUCCEEDS and every Flux condition stays green, and
+  # the only symptom is a VPN that no client can handshake with. The DaemonSet's init container
+  # validates the value with `wg pubkey` specifically so that state is a named CrashLoop instead.
+  # The `sops-age` Secret is out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: vpn-ingress
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
   # pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries
   # pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot
   # account and the Anthropic subscription token from `claude setup-token`. Without this
@@ -522,60 +540,48 @@ patches:
         path: /data/LAN_CIDR
         value: "192.168.2.0/24"
       # ------------------------------------------------------------------------------------------
-      # BULK (EXTERNAL NFS) STORAGE HOST — *** NOT YET DECIDED. THIS PLACEHOLDER IS LOAD-BEARING. ***
+      # BULK (EXTERNAL NFS) STORAGE HOST — the TrueNAS at 192.168.2.30.
       # ------------------------------------------------------------------------------------------
       # Backs every `class: bulk` volume: the 8.33 T media library (Storage/plex — plex, sonarr,
-      # radarr) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
+      # radarr, qbittorrent) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
       #
-      # THE STORAGE HOST IS NOT REACHABLE FROM PROD AT ALL RIGHT NOW. It has one interface,
-      # vmbr0 192.168.1.9/24, on pfSense's LAN; prod's nodes are on 192.168.2.0/24, pfSense's WAN.
-      # 22 and 8006 answer from prod only because pfSense port-forwards them via 192.168.2.102;
-      # 2049 and 111 do not. Opening that path is an OPERATOR decision, and the two candidate
-      # outcomes give DIFFERENT values here:
-      #     port-forward 192.168.2.102:2049 -> 192.168.1.9:2049   =>  "192.168.2.102"
-      #     a WAN->LAN route to 192.168.1.0/24                    =>  "192.168.1.9"
-      # Guessing either one authors a prod cluster that mounts nothing, or — worse, if the guess
-      # ever became reachable — mounts something unintended. So it stays a placeholder.
+      # THIS WAS THE REPO'S ONE LOAD-BEARING PLACEHOLDER, AND THE REASON IT COULD BE IS NOW GONE.
+      # The datasets lived on a Proxmox host with a single interface on pfSense's LAN
+      # (192.168.1.9), which prod could reach only through a DNAT at 192.168.2.102 — so the correct
+      # value here depended on an undecided operator choice, and guessing it would have authored a
+      # cluster that mounts nothing. Both pools were physically moved onto a standalone TrueNAS at
+      # 192.168.2.30, prod's OWN /24. No crossing to arrange, no candidates to choose between.
+      # Measured 2026-10-07: 192.168.2.102:2049 no longer answers; 192.168.2.30:2049 and :111 do.
       #
-      # WHICH PORTS THE DECISION MUST OPEN: exactly ONE, tcp/2049. The PVs mount nfsvers=4.1, and
-      # NFSv4 carries mount and lock in-protocol — no rpcbind/111, no mountd, no statd. Choosing v4
-      # over v3 was in part this decision's cost: v3 would have needed four ports and three of them
-      # on ephemeral numbers.
-      #
-      # WHAT THE EXPORT MUST ALLOW IS ALREADY SETTLED, and it is the same under BOTH outcomes.
-      # Measured 2026-09-07 from this workstation (192.168.2.132) through the existing port-forward:
-      # the storage host logged the connection as coming FROM 192.168.2.132, not from a pfSense LAN
-      # address (`ss -tn` peer 192.168.2.132, and the host's only route off-LAN is its default via
-      # 192.168.1.1). pfSense DNATs WITHOUT source NAT, so the NFS server will see prod's REAL node
-      # addresses either way. The additive `zfs set sharenfs=` values are in docs/BULK-NFS-STORAGE.md.
-      #
-      # *** BLAST RADIUS — READ BEFORE MERGING THIS TO A BRANCH PROD RECONCILES. ***
-      # A PersistentVolume's spec.nfs.server is an unvalidated string: the API server ACCEPTS this
-      # placeholder, the PV is created and the PVC BINDS to it (binding is pure control-plane). The
-      # failure surfaces only when a pod mounts — MountVolume.SetUp "Failed to resolve server
-      # TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr and nextcloud would sit in
-      # ContainerCreating on prod. Unlike the LAN knobs above, this placeholder does NOT fail at
-      # apply time and there is no CRD validation to lean on. Set this value BEFORE prod reconciles
-      # a revision containing the bulk volumes; the dev overlay renders a working value, which is
-      # where these manifests are meant to be proven first.
+      # ONE PORT IS STILL ALL IT NEEDS: tcp/2049. The PVs mount nfsvers=4.1 and NFSv4 carries mount
+      # and lock in-protocol — no rpcbind/111, no mountd, no statd. The NAS advertises
+      # `+3 +4 +4.1 +4.2` (/proc/fs/nfsd/versions, measured), so 4.1 negotiates.
       - op: replace
         path: /data/BULK_NFS_SERVER
-        # 192.168.2.102 is pfSense's WAN address, which DNATs tcp/2049 through to the Proxmox
-        # host's REAL address 192.168.1.9 on pfSense's LAN. Prod sits on the WAN side and cannot
-        # reach 192.168.1.9 directly, so the cluster mounts the NAT address. Dev is on the LAN and
-        # keeps the base value 192.168.1.9.
+        # VERIFIED ON THE NAS 2026-10-07, before any cluster touched it (`exportfs -v`):
+        #   /mnt/Storage/plex       192.168.2.10/.11/.12  rw,sec=sys,secure,root_squash,no_all_squash
+        #   /mnt/Storage/nextcloud  192.168.2.10/.11/.12  ro,sec=sys,secure,root_squash,no_all_squash
         #
-        # PROVEN from k3s-prod-1 on 2026-09-12, not inferred:
-        #   - mount of 192.168.2.102:/mnt/Storage/plex negotiated nfs4 vers=4.1 proto=tcp
-        #   - a write as uid 1000 succeeded. Tested as 1000 and NOT root deliberately: the export
-        #     carries no_root_squash, so a root write would have passed and proved nothing, while
-        #     LSIO images default to PUID 911 and would have gone Ready then failed every import
-        #     with EACCES. The tree is 0755 1000:3000, so uid 1000 is the only writer.
-        #   - ln() across downloads/ -> tv/ returned link count 2 (same inode), so the arrs get
-        #     real atomic moves instead of silently degrading to copy-then-delete on EXDEV.
-        # pfSense DNATs WITHOUT source NAT (the host logs prod's own node addresses as the peer),
-        # so the ZFS export ACL names 192.168.2.10/.11/.12 directly rather than one translated IP.
-        value: "192.168.2.102"
+        # no_all_squash is the flag the entire permission model rests on: client uids pass through
+        # untouched, so the arrs' PUID=1000 arrives as the owner of a 0755 1000:3000 tree. sec=sys
+        # keeps that numeric rather than principal-mapped. The export deliberately does NOT carry
+        # no_root_squash, which the Proxmox one did: nothing here needs it, because every write to
+        # the tree happens after the LSIO drop to uid 1000 and the images only chown their Longhorn
+        # /config. Both subPath directories (plex/downloads, nextcloud/nextcloud) already exist at
+        # 0755 1000:3000, so a squashed root can still traverse them for subPath resolution.
+        #
+        # The ro= posture on the archive is now SERVER-enforced, which the old export also managed.
+        # The media tree's read-only half (plex) stays a CLIENT-side guarantee — all four media pods
+        # can land on any node, so the server sees the same three addresses for reader and writers
+        # alike and cannot tell them apart. Unchanged by the move; see docs/BULK-NFS-STORAGE.md §5.
+        #
+        # *** CHANGING THIS VALUE REQUIRES DELETING AND RECREATING THE PVs. *** A bound PV's
+        # spec.nfs is immutable — measured by server dry-run 2026-10-07:
+        #     spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation
+        # so Flux's apply FAILS rather than updating and the pods keep their old, dead mount. The
+        # operator sequence is docs/BULK-NFS-STORAGE.md §10. Retain + the renderer's pre-set
+        # claimRef make it safe: no data is touched and each recreated PV rebinds its own PVC.
+        value: "192.168.2.30"
       # ==========================================================================================
       # OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05; bucket + token 2026-09-07).
       # ==========================================================================================

conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh

@@ -363,6 +363,54 @@ SCRAPE_OWNERS: list[dict] = [
         "why": "velero_* powers VeleroBackupFailure / PartialFailure / NoRecentBackup — the rules "
                "that failed to notice prod running with zero volume backups",
     },
+    # ── ADDED 2026-10-17 (feat/automated-upgrades). A NEW OWNER, NOT A RELAXED ASSERTION. ──────
+    # This entry is purely additive: it registers a new metric family and its enablement, and it
+    # changes nothing about how any existing owner is evaluated. It makes the check STRICTER for the
+    # component it covers, not looser — it adds two `require` patterns plus a `companion` obligation
+    # that platform/kured must keep satisfying, so a future edit that disables kured's Service,
+    # ServiceMonitor or NetworkPolicy FAILS the gate instead of silently re-inerting three alerts.
+    #
+    # WHY THE CHECK HAD TO BE TOUCHED AT ALL, rather than the manifest being wrong: assertion (3)
+    # fails any alert over a metric no owner claims, and the escape hatch (KNOWN_UNWIRED_METRICS) is
+    # SHRINK-ONLY, so declaring `kured_*` as accepted debt is not permitted and would be the wrong
+    # answer anyway — the scrape genuinely IS wired here. The only two honest options were to
+    # register the owner (this) or to ship the reboot automation with no alerting at all, which for a
+    # component whose every failure mode is silent would have been the worse engineering choice.
+    # `conformance/**` is a sensitive approval path precisely so that this reasoning gets read.
+    {
+        "owner": "kured: service.create + metrics.create in the HelmRelease",
+        "prefixes": ("kured_",),
+        "file": "platform/kured/helmrelease.yaml",
+        # BOTH keys are required and the chart defaults BOTH to false. `metrics.create` makes the
+        # ServiceMonitor; `service.create` makes the Service it selects. Enable only the monitor and
+        # it selects nothing, so the single kured metric is never scraped — and a PrometheusRule over
+        # a series no target produces is INERT, which for this component means its own liveness
+        # alarm is the thing that breaks. Matched as two separate anchored blocks rather than one
+        # pattern because they are separate top-level values keys, not nested.
+        "require": [
+            r"service:\s*\n\s*create:\s*true",
+            r"metrics:\s*\n\s*create:\s*true",
+        ],
+        "forbid": [],
+        "why": "kured_reboot_required is the ONLY metric kured exposes (one GaugeVec, "
+               "cmd/kured/main.go:92-96) and it powers all three platform-health.node-reboots "
+               "rules. Those rules are the only way to learn that OS reboot automation has stalled: "
+               "kured stays 1/1 Running while rebooting nothing, so a missed scrape here is "
+               "indistinguishable from a cluster that needed no reboots",
+        # The ServiceMonitor ALONE is not sufficient — the same NetworkPolicy trap Longhorn hit.
+        "companion": {
+            "file": "platform/kured/kustomization.yaml",
+            "require": [r"^\s*-\s*networkpolicy\.yaml\s*$"],
+            "why": "platform/kured/networkpolicy.yaml ships a default-deny baseline (the kured "
+                   "namespace is a platform namespace, so Kyverno's app-namespace-scoped generator "
+                   "does not cover it) AND the ingress allow that lets Prometheus reach :8080 "
+                   "through it. Drop the networkpolicy from this kustomization and the deny-all "
+                   "goes too, so this currently fails SAFE — but reinstating the baseline without "
+                   "the allow reproduces Longhorn's measured failure exactly: targets DISCOVERED, "
+                   "every scrape 'context deadline exceeded', TSDB empty, all three rules quiet. "
+                   "Keeping the two in one file makes that pair inseparable",
+        },
+    },
 ]
 
 # `up` is produced by EVERY scrape target, so no owner can meaningfully claim it and assertion (1)

decisions/2026-10-05-pm-agent-registry-pull-prereq.md

@@ -0,0 +1,127 @@
+---
+date: 2026-10-05
+branch: docs/pm-agent-registry-pull-prereq
+title: pm-agent reconciled exactly as designed — and is the first workload ever to pull from the in-cluster registry, which does not work yet
+---
+
+## What changed
+
+Documentation only. No manifest, no policy, no code.
+
+PR #57 merged and `pm-agent` reconciled. I measured what landed, and the measurement turned
+up **two prerequisites for the next operator step that nothing in this repository
+declares** — and that the handoff entry for #57 could not have known, because it listed the
+image pull as unverified and that is exactly where it bit.
+
+So this entry records the post-merge measurement, and
+`platform/pm-agent/README.md`, `docs/plans/pm-agent-issue-to-pr.md` and
+`runbooks/pm-agent.md` now name both prerequisites where they say "build, push, pin the
+digest". Before this, that step read as one action; it is three, and the two new ones are
+not an agent's to perform.
+
+**The finding is not specific to pm-agent.** `conformance-runner` has the identical
+registry path (`gitea.home.keithjackson.dev/home-network/...`), the identical "operator
+pushes, a digest goes in a PR" model, and has never been pulled either — so the same two
+blockers sit in front of the L2/L3 conformance wiring. Measured: **no pod anywhere in the
+cluster references that registry** except the pm-agent dispatcher, which is therefore the
+first workload ever to try it.
+
+## What I verified
+
+Read-only, through the pinned toolbox, against the live prod cluster.
+
+**The merge reconciled, completely and exactly as designed.**
+
+- `GitRepository/flux-system` artifact revision: `main@sha1:95597bd6883637…` — the merge
+  commit. `platform-config`, `pm-agent` and `apps-wave-0` all report that same
+  `lastAppliedRevision` with `Ready=True`. Compared field to field, per CLAUDE.md §2b,
+  rather than assumed from the merge.
+- **The security boundary is live.** `ClusterPolicy/pm-agent-job-shape`:
+  `validationFailureAction=Enforce`, `failurePolicy=Fail`, `Ready=True`, and all six rules
+  present (`worker-must-be-the-committed-shape`,
+  `worker-must-be-one-container-with-no-args`,
+  `worker-volumes-must-be-workspace-or-tmp-emptydirs`,
+  `worker-may-not-declare-init-or-ephemeral-containers`,
+  `worker-env-names-must-be-the-committed-set`, `worker-may-not-use-envfrom`). This is the
+  check that mattered most: a ClusterPolicy added to the wrong kustomization list ships
+  green and never reaches the cluster, which is what happened to the conformance boundary
+  on 2026-09-29.
+- Namespace contents: both ServiceAccounts, `pm-agent-config` (17 keys), the
+  `pm-agent-credentials` Secret, the `default-deny` NetworkPolicy, and both
+  CiliumNetworkPolicies with `VALID=True`.
+- **SOPS decryption works through the new overlay patch.** Checked by KEY NAMES and VALUE
+  LENGTHS only, never values (CLAUDE.md §6): keys `gitea-token`, `claude-code-oauth-token`,
+  lengths 76 and 52 bytes — which are the base64 lengths of the committed placeholder
+  strings, so the Secret holds the decrypted placeholders and not ciphertext. A
+  ciphertext apply succeeds silently and would have left `ENC[AES256_GCM,…]` values an
+  order of magnitude longer.
+- **`pm-agent-worker` really is bound to nothing.** The namespace has exactly one
+  RoleBinding (`pm-agent` → Role `pm-agent`, subject `ServiceAccount/pm-agent`), and no
+  ClusterRoleBinding anywhere names either pm-agent ServiceAccount.
+- `gitea-allow-pm-agent-ingress` is applied in the `gitea` namespace, matching
+  `io.kubernetes.pod.namespace: pm-agent`.
+- The dispatcher is `ImagePullBackOff` on the all-zero placeholder digest — the designed,
+  visible failure.
+
+**The two prerequisites, measured.**
+
+1. **The nodes cannot verify the registry's TLS certificate.** containerd's error on the
+   pull is not about the digest:
+
+       failed to resolve reference "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:000…":
+       tls: failed to verify certificate: x509: certificate is not valid for any names,
+       but wanted to match gitea.home.keithjackson.dev
+
+   "not valid for **any** names" means the certificate presented to the node carries no
+   matching SAN at all. From the operator's workstation the same name is healthy: it
+   resolves to `192.168.2.20` and serves a Let's Encrypt certificate
+   `CN=*.home.keithjackson.dev` with SANs `*.home.keithjackson.dev, home.keithjackson.dev`
+   — and `Certificate gateway/wildcard-home` is `Ready=True` for exactly those names. So
+   the certificate that *should* be served is fine, and what the NODE reaches is not what
+   the workstation reaches. **Correcting the digest will not clear this.**
+2. **Anonymous pulls from this registry are refused.** Not inferred from a bare 401 — that
+   would have been wrong, because this Gitea answers 401 for *everything* anonymously:
+   `/v2/home-network/agent-runner/manifests/latest`, `/v2/home-network/conformance-runner/…`,
+   a randomly-named nonexistent repository, and `nosuchowner/nosuchimage` all return 401,
+   and `/api/v1/packages/Home-Network` returns `{"message":"token is required"}`. The
+   registry *does* issue a token to an anonymous client, but its `access` claim is `null` —
+   no pull scope. So the conclusion that holds is the simple one: **an unauthenticated
+   client gets nothing from this registry**, so a pull with no credential cannot succeed.
+   And `grep` over the tree finds **no `imagePullSecrets` anywhere** — not in
+   `platform/pm-agent/deployment.yaml`, not in `infra/agent/worker-job-template.yaml`, not
+   in `conformance/incluster/`.
+
+## What I did NOT verify
+
+- **Why the node sees a nameless certificate.** I measured that it does, and that the LAN
+  does not. I did not diagnose it: I did not read the nodes' resolver or
+  `/etc/rancher/k3s/registries.yaml` (node files, not reachable read-only through the API),
+  and I did not check whether the node resolves the name to the same `192.168.2.20` the
+  workstation does or to something else.
+
+  **A correction to a remark I first wrote here:** I cited
+  `cluster-vars`'s `GATEWAY_VIP: 192.168.1.53` as suspicious next to the nodes'
+  `192.168.2.x`. That was noise — the operator confirms those base `cluster-vars` values are
+  **dev-era leftovers for a cluster that is decommissioned**, so the 192.168.1.x addresses in
+  that file say nothing about prod and should not be read as evidence about anything. The
+  measured facts stand on their own and none of them depended on that knob.
+- **Whether the package would be pullable once authenticated.** Nothing has ever been
+  pushed to that path, so "a credential is sufficient" is untested. The only thing
+  established is that no credential is insufficient.
+- **Which fix the operator wants.** Two routes, and they are not equivalent:
+  (a) an `imagePullSecret` — a `dockerconfigjson` holding a Gitea token, SOPS-encrypted
+  like `pm-agent-credentials`, referenced from the Deployment *and* from the committed
+  worker Job template; note the Job template's shape is pinned by admission, and
+  `imagePullSecrets` is a field that policy does not currently constrain, so adding one is
+  a template change that needs a deliberate look at whether the policy should pin it too.
+  (b) allowing anonymous reads of that package in Gitea — no manifest change at all, but it
+  makes the image world-readable to anything that can reach the host.
+  I have implemented neither: both need an operator action I must not take, and (b) is a
+  posture decision. The docs now present both.
+- **Everything the #57 entry listed as unverified about RUNTIME remains unverified.** The
+  image has still never been built or pushed. No dispatcher has authenticated to Gitea, no
+  issue has been triaged, no worker Job has been created — so the admission policy, though
+  now provably applied and Enforce, has still never refused or admitted a real Job. The
+  network policies have not carried traffic.
+- **I did not re-run the merge gate against `main` after the merge.** This branch's own gate
+  run is the evidence for this change; the merged state's is CI run on PR #57.

decisions/2026-10-05-vpn-runbook-ansible-command.md

@@ -0,0 +1,112 @@
+---
+date: 2026-10-05
+branch: fix/vpn-runbook-ansible-command
+title: vpn-remote-access runbook — step 1 named a script that does not exist, and step 2 cited the wrong reason the key is gitignored
+---
+
+## What changed
+
+Two defects in `runbooks/vpn-remote-access.md` (merged in #55), both authored by me, both reported
+by the operator the first time they tried to follow it.
+
+**1. Step 1 named a command that does not exist.** It said:
+
+    cd infra/ansible && ./site.sh --tags system --limit k3s_prod
+
+There is **no `site.sh` anywhere in this repo** (`find . -name site.sh` → nothing). The invocation
+was wrong in four separate ways at once, which is what makes it worth recording rather than just
+patching: wrong kind of thing (this repo drives Ansible with `ansible-playbook <playbook>`, never a
+wrapper script — `playbooks/site.yml` exists and is the full bring-up chain, not this); wrong
+playbook (the module work lives in `playbooks/node-baseline.yml`); wrong targeting flag (`--limit`
+instead of this repo's `-e k3s_hosts=...` convention, which `infra/ansible/README.md` "Targeting
+prod" documents for every playbook); and no toolbox wrapper, though Ansible exists only inside the
+pinned container and `ansible.cfg` is only read from `infra/ansible`.
+
+Replaced with the real command, plus the three things about it that are load-bearing:
+
+    ./infra/devcontainer/dev.sh run bash -c \
+      'cd /work/infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags system'
+
+- `-e k3s_hosts=k3s_prod` — the playbook defaults to `k3s_servers`, which is the **dev** group;
+  dev is decommissioned, so an unqualified run does nothing useful.
+- `--tags system`, never untagged — the two kernel-module tasks carry that tag, and an untagged run
+  would also hit the `disks` tasks, which assert on block-device variables prod deliberately leaves
+  as placeholders (the playbook's own header warns about this). `--tags system` is broader than the
+  modules alone (sysctls, iscsid, qemu-guest-agent, swap masking) but each is idempotent and
+  already in its desired state on a live node.
+
+**2. Step 2 gave the wrong reason the server key is safe from git — and the right reason is
+fragile.** It told the operator to stage the key at `/work/.cache/server.key` "inside the repo's
+gitignored cache". `.cache/` is **not** gitignored: `git check-ignore -v .cache/serverkey.txt`
+matches nothing. The file was ignored only incidentally, by `.gitignore` line 11 `*.key` — so the
+path happened to be safe for a reason I had not identified, and **any operator who renamed it
+(`server.pem`, `wg-server`, `privatekey`) would have had a committable WireGuard private key sitting
+in the checkout.** The gate would not have caught it either; `no-plaintext-secrets` does not
+recognise a bare WireGuard key.
+
+The step now states the real rule, marks the `.key` suffix as the thing doing the work, and gives
+the operator a verification they can run before trusting it (`git check-ignore -v` must print a
+rule, and `git status --short` must not list the file).
+
+Nothing outside `runbooks/vpn-remote-access.md` changed; no manifest, no behaviour.
+
+## What I verified
+
+- **`site.sh` does not exist**: `find . -name 'site.sh' -not -path './.git/*'` → no output.
+- **The real surface**: `infra/ansible/playbooks/` contains `node-baseline.yml` and `site.yml` (and
+  11 others); `node-baseline.yml`'s own header documents both
+  `-e k3s_hosts=k3s_prod` and the tag-scoping hazard. The `-e <playbook>_hosts=` convention is
+  confirmed by `infra/ansible/README.md` §"Targeting prod" and by the existing
+  `runbooks/node-memory-swap.md`, which invokes `ansible-playbook playbooks/node-health.yml -e
+  node_health_hosts=...`.
+- **`k3s_prod` is a real inventory group**: `[k3s_prod:children]` in `inventory/hosts.ini`, with
+  `[k3s_prod_primary]`/`[k3s_prod_secondary]`.
+- **`--tags system` covers the kernel-module tasks**: both "Persist kernel modules…" and "Load
+  kernel modules now" carry `tags: [system]` in `roles/node_baseline/tasks/main.yml`.
+- **The gitignore claim, measured both directions**:
+  `git check-ignore -v .cache/server.key` → `.gitignore:11:*.key`;
+  `git check-ignore -v .cache/serverkey.txt` → no match. So the suffix is the only thing ignoring it.
+- **Every path the component's docs reference resolves**: all six relative markdown links in
+  `platform/vpn-ingress/README.md` (checked from that file's own directory), and both Ansible
+  playbook paths.
+- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`,
+  `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; the rc 3 is the two
+  environmental skips (`gitea-settings` — no local token; conformance L2/L3 — no `CLUSTER_AVAILABLE`).
+  **Not green. Reported as rc 3, not as pass.**
+- **This branch touches NO sensitive path** (`runbooks/**` and `decisions/**` are not in
+  `infra/gitea/approval-policy.json`), so unlike #55 it needs no `/approve-override`.
+
+## What I did NOT verify
+
+- **The corrected command has not been RUN.** Ansible is an operator action, and this is the
+  decisive gap: I have verified the playbook, the group, the tag and the flag all exist and are
+  the documented convention, but not that the run succeeds end to end or that
+  `lsmod | grep '^wireguard'` then reports the module. It is a better-evidenced command, not a
+  proven one.
+- **Whether `--tags system` is entirely side-effect-free on a live prod node.** Each task it runs is
+  idempotent by construction, but I did not dry-run it (`--check`) to confirm it reports zero
+  changes beyond the module load.
+- **The rest of the runbook's commands past step 2** are unchanged and still unexercised — in
+  particular `kubectl exec ds/wireguard -- wg show`, which cannot be reached while the pods sit in
+  `Init` waiting on the server key.
+- **Whether any OTHER document I wrote in #55 contains an invented command.** I re-checked every
+  file path and markdown link in the component (all resolve) and every script the docs tell an
+  operator to run, but "the referenced file exists" is weaker than "the command is correct" — which
+  is exactly the gap that produced `site.sh`.
+
+## Post-merge state of #55, recorded here since its own entry predates the merge
+
+Measured on live prod after #55 merged as `ad3f56e`:
+
+- Flux applied it by the real path: Kustomization `vpn-ingress` `lastAppliedRevision:
+  main@sha1:ad3f56e`, equal to the GitRepository's artifact revision, `Ready=True`.
+- All **18** Flux Kustomizations `Ready=True` — the new tier disturbed nothing, and `wait: false`
+  behaved as intended (`vpn-ingress` is Ready while its own pods are not, so nothing downstream
+  stalled on it).
+- **`hostNetwork` is confirmed in effect, not merely configured**: the three pods report pod IPs
+  `192.168.2.10/.11/.12` — the node addresses, one per node.
+- **It fails closed exactly as designed**, with a legible cause: all three pods `Init:0/1`,
+  `MountVolume.SetUp failed for volume "wg-server-key" : secret "wg-server-key" not found`, and no
+  Secret in the namespace. The #55 entry's "the DaemonSet has never been reconciled" is superseded
+  by this paragraph; its remaining unverified claims (the tunnel carrying a packet, the
+  iptables/nftables backend question) still stand.

decisions/2026-10-05-vpn-server-key-in-sops.md

@@ -0,0 +1,187 @@
+---
+date: 2026-10-05
+branch: fix/vpn-server-key-in-sops
+title: VPN server key moves into SOPS-encrypted git — the out-of-band Secret broke R-SEC-01 and the DR story
+---
+
+## What changed
+
+**Operator correction (2026-10-05): "secrets are kept in source through sops".** They are right, and
+#55 shipped the WireGuard server private key the wrong way: as a Secret the operator was told to
+`kubectl create` out of band, referenced by name only. This branch moves it into the tree as
+`platform/vpn-ingress/wg-server-key.sops.yaml`, where every other secret in this repo lives.
+
+**Why the original choice was wrong, not merely unconventional.** Two concrete costs, both of which
+#55's own README half-admitted and then shipped anyway:
+
+1. **R-SEC-01 / spec docs/07 says secrets live in source, SOPS-encrypted, reaching the cluster only
+   through Flux.** An out-of-band Secret is outside the GitOps tree by construction.
+2. **It broke DR for the one credential that cannot be re-derived.** A cluster rebuild restores
+   everything in git; it does not restore a hand-made Secret. Losing this key means **re-enrolling
+   every device**, because each client profile pins its public half. #55 papered over this with
+   "keep a copy in your password manager", which is a workaround for a self-inflicted gap.
+
+Note `platform/kyverno/admission/secret-provenance.yaml` did **not** forbid the original shape —
+it scopes to `app-namespace=true` namespaces and `vpn-ingress` is deliberately a platform namespace,
+so admission would have accepted the hand-applied Secret. The rule it violated was the written
+requirement, not an enforced control. Recording that because "the gate let it through" was never
+evidence it was correct.
+
+**How the unfillable-key problem is solved.** An agent still must not mint a long-lived private key
+(CLAUDE.md §6), so the file is committed **already encrypted, with `privatekey: REPLACE-ME`**. That
+makes the wiring complete and the gate green from the first commit — the resource line and both
+overlay decryption patches are present — and leaves the operator exactly one act: replace the value.
+The alternative, a commented-out resource line to uncomment later, is precisely the "file in git with
+no generator entry: reviewed, merged, apparently done, still broken" shape this repo keeps getting
+bitten by (`alert-runbooks-resolvable` exists because of it).
+
+**The init-container guard is now a validation, not an emptiness check.** It was `[ -z "$key" ]`.
+Because the committed placeholder is non-empty, the likely failure is now a value that is present
+and *wrong*, so the guard pipes the value through `wg pubkey` — which accepts only a real 32-byte
+curve25519 private key — and names which of four cases it hit:
+
+| case | why it matters |
+|---|---|
+| `REPLACE-ME` placeholder | the expected fresh-install state |
+| empty | blanked value |
+| **starts with `ENC[`** | **the Secret reached the cluster UNDECRYPTED.** A missing overlay decryption patch applies cleanly and leaves every Flux condition green — this is the silent failure the guard exists for |
+| rejected, none of the above | mangled value, or a *public* key pasted by mistake |
+
+Files:
+
+- `platform/vpn-ingress/wg-server-key.sops.yaml` (new) — SOPS-encrypted to the repo's age recipient,
+  `stringData.privatekey` only, placeholder value.
+- `platform/vpn-ingress/wg-server-key.sops.yaml.example` (new) — plaintext template + operator
+  hand-off, following the `crafty-oauth2-proxy-credentials.sops.yaml.example` convention.
+- `platform/vpn-ingress/kustomization.yaml` — the resource added; the "DELIBERATELY ABSENT: a
+  Secret" block replaced with the overlay-patch warning.
+- `platform/vpn-ingress/daemonset.yaml` — the `wg pubkey` guard; volume comment rewritten.
+- `clusters/dev/kustomization.yaml`, `clusters/prod/kustomization.yaml` — **the `vpn-ingress`
+  `decryption:` patch, in both.** This is the load-bearing half: #55's own kustomization comment
+  predicted that forgetting it applies ciphertext cleanly.
+- `clusters/base/vpn-ingress.yaml` — its "NO decryption stanza is needed" comment was now false.
+- `platform/vpn-ingress/README.md`, `runbooks/vpn-remote-access.md` — step 2 rewritten.
+- `infra/vpn-ingress/new-peer.sh` — stale "step 1" pointer.
+
+**A second finding: `wg` and `sops` were not in the same place — and the first fix for that was
+also wrong.** Measured: `wg` existed only on the host (`/usr/bin/wg`), `sops` only in the dev
+container, `qrencode` on neither. So `sops -d … | wg pubkey` could not run in one shell, and #55's
+step 2 — which assumed it could — would have failed.
+
+My first fix was to **split every step by where it runs**, and to tell the operator to
+`dnf install qrencode` on their workstation. **Operator correction: "sop, wg, and any other tool
+requirement is supposed to be baked into the devcontainer."** That is CLAUDE.md §5 and the
+devcontainer README's "Don't install the toolchain on the host", and the split fix broke the
+invariant rather than honouring it — it also produced a procedure that could never be reproduced in
+CI, and which no one should have to follow.
+
+**The real fix:** `wireguard-tools` and `qrencode` are now distro packages in
+`infra/devcontainer/Containerfile`, alongside ansible/openssh/diffutils. Both are in Fedora 44's
+repos (`wireguard-tools` 1.0.20260223 — the same upstream version as the container image this
+component runs — and `qrencode` 4.1.1), unpinned like every other dnf package there, because the
+pinned thing is the Fedora base and neither is a gate tool whose version changes what green means.
+
+Three things got *simpler* rather than merely relocated, which is the tell that the operator's rule
+was the right one:
+
+1. **Step 2 is now one pipeline in one shell:** `sops set … "\"$(wg genkey)\""` generates the key
+   straight into the already-encrypted document. The plaintext never lands on disk, never reaches a
+   terminal, and there is no staged copy to shred. The previous version needed a copy in
+   `.cache/server.key`, a `git check-ignore` warning about the `*.key` suffix being the only thing
+   protecting it, and a sha256 comparison to prove the right bytes arrived. All of that is gone.
+2. **`new-peer.sh` now derives the server's public key from `wg-server-key.sops.yaml` itself**
+   (`sops -d --extract … | wg pubkey`), so there is no local key copy to keep in step with the
+   encrypted one. It previously required the operator to keep the private key at a magic host path.
+3. **`.cache/` is now gitignored at the repo root** (mirroring `infra/ansible/.gitignore`), so the
+   container has a persistent, non-committable scratch directory. `new-peer.sh` writes client
+   profiles to `.cache/wg-peers/` by default — necessary because the repo bind-mount is the
+   container's ONLY persistent read-write mount, so a profile written to `/tmp` in there is gone
+   before it can be moved to a desktop. Its old blanket "refuse to write inside the checkout" guard
+   is replaced by `git check-ignore`, which tests the actual property instead of a proxy.
+
+**SENSITIVE PATHS — this PR touches `clusters/**` (two overlays) AND adds a `**/*.sops.yaml`, so
+`ci/approval-policy.sh` will BLOCK it.** It needs the operator's `/approve-override <reason>` as a
+PR **review** of type "Comment". I have deliberately not posted it (CLAUDE.md §1).
+
+## What I verified
+
+- **The encrypted file is real and correctly targeted.** `sops -e -i` in the pinned toolbox
+  produced `stringData.privatekey: ENC[AES256_GCM,...]` with cleartext keys, `encrypted_regex
+  ^(data|stringData)$`, and exactly one age recipient:
+  `age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg` — the recipient in the root
+  `.sops.yaml`, i.e. the key Flux holds as `sops-age`.
+- **Round-trip, without exposing anything**: `sops -d` then a Python read of key names only →
+  `stringData keys: ['privatekey']`, and `value is still the placeholder: True`.
+- **Both overlay patches are present and name the right Kustomization** (`name: vpn-ingress`,
+  `namespace: flux-system`, `decryption.secretRef.name: sops-age`), inserted after each overlay's
+  `velero` stanza, mirroring headlamp/crafty/velero.
+- **`kubectl kustomize platform/vpn-ingress` builds** and now emits five objects — Namespace,
+  ConfigMap, **Secret**, DaemonSet, NetworkPolicy — with `${WG_CLIENT_CIDR}` still an unexpanded
+  token for Flux.
+- **The tool gap, measured on both host and container before the fix**: `wg` host-only, `sops`
+  container-only, `qrencode` on neither.
+- **Both packages exist in the image's repos**, queried from inside the running container:
+  `wireguard-tools` 1.0.20260223 (repo `updates`) and `qrencode` 4.1.1 (repo `fedora`).
+- **The image REBUILDS with them and both tools resolve inside it** —
+  `./infra/devcontainer/dev.sh build` succeeded, and `wg --version` / `qrencode --version` /
+  `sops --version` all answer in the rebuilt container. This is the claim the change rests on, so
+  it was executed rather than reasoned.
+- **`new-peer.sh`'s new SOPS path fails correctly against the placeholder.** Run in the rebuilt
+  container it refused with "could not derive the server public key … it still holds the REPLACE-ME
+  placeholder", which is both the expected pre-key behaviour and a live test of the
+  `sops -d | wg pubkey` pipeline the whole design now depends on.
+- **`.cache/` is ignored regardless of filename** now: `git check-ignore -v .cache/anything.conf`
+  and `.cache/serverkey.txt` both match `.gitignore:46:.cache/`. Before this change only `*.key`
+  matched, which is the fragility the previous runbook had to warn about.
+- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`,
+  `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; rc 3 is the two
+  environmental skips (`gitea-settings` — no local token; conformance L2/L3 — no cluster).
+  **Not green. Reported as rc 3, not as pass.**
+- **The L0 gate caught THREE real defects in this branch before it could merge — including one
+  in the fix for another — and I had asserted one of them was fine without running it.** Both are recorded because the gate's value here is the
+  whole point of the exercise:
+  1. **`no-plaintext-secrets` FAILED**: "`wg-server-key.sops.yaml.example`: an unencrypted .example
+     template is referenced by a kustomization — it would be applied as a plaintext Secret". My
+     kustomization named the template in a *comment*, and the check matches comments too (correctly:
+     naming one is how a plaintext Secret gets applied by accident). `platform/crafty/kustomization.yaml`
+     never names its own example either — the convention was there to copy and I missed it. Fixed by
+     not naming the file; the hand-off lives inside the example, where it belongs.
+     **An earlier draft of this entry claimed this check passed. It did not. The claim was written
+     from expectation rather than from a run.**
+  2. **`flux-postbuild-substitution` FAILED** on `${key#ENC[}` in my new guard
+     (`daemonset.yaml:155`), flagged `[SILENT-EMPTY]`: envsubst consumes every `${...}`, so my
+     shell prefix-strip would have rendered to the empty string — the ciphertext branch of the guard
+     would never have fired, which is precisely the silent failure the guard was added to catch.
+     Rewritten with `case`, which uses no braces and so has nothing to escape.
+  3. **And then it failed a THIRD time, on the fix's own comment** — `[FATAL-PARSE]
+     daemonset.yaml:156 ${...}`, the prose explaining the rule, written with the literal form. That
+     class is worse than it looks: a `FATAL-PARSE` hazard fails the **entire** Kustomization, every
+     resource in the path, not just the line. Rewritten as "dollar-brace expansion". This is the
+     second time in this component that writing the hazard literally inside a comment about the
+     hazard has been the defect (the first was in #55), so the habit to build is: **in any
+     substituted path, describe the construct in words, never type it.**
+- **After both fixes, `no-plaintext-secrets` and `flux-postbuild-substitution` pass** (446 and 198
+  assertions respectively), and `l0` is green.
+
+## What I did NOT verify
+
+- **That Flux decrypts this file on prod.** The overlay patches have never reconciled. This is the
+  whole risk of the change and it has exactly one observable outcome: after merge the pods should
+  CrashLoop reporting the **`REPLACE-ME` placeholder** (proving decryption worked), *not* reporting
+  **`ENC[`** (which would prove a patch is wrong). I built the guard to tell those apart because I
+  cannot test them.
+- **That `wg pubkey` accepts the operator's real key through this path.** The guard is reasoned from
+  `wg pubkey`'s behaviour and has never run — `wg` is absent from both this workstation's container
+  and the gate, so the first execution will be in the pod.
+- **The `sops set` command in the hand-off.** I verified `sops -e -i` and `sops -d` in the toolbox;
+  I did **not** run `sops set` with the quoting shown, and nested quoting inside
+  `dev.sh run bash -c` is exactly where a documented command tends to break. If it misbehaves, the
+  editor form (`sops <file>`) is the fallback and does not depend on quoting.
+- **The sha256 comparison in step 2 step 4** is untested end to end (it needs the real key).
+- **Whether anything else in #55 assumed a tool is available where it is not.** I checked `wg`,
+  `sops` and `qrencode` because step 2 depended on them. I did not audit every command in the
+  runbook the same way — and that class of defect has now produced two corrections in a row
+  (`site.sh`, then `sops | wg`), so assume more remain until each step has actually been run.
+- **The live VPN still carries no traffic**, and the earlier gaps stand unchanged: the
+  iptables-vs-nftables question, `allowPrivilegeEscalation: false` versus the image's s6 init, and
+  any off-LAN reachability.

decisions/2026-10-06-new-peer-ignores-template-comments.md

@@ -0,0 +1,73 @@
+---
+date: 2026-10-06
+branch: fix/new-peer-ignores-template-comments
+title: new-peer.sh refused the first VPN address because its duplicate check matched its own commented-out example
+---
+
+## What changed
+
+The operator's first real enrolment failed:
+
+```
+$ ./infra/vpn-ingress/new-peer.sh keith-phone 2
+new-peer.sh: 10.90.0.2/32 is already allocated in platform/vpn-ingress/peers.yaml. Pick another octet.
+```
+
+**Nothing was allocated.** `peers.yaml` had zero peers — the pod's own init container reported
+`wg0.conf assembled: 0 peer(s)`. The duplicate-address check grepped the WHOLE file, and
+`peers.yaml` line 66 is a **commented-out example stanza** I wrote, containing literally
+`#   AllowedIPs = 10.90.0.2/32`. So `.2` — the first address anybody would ask for, and the one my
+own runbook uses in its example — was permanently un-allocatable, with a message asserting the
+opposite of the truth.
+
+Both halves fixed, because either alone leaves the trap half-set:
+
+- **`infra/vpn-ingress/new-peer.sh`**: the check strips comment lines before matching
+  (`grep -vE '^[[:space:]]*#' | grep -q …`). That is the correct predicate rather than a patch: a
+  real stanza's `AllowedIPs` line is never commented, while both the template and every peer's
+  `# <name> — <owner>` label always are.
+- **`platform/vpn-ingress/peers.yaml`**: the example now uses placeholders that cannot match
+  anything real — `# <device-name> — <owner>, enrolled <YYYY-MM-DD>` and
+  `AllowedIPs = 10.90.0.<octet>/32`. A future grep cannot be fooled by it either.
+
+The bug class is worth naming: **a documentation example that is byte-identical to real data will
+eventually be parsed as real data.** The same shape caused two earlier defects in this component —
+`flux-postbuild-substitution` flagged a `${...}` written inside a comment *explaining* that hazard,
+and `no-plaintext-secrets` flagged the `.sops.yaml.example` because the kustomization named it in a
+comment. Three instances now; the rule is to make examples structurally unmatchable, not merely
+commented out.
+
+No sensitive paths (`platform/vpn-ingress/**`, `infra/vpn-ingress/**`, `decisions/**`), so this
+merges on a green gate with no override.
+
+## What I verified
+
+Run in the pinned toolbox against the live tree, 2026-10-06.
+
+- **`.2` now works**, and it went through the REAL key path: `new-peer.sh keith-phone 2` rendered a
+  profile with `Address 10.90.0.2/32`, `AllowedIPs = 10.90.0.2/32` in the printed stanza, full
+  tunnel, at `.cache/wg-peers/keith-phone.conf` (0600). The throwaway keypair was deleted after.
+- **This incidentally closes a gap left open in #60**: the profile could only have been rendered by
+  deriving the server's public key via `sops -d --extract … | wg pubkey` from
+  `wg-server-key.sops.yaml` — so that pipeline is now proven **against the operator's real key**,
+  not just against a placeholder it was expected to reject.
+- **A real duplicate is still refused.** Appending an uncommented
+  `AllowedIPs = 10.90.0.5/32` stanza and asking for octet 5 produced the correct refusal; the tree
+  was restored afterwards and `git diff --stat` showed only the intended template edit.
+- **Zero active allocations remain**: `grep -vE '^[[:space:]]*#' peers.yaml | grep -c AllowedIPs`
+  → `0`.
+- **Toolbox gate**: rc 3, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`,
+  `l1` and `approval-policy` all pass; rc 3 is the two environmental skips (`gitea-settings`,
+  conformance L2/L3). **Not green. Reported as rc 3.**
+
+## What I did NOT verify
+
+- **No peer is enrolled by this branch.** The keypair I generated was a throwaway and was deleted;
+  the operator must run the script themselves, since the device's private key must exist only on
+  the device. So the end of the enrolment path — a stanza merged into `peers.yaml`, Flux rolling
+  the DaemonSet, and `wg show` listing a peer — is still unexercised.
+- **The duplicate-NAME warning** (`grep -qi "^ *# *${NAME} "`) deliberately still inspects
+  comments, because per-peer labels are comments. It is no longer tripped by the template, but I
+  did not test it against a real enrolled peer.
+- **Nothing about client traffic.** The WAN port-forward is still not configured and there is still
+  no off-LAN vantage point, so the tunnel has carried no packet.

decisions/2026-10-06-peers-configmap-content-hash.md

@@ -0,0 +1,103 @@
+---
+date: 2026-10-06
+branch: fix/peers-configmap-content-hash
+title: A merged VPN peer never reached the tunnel — the peer register was a fixed-name ConfigMap, so nothing rolled the pods
+---
+
+## What changed
+
+The operator enrolled a phone, merged the peer, and **`wg show` listed no peers.** Measured on prod
+2026-10-06, with the peer already on `main`:
+
+| observation | value |
+|---|---|
+| peer stanza on `main` | present (`AllowedIPs = 10.90.0.2/32`) |
+| GitRepository artifact | `main@sha1:e1bebb8` — Flux had fetched it |
+| `vpn-ingress` Kustomization `lastAppliedRevision` | `main@sha1:5928d81` — **behind**, 10m interval |
+| live `wg-peers` ConfigMap, active `AllowedIPs` lines | **0** |
+| DaemonSet volume reference | `configMap.name: wg-peers` — a **fixed** name |
+| DaemonSet `generation` / `observedGeneration` | **1 / 1** — it had **never rolled** |
+| init container, all three pods | `wg0.conf assembled: 0 peer(s)` |
+
+The lag was only the reconcile interval. **The defect is the fixed name.** When Flux does update a
+fixed-name ConfigMap it updates it *in place*: the DaemonSet's pod template is byte-identical, so
+no rollout is triggered — and because the init container assembles `wg0.conf` **once at pod start**
+into a tmpfs, the running tunnel keeps the old peer list indefinitely. Adding or revoking a peer
+would have had **no effect at all** until something unrelated happened to restart the pods. For a
+*revocation* that is a security defect, not an inconvenience: the operator would delete a stanza,
+see a green PR, and the device would keep working.
+
+My runbook and README both asserted "PR → merge → Flux rolls the DaemonSet (~1 min)". That was
+never true.
+
+**This repo already documents the fix, and I had read the file.** `runbooks/kustomization.yaml`:
+*"The generated name carries kustomize's content hash, so editing any runbook rolls the server
+automatically and a stale projection is impossible."*
+
+So the register becomes a **content-hashed generator**:
+
+- `platform/vpn-ingress/peers.yaml` (inline ConfigMap) → **`platform/vpn-ingress/peers.conf`**, a
+  plain file, plus a `configMapGenerator` in `kustomization.yaml`. The operator's enrolled peer was
+  carried across verbatim. Every line is WireGuard config and `#` is a legal comment there, so the
+  documentation now travels *inside the mounted file* instead of in a YAML wrapper that gets
+  stripped.
+- **`namespace: vpn-ingress` added to the kustomization — REQUIRED, not tidiness.** A generated
+  ConfigMap inherits no namespace; every other object here declares `vpn-ingress`; and kustomize's
+  `nameReference` transformer will not rewrite a reference across a namespace boundary. Without it
+  the build emitted `ConfigMap wg-peers-m9kcckcb5t` while the DaemonSet still mounted the unhashed
+  `wg-peers`, which would not exist — the pods would wedge on a missing volume. I caught this only
+  by building the output and reading the volumes block.
+- **`infra/vpn-ingress/new-peer.sh`**: `PEERS_FILE` repointed to `peers.conf`, and a missing
+  register is now **fatal** instead of skipped. The old guard was
+  `[ -r "$PEERS_FILE" ] && grep …`, so after the rename the only symptom would have been
+  duplicate-address detection quietly never firing again — the silent-skip shape this repo treats
+  as a defect in its own right.
+- Stale `peers.yaml` references swept from the README, runbook, `daemonset.yaml`,
+  `networkpolicy.yaml`, the SOPS example, and the script's own **user-facing output** (it told the
+  operator to paste into "peers.yaml, under data.peers.conf", which would no longer exist).
+- The runbook gained the verification that actually distinguishes this failure — the DaemonSet's
+  `generation` must *increase* after the merge — and a troubleshooting row for `0 peer(s)` after a
+  merge.
+
+Flux's `prune: true` removes the orphaned fixed-name `wg-peers` ConfigMap on the next reconcile.
+
+No sensitive paths (`platform/vpn-ingress/**`, `infra/vpn-ingress/**`, `runbooks/**`,
+`decisions/**`), so this merges on a green gate with no override.
+
+## What I verified
+
+Built and run in the pinned toolbox, 2026-10-06 (kustomize v5.8.1 via `kubectl kustomize`).
+
+- **The reference is now rewritten end to end**: generated `ConfigMap wg-peers-m9kcckcb5t` in
+  namespace `vpn-ingress`, and the DaemonSet mounts **`wg-peers-m9kcckcb5t`**. Before adding
+  `namespace:` the same build mounted the unhashed `wg-peers` — so the namespace line is load
+  bearing, measured both ways.
+- **The hash tracks content, which is the whole mechanism.** `wg-peers-m9kcckcb5t` →
+  append a second peer → `wg-peers-dgg7ht9c74` → restore → `wg-peers-m9kcckcb5t`. Deterministic
+  and reversible, so an edit necessarily changes the pod template.
+- **The operator's enrolled peer survived the file conversion byte-for-byte**: the active stanza in
+  `peers.conf` (`PublicKey = pBRSHrRy…`, `AllowedIPs = 10.90.0.2/32`) is identical to the one on
+  `main`, compared directly against `git show origin/main:platform/vpn-ingress/peers.yaml`.
+- **The build still emits all five objects**: Namespace, ConfigMap, Secret, DaemonSet,
+  NetworkPolicy — and the `Namespace` object's own name is still `vpn-ingress` (the namespace
+  transformer did not rewrite it).
+- **`new-peer.sh` works against the new register**: `spare-laptop 7` renders a profile and prints
+  "append this to platform/vpn-ingress/peers.conf"; `dupe 2` is **refused** because the operator's
+  real `.2` now occupies it — the first time the duplicate check has been exercised against real
+  data rather than against its own example.
+- **Toolbox gate**: rc 3, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`,
+  `l1` and `approval-policy` all pass; rc 3 is the two environmental skips. **Not green. Reported
+  as rc 3.**
+
+## What I did NOT verify
+
+- **That the roll actually happens on the cluster.** The whole point of the change is unproven
+  until Flux applies it: the success criterion is the DaemonSet's `generation` increasing and
+  `wg0.conf assembled: 1 peer(s)` in the init log, followed by `wg show` listing the peer.
+- **That the tunnel then carries traffic.** No handshake has ever been observed. The WAN
+  port-forward is reportedly configured but has never been exercised by a packet I can see, and I
+  have no off-LAN vantage point — runbook step 6 from cellular remains the only honest test.
+- **Revocation**, which is the case where this defect was a security problem. Deleting a stanza
+  should now roll the pods and drop the peer; untested.
+- **Whether the orphaned fixed-name `wg-peers` ConfigMap is actually pruned.** Reasoned from
+  `prune: true` on the Kustomization, not observed.

decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md

@@ -0,0 +1,126 @@
+---
+date: 2026-10-06
+branch: docs/pm-agent-registry-pull-prereq
+title: The nodes resolve gitea to the WAN, so the failing certificate is the router's — the second node process to need the issuer pin
+---
+
+## What changed
+
+The image pull for `pm-agent` failed TLS verification, and the previous entry on this branch
+honestly recorded it as **not diagnosed**. The operator asked for the diagnosis before any
+registry design. This is it, and the fix turns out to be a pattern this repository already
+has.
+
+`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` now pins
+`gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `node_baseline_hosts_pins`, beside the
+Keycloak issuer pin that is there for the identical reason.
+
+**It reaches the nodes the way every other node setting in this project does — through
+Ansible, from the inventory.** Node configuration is Ansible's (and, for the dev VMs,
+OpenTofu's); nothing here touches a node by hand. For prod specifically, Ansible is the
+*only* surface: `infra/PROD-PROVISIONING.md` records that prod is 3× bare metal and that the
+dev path (OpenTofu → Proxmox VM → cloud-init) does not apply, so there is no
+cloud-init/tofu half to keep in parity for this. The one scoped, idempotent command:
+
+    cd infra/ansible && ansible-playbook playbooks/node-baseline.yml \
+      -e k3s_hosts=k3s_prod --tags hosts
+
+`--tags hosts` runs ONLY the `/etc/hosts` task, so applying this pin cannot drift any other
+baseline setting. Running the play is an operator action (it has become/root on prod), but
+the configuration itself is declared, reviewed and version-controlled — not typed on a node.
+
+The docs that described this as an undiagnosed blocker now describe the cause and the fix:
+`platform/pm-agent/README.md`, `runbooks/pm-agent.md` (its symptom row now names the pin and
+the play), and `docs/plans/pm-agent-issue-to-pr.md`.
+
+Also recorded, in the two places an operator reads when creating the credential: **the agent
+authenticates to Gitea with a local account and a scoped PAT, by decision** (operator,
+2026-10-06), not with a federated identity. Written down because the alternative looks
+obviously better until you check whether it is possible — see below.
+
+## What I verified
+
+Every step measured, read-only, in this order.
+
+**The failing certificate is the router's, not the Gateway's.**
+
+- containerd's error names TLS, not the digest: `tls: failed to verify certificate: x509:
+  certificate is not valid for any names, but wanted to match gitea.home.keithjackson.dev`.
+  "not valid for **any** names" means the presented certificate carries no SANs at all.
+- `192.168.2.20` is the shared Gateway's own LoadBalancer VIP
+  (`gateway/cilium-gateway-shared`, from the Cilium pool `192.168.2.20-30`), and its `https`
+  listener serves `*.home.keithjackson.dev` with `wildcard-home-tls`. From the operator's
+  workstation the name resolves there and serves the real Let's Encrypt
+  `CN=*.home.keithjackson.dev` with correct SANs. **So the certificate and the Gateway are
+  healthy.**
+- **The node asks a different resolver and gets a different answer.** Prod's
+  `node_baseline_dns_servers` is the LAN gateway, not Pi-hole — deliberately, as that file
+  explains. Public DNS for this zone is a CNAME to `wan.home.keithjackson.dev`: measured via
+  both `1.1.1.1` and `8.8.8.8`, `gitea.home.keithjackson.dev` →
+  `wan.home.keithjackson.dev.` → `136.51.115.252`, and the apex answers the same.
+- **That address serves a certificate with no names.** Measured from the workstation against
+  the WAN address with the right SNI:
+
+      subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D
+      issuer=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D
+      No extensions in certificate
+
+  No extensions means no `subjectAltName`, which is exactly "not valid for any names". The
+  `CN=%D` is the router's placeholder.
+
+**This repository had already measured the same thing, for Keycloak, and I did not need to
+re-derive it.** `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` says, above the pin:
+"Since platform/gateway/gateway.yaml set an external-dns `target`, every public name is a
+CNAME to wan.home.keithjackson.dev = the home WAN IP. The SOHO router does NOT hairpin: from
+the LAN, WAN:443 answers with the router's own self-signed placeholder cert (measured from
+k3s-prod-2/-3 2026-10-03: `curl` exit 60, ssl_verify_result=18)." Same cause, same symptom,
+one name later. The pin exists precisely for node processes that must reach the Gateway
+directly; containerd is one, and nobody had added it.
+
+**The parity the pin has to satisfy:** `192.168.2.20` equals the live
+`GATEWAY_VIP`, and `k3s_base_domain` equals the live `BASE_DOMAIN` — both read from the
+cluster, not from the comment that claims prod serves `prod.home.keithjackson.dev` (it does
+not; `flux-system/cluster-vars` reports `BASE_DOMAIN=home.keithjackson.dev` and
+`GATEWAY_VIP=192.168.2.20`, and Ansible's `k3s_base_domain` is `home.keithjackson.dev`).
+That comment is stale history.
+
+**The credential decision, and why the obvious alternative is not available.** Gitea's API
+accepts basic auth, PATs, SSH signatures and its own OAuth2 bearer tokens; as an OAuth2
+provider Gitea implements **only the Authorization Code Grant** (plus PKCE/OIDC), so there
+is no client-credentials grant, and nothing in Gitea validates a token issued by a
+third-party OIDC provider. A Keycloak service-account token therefore cannot authenticate to
+Gitea's API or to git-over-HTTP. Read from Gitea's own documentation
+(`docs.gitea.com/development/api-usage`, `/development/oauth2-provider`), not assumed.
+
+**Gate:** `./ci/gate.sh` in the pinned toolbox from a §0 worktree — see the branch's run; rc 3
+with `gitea-settings` the only stage that did not run.
+
+## What I did NOT verify
+
+- **That the pin fixes the pull.** It is IaC, and nothing has applied it: `/etc/hosts` on the
+  nodes is written by the `node_baseline` play's `hosts`-tagged task, which is an operator
+  action. I did confirm the mechanism is not inert — `roles/node_baseline/tasks/main.yml`
+  renders `node_baseline_hosts_pins` into a marked `blockinfile` with `backup: true`, and
+  `state: absent` when the list empties, so a removed entry really leaves the node. Until that runs
+  and an image exists, "the node now reaches the Gateway" is a design claim. The evidence
+  that it *will* is the Keycloak pin working for the same cause — which I also did not
+  re-measure.
+- **Why public DNS answers at all for an `internal` service.** `catalog/gitea.yaml` sets no
+  `exposure`, so it inherits `internal`, and the renderer contract says an internal service's
+  HTTPRoute carries `external-dns.alpha.kubernetes.io/controller: none` and publishes no
+  public record. Yet `gitea.home.keithjackson.dev` resolves publicly to the WAN CNAME. I did
+  not chase that: it may be a zone-level wildcard rather than a published record, and it did
+  not change the diagnosis. It is worth someone's attention, because "internal services
+  publish no public DNS" is an L0-checked claim and this looks like a case where the *name*
+  resolves regardless of what the renderer withheld.
+- **Registry authentication.** Unchanged and still open: an unauthenticated client gets
+  nothing from this registry, and no `imagePullSecret` exists anywhere in the tree. With the
+  credential model now decided (local Gitea accounts with PATs) the shape is a
+  `dockerconfigjson` for an account whose token carries `read:package`, referenced from the
+  Deployment and the committed worker Job template — and the worker Job template's shape is
+  pinned by admission, so whether the policy should also pin `imagePullSecrets` deserves a
+  deliberate look rather than a quiet addition. Not built: minting that token is an operator
+  action, and I did not want to guess at a second credential's scope.
+- **Anything about runtime.** The image still does not exist. No dispatcher has
+  authenticated, no issue has been triaged, no worker Job has been created, and the admission
+  policy — live, `Enforce`, six rules — has still never admitted or refused a real Job.

decisions/2026-10-06-vpn-ingress-ip-forward.md

@@ -0,0 +1,113 @@
+---
+date: 2026-10-06
+branch: fix/vpn-ingress-ip-forward
+title: The VPN tunnel never came up — a hostNetwork pod cannot write net.ipv4.ip_forward, and wg-quick treats a failed PostUp as fatal
+---
+
+## What changed
+
+The remote-access VPN reconciled for the first time with a real server key (#60 then #62) and
+**the tunnel did not come up**. One line in my own manifest was the whole cause.
+
+Measured in the `wireguard` container on prod, 2026-10-06:
+
+```
+[#] ip link add dev wg0 type wireguard
+[#] wg addconf wg0 /dev/fd/63
+[#] ip -4 address add 10.90.0.1/24 dev wg0
+[#] ip link set mtu 1420 up dev wg0
+[#] sysctl -q -w net.ipv4.ip_forward=1
+sysctl: permission denied on key "net.ipv4.ip_forward"
+[#] ip link delete dev wg0
+**** Tunnel /config/wg_confs/wg0.conf failed, will stop all others! ****
+```
+
+Everything up to that point worked. **The cause is not a missing capability.** With
+`hostNetwork: true` the runtime mounts `/proc/sys` **read-only** in the container — confirmed in the
+pod's own `/proc/mounts`: `proc /proc/sys proc ro,nosuid,nodev,noexec,relatime` — precisely because
+those sysctls are the *host's* globals and a container must not rewrite them. `CAP_NET_ADMIN` cannot
+lift a read-only mount, so this can never succeed from here. And `wg-quick` treats any failed
+`PostUp` as fatal, so it deleted the interface it had just finished building.
+
+**The line was never needed.** `node_baseline` writes `net.ipv4.ip_forward=1` to
+`/etc/sysctl.d/99-k3s-cilium.conf`, and reading the value from inside the pod returns `1`. I had
+written the header comment *correctly* — "ip_forward is ALREADY 1 on these nodes … so no pod sysctl
+is needed" — and then emitted the sysctl anyway, justified in its own comment as "re-asserted so
+this component does not silently depend on another role's sysctl file surviving". That defensive
+re-assertion is the entire outage. **The lesson is narrow and reusable: in a hostNetwork pod,
+verify host state, never write it.**
+
+Fix:
+
+- **`platform/vpn-ingress/daemonset.yaml`**: the `PostUp = sysctl …` line is gone, replaced by a
+  **read** in the init container — `grep -qx 1 /proc/sys/net/ipv4/ip_forward` — which fails closed
+  with a message naming the `node_baseline` role. Reading `/proc/sys` is permitted; only writing is
+  not. Fail-closed still matters: a tunnel that comes up but cannot forward would satisfy every
+  probe in this component and silently drop every client's traffic.
+- The header bullet that said "no pod sysctl is needed" now also says **do not attempt one**, with
+  the mechanism, so the next author does not re-add it for the same reason I did.
+- The `securityContext` comment claimed NET_ADMIN was for "create the wg0 link, set the host sysctl,
+  and write the netfilter rules". The middle clause was false; corrected.
+- The image's `chown: … Operation not permitted` startup run — plus its own "Permissions could not
+  be set … we will not provide support for it" — is now documented as **expected and harmless**, in
+  both the manifest and the runbook, so nobody chases it. Capabilities are dropped to `NET_ADMIN`
+  only, so its s6 init has no `CAP_CHOWN`; `/config` is a writable tmpfs, everything runs as root,
+  and `wg-quick` proceeds past it (as it did here — the failure was 30 lines later).
+- **`runbooks/vpn-remote-access.md`**: a "what a healthy start looks like" block with the real log
+  lines; `ip_forward is 0` added to the init-failure table; and the note that `0/1 Running` with
+  `Startup probe failed: Unable to access interface: No such device` is the signature of `wg-quick`
+  tearing the tunnel down — read the `[#]` line it died on.
+- **`platform/vpn-ingress/README.md`**: a table of what the first reconcile settled (below).
+
+No sensitive paths: `platform/vpn-ingress/**`, `runbooks/**` and `decisions/**` only. This merges on
+a green gate with no override.
+
+## What I verified
+
+Live prod, read-only, 2026-10-06. **Four questions this component's design rested on are now
+measured rather than assumed — three went the right way, one was the defect above.**
+
+- **The GitOps path is proven end to end.** `Kustomization vpn-ingress` `lastAppliedRevision:
+  main@sha1:cf7d5e4` equals the GitRepository artifact revision. The Secret carries
+  `kustomize.toolkit.fluxcd.io/name: vpn-ingress`, so Flux delivered it.
+- **SOPS decryption works, proven by digest rather than by eye.** The value in
+  `platform/vpn-ingress/wg-server-key.sops.yaml` on `main`, decrypted, is 44 bytes, is **not** the
+  placeholder, and **`wg pubkey` accepts it**. Its SHA-256 equals the SHA-256 of the live Secret's
+  decoded `data.privatekey` (`d7c8f8de6f252649…` both sides). No key material was printed at any
+  point — lengths, booleans and digests only.
+- **The `wg pubkey` guard fired correctly, and its `ENC[` branch was never needed.** During the
+  kubelet window before the updated Secret propagated, all three pods reported *"It is still the
+  committed REPLACE-ME PLACEHOLDER"* — the guard distinguishing a stale/unfilled value from
+  ciphertext, exactly as designed. It cleared on its own once the mount refreshed; **a transient
+  `REPLACE-ME` CrashLoop for a minute or two after the key PR merges is expected, not a fault.**
+- **The kernel module reaches the pod**: `ip link add dev wg0 type wireguard` succeeded and
+  `/proc/modules` lists `wireguard`. So the `node_baseline` addition works and `CAP_SYS_MODULE` is
+  correctly not required.
+- **`allowPrivilegeEscalation: false` does NOT break the image's s6 init** — the container reaches
+  `wg-quick` normally. The flag stays; the fear recorded in #55 is closed.
+- **The iptables backends MATCH, which was my single biggest documented doubt.** The image ships
+  `iptables v1.8.13 (nf_tables)` via `/usr/sbin/xtables-nft-multi`, and Debian 13 is nf_tables. The
+  nightmare case — rules accepted into a table the kernel never consults, so handshakes succeed and
+  every forwarded connection hangs — cannot occur on this pairing.
+- **`net.ipv4.ip_forward` is `1`** in the host netns the pod shares, and `/proc/sys` is `ro` there.
+  Both read directly from the pod.
+- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`,
+  `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; rc 3 is the two
+  environmental skips (`gitea-settings`, conformance L2/L3). **Not green. Reported as rc 3.**
+
+## What I did NOT verify
+
+- **That the tunnel now comes up.** This fix has not reconciled. The next reconcile is the test, and
+  its success criterion is specific: pods `1/1 Running`, and `wg show` reporting `interface: wg0`
+  with `listening port: 51820`.
+- **That the MASQUERADE and FORWARD rules actually apply.** They are the next statements after the
+  one that failed, so **they have never executed**. The backends matching (above) removes the worst
+  failure mode but proves nothing about the rules themselves. First check after the pods go Ready:
+  `iptables -t nat -S POSTROUTING | grep 10.90.0.0/24` on a node.
+- **Anything about client traffic.** Still zero peers enrolled, still no WAN port-forward, still no
+  off-LAN vantage point. The VPN has carried no packet.
+- **Whether the transient `REPLACE-ME` window has an upper bound.** It cleared within ~2 minutes
+  here; I did not determine the kubelet refresh behaviour, so a longer stall after a key change is
+  uncharacterised.
+- **The `sops set` hand-off command**, still. The operator filled the key in by their own route
+  (#62), so the exact quoting I documented inside `dev.sh run bash -c` remains unexercised.

decisions/2026-10-07-bulk-nfs-truenas.md

@@ -0,0 +1,85 @@
+---
+date: 2026-10-07
+branch: fix/bulk-nfs-truenas
+title: Repoint bulk NFS at the TrueNAS (192.168.2.30); the PVs must be recreated, not reconciled
+---
+
+## What changed
+
+Both ZFS pools were physically moved off the Proxmox host onto a standalone TrueNAS Community
+Edition at **192.168.2.30**, which sits on prod's own `192.168.2.0/24`. `BULK_NFS_SERVER` moves from
+`192.168.2.102` (the pfSense DNAT prod used to mount through) to `192.168.2.30`.
+
+- `clusters/prod/kustomization.yaml` — `BULK_NFS_SERVER` → `192.168.2.30`. The comment block that
+  justified the DNAT, and the "NOT YET DECIDED / THIS PLACEHOLDER IS LOAD-BEARING" banner, are
+  replaced with the measured export configuration and the recreate requirement below.
+- `clusters/base/cluster-vars.yaml` — base default `192.168.1.9` → `192.168.2.30`. That host no
+  longer exists.
+- `clusters/dev/kustomization.yaml` — same; marked explicitly **inert**, since dev is decommissioned
+  and its nodes are gone. It carries the real address only so nothing in the tree renders a dead one.
+- `docs/BULK-NFS-STORAGE.md` — §1 and §5 rewritten for TrueNAS (exports, service settings, the ZFS
+  `sharenfs` properties that had to be cleared, and the ACL hazard); §7 rewritten now that the
+  placeholder is gone; §9 retargeted from dev to prod; **new §10** documents the PV
+  delete-and-recreate. §2/§3/§4/§6 are unchanged by the move.
+- `catalog/qbittorrent.yaml`, `platform/monitoring/node-pressure-dashboard.yaml` — two comment/prose
+  references to `192.168.2.102` as the live export address.
+
+**The headline operational fact: repointing this knob is NOT a plain reconcile.** A bound PV's NFS
+source is immutable, so Flux's apply fails rather than updating and the pods keep their old, dead
+mount. The five PVs must be deleted and recreated. That sequence is §10 and is an operator action;
+it is **not** performed by this PR.
+
+## What I verified
+
+**On the NAS (192.168.2.30), measured 2026-10-07 by the operator at my request:**
+
+- `zfs list` — `Storage` → `/mnt/Storage`, `Storage/plex` → `/mnt/Storage/plex`,
+  `Storage/nextcloud` → `/mnt/Storage/nextcloud`. The PV paths resolve unchanged.
+- Ownership intact: `/mnt/Storage/plex`, `tv/`, `downloads/`, `movies/`,
+  `downloads/complete/`, `nextcloud/nextcloud/` all `drwxr-xr-x 1000:3000`. `movies/` 1171 entries
+  and `tv/` 107, against 1165 / 107 measured on Proxmox.
+- `acltype=nfsv4` (SCALE's built-in default, `SOURCE: default`), `aclinherit=passthrough`, and no
+  non-trivial ACLs — mode bits still govern.
+- Three stale `sharenfs` dataset properties (`plex`, `nextcloud`, `docker-volumes`) carried over from
+  Proxmox; cleared with `zfs inherit`, re-checked empty.
+- `exportfs -v`: both shares to `192.168.2.10/.11/.12` with `sec=sys,secure,root_squash,no_all_squash`,
+  `rw` on plex and `ro` on nextcloud.
+- `/proc/fs/nfsd/versions` → `+3 +4 +4.1 +4.2`, so the PVs' `nfsvers=4.1` negotiates.
+- **A defect was found and fixed during setup:** the plex share listed `192.168.2.13` — an address no
+  node holds — and so omitted `192.168.2.12` (k3s-prod-3). Corrected and re-verified.
+
+**From the cluster:**
+
+- Node InternalIPs are `192.168.2.10/.11/.12` (`kubectl get nodes`), confirming `.13` was wrong.
+- All five NFS PVs currently name `192.168.2.102`.
+- `192.168.2.30:2049` and `:111` accept connections; `192.168.2.102:2049` does not.
+- **`spec.nfs` immutability, by server-side dry-run** — this is the finding that shapes the change:
+  `spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation`.
+- The pods report `Ready` while their mounts are dead: a `stat /media` answers from attribute cache,
+  but a `readdir` blocks indefinitely (the PVs mount `hard`). Hence "scale to zero first" in §10.
+
+**Gate:** see the PR; run in the pinned toolbox from this worktree.
+
+## What I did NOT verify
+
+- **Nothing has mounted 192.168.2.30.** No pod has touched the new server — the PVs still point at
+  the old address and cannot be updated in place. Every §9 probe (numeric-id passthrough, the uid-1000
+  write, the hardlink/EXDEV check, Plex's read-only posture) is **outstanding**, not passing. The
+  equivalents passed against the old host on 2026-09-12; that is not evidence about this one.
+- **The §10 sequence is reasoned, not executed.** Retain + the renderer's pre-set `claimRef` are read
+  from the manifests and from `catalog/render.py`; I have not watched a PVC rebind after a PV delete
+  in this cluster.
+- **`root_squash` is a behaviour change from the Proxmox export**, which carried `no_root_squash`. The
+  argument that nothing needs root on the tree (LSIO drops to 1000; the images chown only their
+  Longhorn `/config`; both subPath directories already exist at 0755) is reasoned from the images and
+  the measured permissions. It has not been exercised. If a mount or subPath resolution fails with
+  EACCES, Maproot `root:root` reproduces the old export and is the first thing to try.
+- **Whether NFSv3 can be disabled.** Enabled Protocols is `NFSv3, NFSv4`; the PVs only need v4.1, so
+  narrowing would close the rpcbind/mountd/statd surface. Not done — it is a change to make after the
+  cutover is proven, not during it.
+- **Whether pfSense / the legacy stack is actually retired.** I measured only that
+  `192.168.2.102:2049` stopped answering. The other legacy consumers of these datasets
+  (`192.168.1.6`, `10.123.1.10`, `192.168.1.18`) were dropped from the new exports; if any is still
+  live, it has lost access. Not checked.
+- **L2/L3 conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline subset
+  only. Nothing in the gate exercises an NFS mount.

decisions/2026-10-17-automated-upgrades.md

@@ -0,0 +1,300 @@
+---
+date: 2026-10-17
+branch: feat/automated-upgrades
+title: Automated upgrades — kured + security-only unattended-upgrades, and Renovate finally gets a runner
+---
+
+## What changed
+
+Two halves of one process. Neither is useful alone and they must land together.
+
+### (a) OS upgrades and reboots
+
+**`platform/kured/`** — the Kubernetes Reboot Daemon, chart `6.1.0` / app `1.23.0`, in its own Flux
+Kustomization (`clusters/base/kured.yaml`, `dependsOn: [platform]`, `wait: true`), registered in
+`clusters/base/kustomization.yaml`. Not a line in `platform/kustomization.yaml`: that tier is
+`wait: true` and gates `platform-config` → every apps wave, so a wedged reboot daemon placed there
+could stall every application in the cluster. A component added to make maintenance safer must not
+be able to cause an outage.
+
+**`infra/ansible/roles/unattended_upgrades/`** + `playbooks/unattended-upgrades.yml`, enabled for
+prod in `inventory/group_vars/k3s_prod/vars.yml`. A **new** role rather than an extension of
+`node_baseline` — different lifecycle, and other open branches touch that role.
+
+**The design, and how it answers this repo's two standing objections to reboot automation.**
+`roles/node_baseline/tasks/main.yml:61-67` says "IT DOES NOT REBOOT" (k3s-prod-2 hung on a warm
+reboot 2026-09-03; physical power button required), and `docs/plans/cluster-upgrade-plan.md:291-299`
+rejected system-upgrade-controller for "automating away the human gate between nodes". Both still
+stand. **The human gate is the sentinel.** unattended-upgrades is security-only with the kernel
+blacklisted, so it essentially never produces `/var/run/reboot-required`; that file appears when an
+operator deliberately runs `node-baseline.yml --tags kernel`. kured then performs only the
+*mechanical* half the operator would otherwise type by hand — including the ordering rule
+`node_baseline`'s own output has to shout ("NEVER node 2 first"), which the operator keeps by
+running the kernel play with `--limit <host>` one node at a time. `cluster-upgrade-plan.md:329-330`
+names the condition under which its own rejection reverses (fast disks), and prod is that cluster.
+
+**Quorum: concurrent reboots are structurally impossible.** All three nodes are control-plane+etcd
+(quorum 2/3). `concurrency: 1` plus kured's single cluster-wide lock annotation on its own DaemonSet
+means a node reboots only while holding that lock, enforced by API-server optimistic concurrency
+rather than by timing. **`lockTtl: 0`** (expiry disabled) is the deliberate choice: a TTL exists so
+reboots continue after the lock holder dies, which on 3 etcd nodes is exactly the catastrophe — a
+hung node's lock expiring would let a second node take the cluster to 1/3. So a hang stops the cycle
+forever and pages a human. `lockReleaseDelay: 30m` holds the lock after a node returns.
+
+**The Longhorn gate.** Measured: 25 volumes, all `numberOfReplicas: 3`, one replica per node, with
+`replica-soft-anti-affinity: false` and `disable-scheduling-on-cordoned-node: true`. So every drain
+takes **all 25 volumes to degraded 2/3**, and recovery is serialized —
+`replica-replenishment-wait-interval: 600`, `concurrent-replica-rebuild-per-node-limit: 1`. Reboot
+the next node too early and volumes reach 1/3, where `node-drain-policy:
+block-if-contains-last-replica` can block a drain forever. The inter-node gate is therefore **volume
+robustness, not pod readiness**, and it is doubled: `lockReleaseDelay: 30m` (unconditional, covers
+the 600s wait) plus kured's Prometheus alert gate (semantic, covers a rebuild longer than 30m).
+
+**`alertFilterMatchOnly: true` is load-bearing.** The flag inverts the regexp's meaning: `false`
+(chart default) makes it an *ignore* list and kured blocks on **any** firing alert. Measured
+2026-10-17, five alerts were firing with no end in sight (`CNPGPrimariesCoLocated`,
+`KubeDeploymentReplicasMismatch`, `KubeDeploymentRolloutStuck`, `KubePodNotReady`,
+`OAuth2ProxyGateUnobserved` — the last three all tracing to `pm-agent`, ImagePullBackOff on a
+placeholder digest for 12 days). Under the default, kured would have been blocked on day one and
+never rebooted anything, indistinguishable from "no reboots needed". The allow-list is anchored:
+`^(LonghornVolumeDegraded|LonghornVolumeFaulted|KubeNodeNotReady|NodeExporterDown)$`.
+`NodeEtcdRequestsSlow` and the `NodeIOPressure*` family are **deliberately excluded** despite being
+exactly what you'd want to pause for — prod-1 has a documented etcd WAL I/O starvation history and
+including them risks a permanent wedge that looks like safety.
+
+**The kernel floor (the highest-risk item), handled with two independent exclusions.** Measured: all
+three nodes run `7.1.8+deb13-amd64` (`7.1.8-1~bpo13+1`, the backports metapackage), far above the
+6.13 memattr floor — **but `linux-image-6.12.107` and `linux-image-6.12.94` are also still
+installed, below the floor, bootable, and `apt-mark showhold` is empty on every node.** So: (1) the
+allowed-origins list is `label=Debian-Security` only, and the running kernel comes from
+`o=Debian Backports` (pin 100, the `NotAutomatic`+`ButAutomaticUpgrades` signature), so u-u cannot
+*see* a kernel upgrade; (2) `^linux-` is blacklisted anyway, so widening origins later — the most
+likely future edit — cannot re-arm the hazard. The role **asserts both against `apt-config dump`**,
+apt's own parse of the whole directory, rather than trusting the file it just wrote. Also blacklisted:
+`intel-microcode`, `amd64-microcode`, `grub-*`. `Remove-Unused-Kernel-Packages` is **false** against
+the Debian default, because those older kernels are the documented rollback path. None of this stops
+the kernel moving — the operator's deliberate Ansible run still does that, unchanged.
+
+**The silent-no-op trap this closes.** Measured: `update-notifier-common`, `needrestart` and
+`unattended-upgrades` were **absent on all three nodes**, so *nothing could create
+`/var/run/reboot-required`*. kured deployed alone would have been permanently idle while 1/1 Running
+and reporting `kured_reboot_required=0` — green forever. The role installs the sentinel writer and
+**asserts a hook exists** by discovery (not a hard-coded path), failing loudly if not.
+`needrestart` runs in auto mode with `k3s.service` blacklisted — auto-restart is what makes a
+security-only policy effective, but restarting the control plane unattended is worse than a delayed
+library fix. (k3s is not an apt package at all here: no `k3s`, no `containerd` in dpkg.)
+
+**Alerts + runbook.** `platform/kured/alerts.yaml` — three rules over kured's *only* metric
+(`kured_reboot_required`, one GaugeVec, read from `cmd/kured/main.go:92-96`) plus `kube_node_info`:
+`KuredNotReporting` (`absent()` — the pipeline is dead and the other two are inert),
+`KuredNodeCoverageGap` (`count(kured) < count(nodes)` — a node silently uncovered, which `absent()`
+cannot see), `KuredRebootPending` (`for: 36h` — a reboot was offered and blocked). Companion
+promtool suite `alerts.promtool-test.yaml` pins all three both ways, including that the rule
+**stops** when a reboot succeeds. `runbooks/node-reboot-automation.md` registered in
+`runbooks/kustomization.yaml`.
+
+**The window is DAILY 03:00-06:00, and that choice is driven by the alert.** A weekly window means a
+pending reboot legitimately waits 6 days, forcing `for: 8d`+ — longer than this cluster's 7d
+Prometheus retention and reset by any restart past the 1h for-outage-tolerance. Daily caps the
+legitimate wait under 24h and makes `for: 36h` both robust and meaningful.
+
+### (b) Image / dependency automation
+
+**Renovate was aspirational. It now has a runner.** `.gitea/workflows/renovate.yaml` — scheduled
+daily 03:15 UTC, `workflow_dispatch` with `dry_run` defaulting to **true** for a safe first run.
+Preflight **fails hard (exit 1)** if `RENOVATE_TOKEN` is missing rather than skipping, because a
+skipped Gitea job reports `success` and that is precisely how `renovate.json` stayed inert.
+
+**`renovate.json`: Renovate now owns both halves, and Flux image-automation is formally closed.**
+ADR-0009 gave container image tags to Flux image-automation; that cannot be delivered on this
+repo's terms. (1) `ImageUpdateAutomation` **pushes commits, it does not open PRs**, so it bypasses
+the CLAUDE.md §1 gate; pointed at a side branch instead, nothing opens the PR. (2) It cannot work
+here anyway: `main` is protected `user_can_push: false` and the `flux-system` GitRepository has **no
+`spec.secretRef`** — the source is anonymous and read-only. (3) Enabling it means an in-cluster git
+**write** credential for the repo that defines the cluster. So `docker`/`kubernetes` — disabled
+*only* to avoid fighting that controller — are enabled, with `kubernetes.managerFilePatterns`
+covering `platform/`, `platform-config/`, `apps/`, **`catalog/`** (apps are generated from it; a bump
+only to the rendered output is reverted by the next render and fails `ci/render-golden.sh`),
+`runbooks/serving/` and `infra/agent/`. Added a no-automerge `blast-radius` rule for `kured`,
+`longhorn`, `cilium`, `cloudnative-pg`, `kube-prometheus-stack`, because the gate is static and does
+not boot a node or form a quorum. `platform/kyverno/tests/**` added to `ignorePaths`.
+
+### Sensitive paths — this PR needs a non-author approving review
+
+It touches **`clusters/**`**, **`conformance/**`** and **`.gitea/**`**, all in
+`infra/gitea/approval-policy.json`. It will **not** merge on a green gate alone. Per CLAUDE.md §1 I
+have **not** posted `/approve-override` and must not — that is the self-approval the stage exists to
+prevent.
+
+**The `conformance/` edit, justified.** `+34/-0` in
+`alert-metrics-have-scrape-targets/check.sh`: one new `SCRAPE_OWNERS` entry claiming `kured_`. It
+**weakens nothing** — no existing owner's evaluation changes — and it *tightens* the check for my
+own component, adding two `require` patterns (`service.create: true`, `metrics.create: true`; the
+chart defaults **both** to false) plus a `companion` requirement that `networkpolicy.yaml` stay in
+the kustomization. It had to be touched: assertion (3) fails any alert over an unclaimed metric, and
+the escape hatch `KNOWN_UNWIRED_METRICS` is **shrink-only**, so declaring `kured_*` as debt was not
+permitted and would have been wrong anyway. The alternative was shipping reboot automation with no
+alerting, which for a component whose every failure mode is silent is the worse choice. The entry
+carries a dated comment saying all of this.
+
+## What I verified
+
+Measured against the live cluster and nodes, read-only, on 2026-10-17.
+
+- **Flux image-automation controllers ARE installed and idle.** `-n flux-system get deploy` shows
+  `image-automation-controller` and `image-reflector-controller` both `1/1`, age **43d**; all three
+  `image.toolkit.fluxcd.io` CRDs exist. And **zero** `ImageRepository`/`ImagePolicy`/
+  `ImageUpdateAutomation` objects — "No resources found" for each. The controllers have run for 43
+  days with nothing to do. `platform/kustomization.yaml:52` has the component commented out.
+- **Renovate is aspirational, confirmed three ways**: no workflow in `.gitea/` or `ci/` (grep),
+  no Renovate CronJob/Deployment/Job in any namespace, and `renovate.json` itself disabled the
+  `docker`+`kubernetes` managers — so every plain `image:` tag was maintained by **nothing**, while
+  **seven files** claim "Renovate maintains this pin".
+- **`flux-system` GitRepository has no `secretRef`** (empty), URL is the in-cluster Gitea, and its
+  artifact revision was `main@sha1:1a0b208…`, matching my base.
+- **Nodes**: 3× Debian 13 trixie, all `control-plane,etcd`, k3s `v1.36.4+k3s1`, kernel
+  `7.1.8+deb13-amd64`. `apt-mark showhold` **empty**; `/etc/apt/preferences.d/` **empty**;
+  `/var/run/reboot-required` absent; `unattended-upgrades`/`update-notifier-common`/`needrestart`
+  all **absent**; `linux-image-6.12.107` and `6.12.94` installed alongside the backports kernel;
+  no `k3s`/`containerd` apt package; `apt-cache policy` origins and pins as quoted above.
+- **No node taints** on any of the three (so no tolerations needed), and `system-node-critical`
+  exists.
+- **Longhorn**: 1.12.1; `node-drain-policy: block-if-contains-last-replica`;
+  `replica-soft-anti-affinity: false`; `disable-scheduling-on-cordoned-node: true`;
+  `replica-replenishment-wait-interval: 600`; `concurrent-replica-rebuild-per-node-limit: 1`;
+  25 volumes all 3-replica/healthy; 25 replicas per node.
+- **6 PDBs with 0 allowed disruptions**: 3× `longhorn-system/instance-manager-*` (structurally 0 —
+  single-node selector, `minAvailable: 1`) and 3× CNPG `*-primary`. All three Postgres primaries
+  were on `k3s-prod-1`.
+- **kured drain flags do not exist.** Read `cmd/kured/main.go:438-449`: `Force`,
+  `DeleteEmptyDirData` and `IgnoreAllDaemonSets` are **hardcoded true**. Passing
+  `--ignore-daemonsets`/`--delete-emptydir-data`/`--force` as extraArgs would CrashLoop the
+  DaemonSet. All three are genuinely needed (12 DaemonSets, 135 emptyDir pods measured).
+- **kured exposes exactly one metric** — the only `MustRegister` in the project.
+- **Prometheus selectors are all `{}`** (`serviceMonitorSelector`, `serviceMonitorNamespaceSelector`,
+  `ruleSelector`, `ruleNamespaceSelector`), so a `PrometheusRule`/`ServiceMonitor` in a new namespace
+  is picked up with no label plumbing.
+- **`count(kube_node_info) == 3`** and `count(longhorn_volume_robustness) == 100`, queried through a
+  read-only port-forward. The 5 chronically-firing alerts were read from `/api/v1/alerts`.
+- **`LonghornVolumeDegraded`** is `longhorn_volume_robustness{state="degraded"} == 1`, `for: 10m` —
+  shorter than Longhorn's own 600s replenishment wait, so degradation reliably outlasts the latch.
+- **kured chart 6.1.0 → appVersion 1.23.0** is the newest of 80 entries in the chart index.
+- **Kyverno/PSA will not block it**: every policy is scoped
+  `app-namespace: "true"`; the namespace sets it explicitly `"false"`, and
+  `restrict-host-namespaces.yaml`'s own header states that carve-out. No cluster-wide PSA baseline.
+- **`renovate.json` parses** as valid JSON after my edits (`json.load`).
+- **kured is not installed** and the string `kured` appears nowhere in the repo — greenfield.
+
+### CORRECTION 2026-10-18 — the first real run FAILED, and the sentinel design was wrong
+
+`ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod` failed on all three
+nodes at the package task:
+
+```
+fatal: [k3s-prod-1]: FAILED! => "No package matching 'update-notifier-common' is available"
+```
+
+**`update-notifier-common` is an UBUNTU package and does not exist in Debian.** Measured on trixie:
+`apt-cache policy update-notifier-common` → `Candidate: (none)`. The original entry asserted this
+package "owns the hook that creates /var/run/reboot-required" — true on Ubuntu, false here, and the
+role could never have worked. This was the exact risk the first version of this entry flagged ("the
+`find` patterns are a guess at which package ships what") landing in a worse place than predicted:
+not a wrong glob, a nonexistent package.
+
+Three further measurements settled what to do instead:
+
+* `/usr/share/update-notifier/notify-reboot-required` is **absent**.
+* **No `linux-image-*` postinst references `notify-reboot-required`** — invoking it is an Ubuntu
+  patch, so merely placing a script at that path would never be called.
+* No hook under `/etc/apt/apt.conf.d` or `/etc/kernel/` mentions `reboot-required`.
+
+So Debian trixie has **no stock mechanism at all**, and the role now ships the writer itself:
+`/usr/local/sbin/k3s-notify-reboot-required` (compares the running kernel against the newest
+installed via `linux-version`, from `linux-base`, which every `linux-image` depends on — measured
+`linux-base 4.12.1` present) plus a `DPkg::Post-Invoke` hook at
+`/etc/apt/apt.conf.d/99-k3s-reboot-required`. `/run` is a tmpfs (measured), so the sentinel
+self-clears on boot and needs no cleanup task.
+
+The verify assert changed shape too: it was a `find` glob over `/etc/kernel/postinst.d` and
+`/etc/apt/apt.conf.d` for `*update-notifier*` / `*unattended-upgrades*`. That glob was **weaker
+than it looked** — it would have matched Debian's own `unattended-upgrades` apt conffile and
+reported "sentinel writer present" while no writer existed. It now stats the two exact paths this
+role owns and requires the script to be executable.
+
+**Newly verified (this correction):** the sentinel script was executed on prod-1 as the
+unprivileged `k3s` user under `sh -x`. It resolved `running=7.1.8+deb13-amd64`,
+`newest=7.1.8+deb13-amd64`, took the equal branch, exited 0 and wrote nothing — correct, since the
+running kernel *is* the newest. `sh -n` and `bash -n` clean.
+
+**Still NOT verified after this correction:** the *writing* half. Because running == newest on all
+three nodes, the `touch` path has never executed, so I have not seen the sentinel actually created,
+nor seen kured react to it. The honest test is the operator's next kernel upgrade. Also unverified:
+whether a `DPkg::Post-Invoke` hook fires in the `unattended-upgrades` code path specifically (it
+runs dpkg, so it should, but that is inference) — and whether `linux-version sort --reverse` orders
+a mixed backports/stable kernel set the way I assume when a 6.12 and a 7.1 are both installed. It
+produced the right answer with the current set; a future set could differ.
+
+## What I did NOT verify
+
+The honest list. Most of this needs an actual upgrade cycle, which I did not and could not run.
+
+- **I never ran an OS upgrade, never rebooted a node, and never watched kured take a lock.** No part
+  of the runtime behaviour of either half has been observed. `sudo` needs a password I do not have,
+  so I could not run `apt`, `unattended-upgrade --dry-run`, or install anything. **Everything about
+  (a) working is inference from configuration and measured cluster state, not observation.**
+- **The Ansible role has never been executed.** Not syntax-checked by `ansible-playbook --syntax-check`
+  either (no inventory access from the gate). Its asserts, its `apt-config dump` parsing, and the
+  sentinel-hook discovery are all unproven. The `find` patterns for the sentinel hook are a *guess at
+  which package ships what* — I could not read `update-notifier-common`'s file list (packages.debian.org
+  returned no such package for that suite/arch), which is exactly why the role discovers rather than
+  hard-codes. **If no hook matches, the role fails loudly — but it may fail on a correct node.**
+- **Whether CNPG actually switches the primary in response to a drain-induced eviction.** All three
+  `*-primary` PDBs are structurally 0-disruption and `.spec.nodeMaintenanceWindow` is unset on all
+  three clusters. If the operator does not switch over, draining `k3s-prod-1` blocks on three PDBs at
+  once and hits `drainTimeout`. **This is the most likely way the first real reboot fails.** It
+  should be tested with one manual, watched `kubectl drain` before trusting the timer.
+- **Whether `drainTimeout: 15m` is long enough.** Unmeasured. The `unifi` VMI has
+  `evictionStrategy: LiveMigrate` with `migrationMethod: BlockMigration`, i.e. a full root-disk copy
+  over the network on every `k3s-prod-1` drain. I do not know how long that takes.
+- **Whether the alert gate actually blocks as intended.** I never saw kured query Prometheus. The
+  assumption that it treats a query error as "do not proceed" is from its documented behaviour, not
+  observed. The NetworkPolicy allowing kured→Prometheus:9090 and Prometheus→kured:8080 is
+  **unproven**; if either is wrong the symptom is silence, which is why `KuredNotReporting` exists —
+  but that alert is itself unproven against a live scrape.
+- **The alert expressions have not run against real `kured_*` data**, because the metric does not
+  exist in this cluster yet. They pass promtool unit tests against synthetic series whose shape I
+  read from kured's source. `absent()` and the `count() < count()` comparison are the two most
+  likely to behave differently against real label sets.
+- **`enabledManagers: ["regex"]` may be the wrong name.** Modern Renovate expects `custom.regex` for
+  `customManagers`. I left the pre-existing value alone rather than guess, but **the inline
+  `# renovate:` markers may simply not run.** Unverifiable without executing Renovate. Two markers
+  are additionally known-broken independent of this: `external-snapshotter`'s sits two comment lines
+  above its `newTag`, and Keycloak's `kustomization.yaml` has no marker at all despite a comment
+  claiming Renovate maintains it.
+- **Renovate has still never run.** I added the runner; `RENOVATE_TOKEN` does not exist yet, so the
+  first scheduled run **will fail** (by design). Nothing about the config is proven correct until a
+  `dry_run: true` dispatch is read by a human. Expect it to want to open ~30 PRs.
+- **No alert for "automatic patching silently stopped".** u-u's success lives in
+  `/var/log/unattended-upgrades/` on each node and nothing reads it. Likewise **no alert for the
+  Renovate schedule ceasing to fire**. Both are real gaps I did not close.
+- **`coredns` is 1 replica and I did not fix it.** It is k3s-managed, out of this PR's scope, and it
+  means every `k3s-prod-1` reboot causes a cluster-wide in-cluster DNS gap. Recommended follow-up.
+- **kured cannot honour "NEVER node 2 first".** It has no ordering control. The mitigation — run the
+  kernel play with `--limit` one host at a time — is a *procedure*, documented in the runbook, not an
+  enforced property. If the operator flags all three nodes at once, kured picks the order.
+- **kured cannot press a power button.** The 2026-09-03 warm-reboot hang is unmitigated. Bounded, not
+  prevented: quorum holds at 2/3, the lock is held forever so nothing cascades, and
+  `KuredRebootPending` eventually fires.
+- **Upstream version claims are a snapshot, partly from search results**, not all from authoritative
+  release APIs. I could not determine latest for versitygw, crafty, nginx-unprivileged, act_runner,
+  `virt-template-*`, the linuxserver wireguard date-tag, or the exact latest PostgreSQL 16 patch
+  (sources conflicted 16.10 vs 16.15). **The gitea `1.24.3 → v28.1.0` gap is almost certainly an
+  upstream versioning-scheme change, not 26 majors — do not act on that framing.**
+- **I did not verify Renovate's `kustomize` manager still processes `images:`/`newTag`** now that
+  `docker` is enabled, which decides whether `external-snapshotter` and `intel-gpu-plugin` are
+  managed at all.
+- **I did not raise any HelmRelease version ceiling**, so the 9 minor-locking ranges still cap
+  Renovate at patches. Bumping `kube-prometheus-stack` in particular will fail the gate until
+  `ci/versions.env`'s exact promtool/Prometheus `3.14.0` lockstep moves with it.
+- **L2/L3 conformance did not run** (`vars.CLUSTER_AVAILABLE` unset), so nothing here is proven
+  against a cluster by CI either.

decisions/2026-10-17-oauth2-proxy-oidc-startup-race.md

@@ -0,0 +1,99 @@
+---
+date: 2026-10-17
+branch: fix/oauth2-proxy-oidc-startup-race
+title: "oauth2-proxy dies on a cold start because OIDC discovery runs once, before Keycloak serves"
+---
+
+## What changed
+
+Exposed by the 2026-10-17 whole-cluster shutdown for the UPS move. oauth2-proxy performs OIDC
+discovery **once, at startup**, and `os.Exit(1)`s if it fails. On a cold boot Keycloak is not
+serving yet, so the shared Gateway answers 503 and the proxy dies:
+
+```
+[provider.go:55] Performing OIDC Discovery...
+[main.go:73] ERROR: Failed to initialise OAuth2 Proxy: error initialising provider: ...
+             failed to discover OIDC configuration: unexpected status "503": no healthy upstream
+```
+
+It then sits in **exponential CrashLoopBackOff long after Keycloak is healthy** — one of the two
+replicas was still down minutes later and only a manual `kubectl delete pod` cleared it. There is no
+retry flag; the process does not survive a cold start. A readinessProbe cannot help (the container
+is already dead), and CrashLoopBackOff backoff is not tunable per-pod.
+
+`platform/oauth2-proxy/helmrelease.yaml` gains an `extraInitContainers` gate (`wait-for-idp`) that
+polls Keycloak's discovery doc until it answers, then lets the proxy start. A pod in `Init` does not
+accrue CrashLoopBackOff backoff and retries promptly, which is the point.
+
+**Bounded, and fails open.** After ~5 minutes it gives up and starts the proxy anyway, so the worst
+case is exactly today's behaviour and never something new. An unbounded wait would trade a CrashLoop
+for a pod wedged in `Init` forever, which is not an improvement.
+
+## Why this shape and not the alternatives
+
+* **`extraInitContainers` is the chart's own documented hook for precisely this.** Its commented
+  example in `values.yaml` is literally `- name: wait-for-idp`. The key was **confirmed against the
+  pinned chart** (`helm show values oauth2-proxy/oauth2-proxy --version 10.7.1` → `extraInitContainers: []`
+  at line 428), not assumed — a wrong Helm values key is silently ignored, which is the
+  "reviewed, merged, apparently done, still broken" shape this repo keeps getting bitten by.
+* **No new base image.** The chart already runs an `alpine` init container for its built-in
+  `initContainers.waitForRedis` gate, so alpine is a base this chart uses regardless. Pinned to
+  `3.22` (verified to exist upstream) rather than the chart's own `latest`.
+* **Flux `postRenderers` rejected:** it would work independently of the chart schema, but nothing in
+  this tree uses postRenderers, so it would introduce a mechanism with no precedent for a problem
+  the chart already has a hook for.
+* **`--skip-oidc-discovery` rejected:** it removes the startup fetch entirely, which is arguably
+  more elegant, but it means hardcoding four Keycloak endpoints and I could not verify offline that
+  oauth2-proxy then fetches JWKS lazily rather than at startup. oauth2-proxy gates SSO for PiHole,
+  the \*arr apps, qBittorrent, Homepage, Prometheus, Alertmanager and Hubble — a wrong guess locks
+  the operator out of all of them, so the reversible, additive option wins.
+
+The `$$`-free convention matters here: shell variables are written bare (`$i`, `$url`, `$(( ))`)
+with **no braced form anywhere**, because `platform` declares `postBuild.substituteFrom cluster-vars`
+and envsubst consumes braced expansions — a braced shell variable renders to the empty string and
+silently breaks the loop. `${BASE_DOMAIN}` is braced on purpose. Same hazard and same convention as
+the gitea and nextcloud OIDC bootstrap hooks, and `ci: flux-postbuild-substitution` enforces it.
+
+## What I verified
+
+* **The failure is measured, not theorised.** The log above is from the live pod
+  (`oauth2-proxy-5d6c74876b-bzb2p`, `lastState.terminated: {reason: Error, exitCode: 1}`), captured
+  with `--previous` after the cold boot. The surviving replica's log shows the same startup
+  succeeding once Keycloak was up, which is what identifies this as a race rather than a config
+  fault.
+* **Deleting the pod is a sufficient workaround** — done on prod 2026-10-17, the replacement came up
+  clean and both replicas have been healthy since.
+* **The chart key is real**, at the pinned version range: `helm search repo` resolves
+  `>=10.0.0 <11.0.0` to chart `10.7.1` (app `7.15.5`), whose values carry `extraInitContainers: []`,
+  `extraContainers: []` and the built-in `initContainers.waitForRedis` (which uses `alpine`).
+* **`docker.io/library/alpine:3.22` exists** — confirmed against the registry tag list rather than
+  assumed from memory.
+* The edited HelmRelease parses as YAML.
+
+## What I did NOT verify
+
+* **UNAPPLIED. The init container has never run.** I did not reconcile the HelmRelease, so I have not
+  seen `wait-for-idp` execute, exit 0, or gate the proxy. Applying it rolls both oauth2-proxy
+  replicas, which is a mutating action (CLAUDE.md §5) — and it rolls the component that gates SSO for
+  seven apps, so **do it when you can watch it**, not unattended.
+* **I did not re-create the cold-start race to prove the fix.** That needs Keycloak down while
+  oauth2-proxy starts; the honest test is the next whole-cluster boot. Until then this is reasoned
+  from the measured failure, not demonstrated against it.
+* **busybox `wget`'s TLS behaviour here is assumed, not tested.** The claim is that alpine's
+  `ca-certificates-bundle` lets it validate the Gateway's LE-prod certificate, so a genuine TLS
+  failure exits non-zero rather than passing falsely. If that is wrong in either direction the gate
+  degrades safely — a never-succeeding probe just burns the 5-minute budget and then starts the proxy
+  anyway — but the fast path would be silently dead and this would be doing nothing. **Check the
+  init container's log on first apply**: it should print "Keycloak discovery reachable" within
+  seconds on a warm cluster, not the give-up message.
+* **Whether a pod can reach the Gateway hostname at all from this namespace is inherited, not
+  re-proven.** The main container does exactly this fetch today and succeeds, so the path is known
+  good — but note the standing trap that an arbitrary pod gets 403 from every Gateway hostname
+  (`decisions/2026-10-04-plex-advertise-plexdirect.md`); oauth2-proxy is exempt only because
+  Keycloak's ingress policy admits it. An init container in the same pod inherits that identity, so
+  this should hold — unverified.
+* **60 attempts x 5s is not tuned.** It is "comfortably longer than Keycloak's observed cold start",
+  not a measured bound.
+* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline
+  subset only.
+* I did not re-measure branch protection or runner state (CLAUDE.md §2b).

docs/BULK-NFS-STORAGE.md

@@ -1,61 +1,43 @@
 # Bulk NFS storage — the media library and the legacy Nextcloud archive
 
-The operator-facing companion to the `class: bulk` storage entries in `catalog/{plex,sonarr,radarr,nextcloud}.yaml`.
-Everything here was **measured** on the storage host on 2026-09-07, over read-only SSH; nothing is
-assumed. It resolves the long-standing **OQ-2** stub (`platform/longhorn/storageclasses.yaml`) for the
+The operator-facing companion to the `class: bulk` storage entries in
+`catalog/{plex,sonarr,radarr,qbittorrent,nextcloud}.yaml`.
+
+**Rewritten 2026-10-07 for the move off Proxmox onto a standalone TrueNAS at 192.168.2.30.** The
+layout (§2), the mount design (§3, §4) and the uid model (§6) are unchanged by the move and were
+measured on the old host on 2026-09-07. Everything about the storage host itself — §1, §5, §10 —
+was re-measured on the NAS on 2026-10-07. Nothing here is assumed. It resolves the long-standing
+**OQ-2** stub (`platform/longhorn/storageclasses.yaml`) for the
 case that actually exists here — attaching datasets that already hold data — and leaves dynamic bulk
 provisioning unimplemented, because nothing needs it.
 
 ---
 
-## 1. The storage host, and why prod cannot reach it
+## 1. The storage host
 
-The bulk data lives on the **Proxmox host's ZFS pool**. That host has exactly one interface:
+The bulk data lives on a **standalone TrueNAS** (Community Edition) at **192.168.2.30** — the same
+`192.168.2.0/24` as prod's nodes (`192.168.2.10/.11/.12`), so it is reached directly, with no NAT
+and no routing decision.
 
-```
-vmbr0  192.168.1.9/24        default via 192.168.1.1   (pfSense LAN)
-```
-
-| | nodes | reaches 192.168.1.9? |
-|---|---|---|
-| **dev** | 192.168.1.40 / .41 / .42 | **Yes** — same L2 segment, and it already holds an NFS export from this host (`/mnt/Storage/k8s-volumes`, in `/etc/exports`) |
-| **prod** | 192.168.2.10 / .11 / .12 | **No** — 192.168.2.0/24 is pfSense's **WAN**. Ports 22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. 2049 and 111 do not. |
-
-Opening the storage path for prod is a **pending operator decision**. Its two candidate outcomes give
-different values for the `BULK_NFS_SERVER` cluster-var:
-
-| outcome | `BULK_NFS_SERVER` for prod |
-|---|---|
-| port-forward `192.168.2.102:2049` → `192.168.1.9:2049` | `192.168.2.102` |
-| a WAN→LAN route to 192.168.1.0/24 | `192.168.1.9` |
-
-Until it lands, `clusters/prod` holds `TBD-PROD-BULK-NFS-SERVER`. **See §7 for why that placeholder
-behaves differently from the repo's other ones.**
+**This section used to be the hard part of this page.** The datasets were on the Proxmox host, whose
+only interface was `vmbr0 192.168.1.9/24` on pfSense's **LAN**, while prod sits on pfSense's **WAN**.
+tcp/2049 did not traverse that, so prod mounted through a DNAT at `192.168.2.102`, and which address
+belonged in `BULK_NFS_SERVER` depended on an undecided operator choice. Both pools were physically
+moved on 2026-10-07 (`docs/plans/proxmox-to-n5-air-migration.md`, Phase C/D) and all of that is
+simply gone. Measured 2026-10-07: **192.168.2.102:2049 no longer answers; 192.168.2.30:2049 and
+:111 do.**
 
 ### It only needs ONE port
 
 The PVs mount `nfsvers=4.1`. **NFSv4 carries mount and lock in-protocol**, so the only thing that
 must be reachable is **tcp/2049** — no rpcbind/111, no mountd, no statd. NFSv3 would have needed
-four ports, three of them on ephemeral numbers, which is a materially worse firewall story. The
-server advertises `+3 +4 +4.1 +4.2` (`/proc/fs/nfsd/versions`), so v4.1 is available and is the
-conservative floor.
+four ports, three of them on ephemeral numbers. The NAS advertises `+3 +4 +4.1 +4.2`
+(`/proc/fs/nfsd/versions`, measured 2026-10-07), so 4.1 is available and is the conservative floor.
 
-### The export's client list is the SAME under both outcomes — measured, not assumed
-
-**pfSense DNATs the existing port-forward WITHOUT source NAT.** Connecting from this workstation
-(192.168.2.132) to `192.168.2.102:22`, the storage host logged the peer as `192.168.2.132`:
-
-```
-SSH_CONNECTION=192.168.2.132 39768 192.168.1.9 22
-ss -tn  ->  192.168.1.9:22   192.168.2.132:39768
-```
-
-The original WAN-side source address survives the translation (return traffic finds its way back via
-the host's default route, 192.168.1.1). So under **either** outcome the NFS server will see prod's
-**real** node addresses — 192.168.2.10/.11/.12 — and the `sharenfs` change in §5 is the same either
-way. **It does not have to wait for the routing decision.**
-
----
+**NFSv4 is not on by default and its absence does not look like a storage problem.** TrueNAS ships
+NFSv3-only until NFSv4 is added to *Services → NFS → Enabled Protocols*. Without it the mount never
+completes and the pods sit in `ContainerCreating` — no permission error, nothing in the app logs,
+just no volume.
 
 ## 2. The layout, as found
 
@@ -107,16 +89,18 @@ why it is mounted read-only; see §6.
 | plex | `/mnt/Storage/plex` | `/media` | — | **ro** | ReadOnlyMany |
 | sonarr | `/mnt/Storage/plex` | `/media` | — | rw | ReadWriteMany |
 | radarr | `/mnt/Storage/plex` | `/media` | — | rw | ReadWriteMany |
+| qbittorrent | `/mnt/Storage/plex` | `/downloads` | `downloads` | rw | ReadWriteMany |
 | nextcloud | `/mnt/Storage/nextcloud` | `/mnt/legacy-nextcloud` | `nextcloud` | **ro** | ReadOnlyMany |
 
 In-app paths that follow: Plex libraries at `/media/movies`, `/media/tv`, `/media/photos`; Sonarr
 Root Folder `/media/tv`; Radarr Root Folder `/media/movies`; both importing from
 `/media/downloads/complete/{tv-sonarr,radarr}`.
 
-**All three media consumers use the identical mountPath on purpose.** When qbittorrent is eventually
-given the same export (it is not in this change), the arrs and the download client will already agree
-on absolute paths, which is the second precondition for hardlinked imports — otherwise every import
-needs a Remote Path Mapping and quietly falls back to copying when one is missing.
+**The three library consumers use the identical mountPath on purpose**, and qbittorrent has since
+been given the same export (narrowed by `subPath: downloads`, presented at `/downloads` to match its
+existing `Session\DefaultSavePath`). The arrs and the download client therefore agree on absolute
+paths, which is the second precondition for hardlinked imports — otherwise every import needs a
+Remote Path Mapping and quietly falls back to copying when one is missing.
 
 **`/media` and not `/data`, and that is not cosmetic.** The arrs' Longhorn config PVC renders at the
 renderer default `/data/config`. A media mount at `/data` would be mounted **over** it and mask the
@@ -147,62 +131,123 @@ Narrowing that would cost hardlinks, which is the worse trade.
 
 ---
 
-## 5. The export change the operator must apply — STRICTLY ADDITIVE
+## 5. The exports, as configured on TrueNAS — and the settings that decide permissions
 
-**Do not apply this from the repo.** It is an operator step on the storage host, recorded here so the
-exact value is reviewable. The existing consumers **must keep working**: 192.168.1.6, 10.123.1.10 and
-192.168.1.18 on the media tree, and 192.168.1.6 and 192.168.1.5 on the Nextcloud dataset, are the
-live non-k3s prod stack.
+**Do not apply this from the repo.** It is operator configuration on the NAS, recorded here so the
+exact values are reviewable. Everything below was **measured on 192.168.2.30 on 2026-10-07**, after
+the move.
 
-Current values (`zfs get sharenfs Storage`):
+### The exports
 
 ```
-Storage/plex       rw=@192.168.1.6:@10.123.1.10:@192.168.1.18,no_subtree_check,no_root_squash
-Storage/nextcloud  [email protected]:@192.168.1.5,no_subtree_check,no_root_squash
+/mnt/Storage/plex       192.168.2.10 192.168.2.11 192.168.2.12
+                        (sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,root_squash,no_all_squash)
+/mnt/Storage/nextcloud  192.168.2.10 192.168.2.11 192.168.2.12
+                        (sync,wdelay,hide,no_subtree_check,sec=sys,ro,secure,root_squash,no_all_squash)
 ```
 
-### `Storage/plex` — append the six k3s node addresses to the existing `rw=` list
+Per-node addresses rather than `@192.168.2.0/24`, following the precedent of the old host. A `/24`
+would work and would survive adding a node; it also grants the whole subnet.
+
+**All three nodes must appear on BOTH shares.** Any media pod can be scheduled on any node, so a
+missing node is a latent failure, not an immediate one: it surfaces the first time a pod reschedules
+there, as `access denied by server while mounting`, with the cause weeks stale. One was caught
+exactly this way during setup — the plex share listed `192.168.2.13`, an address no node holds, and
+so was missing `192.168.2.12` (k3s-prod-3). Nothing was failing at the time.
+
+### The option flags ARE the permission contract
+
+| flag | why it matters here |
+|---|---|
+| **`no_all_squash`** | The one the whole model rests on. Client uids pass through untouched, so the arrs' `PUID=1000` arrives as the owner of a `0755 1000:3000` tree. In the UI this is **Mapall User/Group left EMPTY** — setting either squashes every client to one identity and erases the reader/writer distinction entirely. |
+| **`sec=sys`** | AUTH_SYS. Numeric uid/gid, not Kerberos principals. §6 is only true because of this. |
+| **`root_squash`** | Maproot left empty. The Proxmox export carried `no_root_squash`; **nothing here needs it** — every write to the tree happens after the LSIO drop to uid 1000, and the images only chown their Longhorn `/config`. This is a deliberate tightening over the old export, not parity. |
+| **`rw` / `ro`** | The archive's read-only posture is **server-enforced**, as it was before. The media tree cannot get the same treatment — see below. |
+| **`secure`** | Reserved source ports required; "Allow non-root mount" off. The kubelet mounts as root from a reserved port and does not need the exception. |
+
+**Plex's read-only posture is still a CLIENT-side guarantee.** Plex is ro and the arrs are rw, but
+all four can land on any node, so the server sees the same three source addresses for all of them
+and cannot distinguish the reader from the writers. `spec.nfs.readOnly: true` on `plex-media` plus
+the volumeMount flag is the whole enforcement. Unchanged by the move; an honest limitation, not an
+oversight. If it ever needs to be server-enforced, the lever is a dedicated node pool for Plex.
+
+### Service settings that are NOT obvious from the export list
+
+- **Enabled Protocols must include NFSv4** (above).
+- **Require Kerberos for NFSv4: off.** The numeric AUTH_SYS pass-through is the model.
+- **Manage Groups Server-side: off.** This is `rpc.mountd --manage-gids`: the server *discards the
+  client's supplied group list* and looks that uid's groups up in TrueNAS's own user database
+  instead. With no local uid 1000 defined, the arrs' `PGID=3000` would simply vanish. Writes would
+  survive anyway — uid 1000 **owns** the tree, so the owner bits grant access without the group —
+  but it is a free way to lose group semantics.
+- **NFSv4 DNS Domain: blank.** It only feeds `idmapd.conf`, which numeric mapping bypasses.
+- **Bind IP 192.168.2.30** means loopback does not answer: check with `showmount -e 192.168.2.30`,
+  not `localhost`.
+
+There is **no** "NFSv3 ownership model for NFSv4" control on current TrueNAS — it was an older
+TrueNAS/CORE option, and no replacement is needed: on a Linux client against a Linux server over
+AUTH_SYS, both the `nfs` and `nfsd` modules default to `nfs4_disable_idmapping=Y`, so NFSv4 carries
+raw numeric ids rather than `user@domain` principals. §9's `ls -n` probe is how you confirm it.
+
+### The ZFS `sharenfs` properties must be CLEARED, not edited
+
+The pools arrived with the old host's exports still set as dataset properties:
+
+```
+Storage/plex              [email protected]:@10.123.1.10:@192.168.1.18:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash
+Storage/nextcloud         [email protected]:@192.168.1.5,[email protected]:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash
+Storage/docker-volumes    [email protected]:@192.168.1.5:@192.168.1.10:@192.168.1.18,no_subtree_check,no_root_squash
+```
+
+TrueNAS generates `/etc/exports` from its own middleware database and will fight a property it did
+not set — the failure mode is two differently-configured exports of one path, with no indication
+which is in force. Cleared 2026-10-07:
 
 ```sh
-zfs set sharenfs='[email protected]:@10.123.1.10:@192.168.1.18:@192.168.1.40:@192.168.1.41:@192.168.1.42:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash' Storage/plex
+zfs inherit sharenfs Storage/plex
+zfs inherit sharenfs Storage/nextcloud
+zfs inherit sharenfs Storage/docker-volumes
 ```
 
-Every pre-existing host is preserved verbatim and no option changes; only clients are added.
+`Storage` is `off`, so each resolves to `off`.
 
-### `Storage/nextcloud` — add the k3s nodes to a **`ro=`** list, not to `rw=`
+### Do NOT touch the dataset's permissions or ACLs
 
-```sh
-zfs set sharenfs='[email protected]:@192.168.1.5,[email protected]:@192.168.1.41:@192.168.1.42:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash' Storage/nextcloud
-```
+The tree's ownership **is** the access story (§6), and TrueNAS offers two ways to destroy it
+irreversibly across 8.33 T.
 
-This is the one place read-only can be enforced **server-side**, and it is worth taking: only
-Nextcloud consumes this dataset from k3s, so the export itself can refuse writes. Even a
-misconfigured client cannot then damage the archive.
+`Storage/plex` carries `acltype=nfsv4` (SCALE's built-in default, `SOURCE: default`) and
+`aclinherit=passthrough` (inherited). Measured 2026-10-07: the directories carry **trivial ACLs
+only** — `nfs4xdr_getfacl` shows just `owner@`/`group@`/`everyone@` matching the mode — so `0755
+1000:3000` governs exactly as it did under Proxmox.
 
-**The media tree cannot get the same treatment.** Plex is read-only and the arrs are read-write, but
-all three may be scheduled on any node, so the server sees the same source addresses for all of
-them and cannot distinguish the reader from the writers. **Plex's read-only posture is therefore a
-client-side guarantee** (`spec.nfs.readOnly: true` on `plex-media`, plus the volumeMount flag) — an
-honest limitation, not an oversight. If it ever needs to be server-enforced, the lever is a
-dedicated node pool for Plex, which is a much larger change.
+- **Stay out of the dataset's Edit Permissions / Edit ACL screen.** On an `acltype=nfsv4` dataset
+  that is a full ACL editor, and its presets write *real* NFSv4 ACLs, recursively if permitted.
+  There is no undo.
+- `aclinherit=passthrough` means a single ACL written at `/mnt/Storage/plex` would be **inherited by
+  everything created under it afterwards** — the blast radius is the whole future library, not one
+  directory.
+- Note GNU `ls`'s `+` marker is **not** a reliable ACL indicator here: it looks for a POSIX ACL
+  xattr, and this dataset stores NFSv4 ACLs elsewhere. Use `nfs4xdr_getfacl`.
 
-Per-node addresses rather than `@192.168.2.0/24` follows the precedent already on this host
-(`/etc/exports` lists the dev nodes individually, as `/32`). A `/24` would work and would survive
-adding a node; it also grants the whole subnet. Per-node is the tighter default.
+`xattr=on` rather than `sa` is a performance detail, not a permissions one, and changing it would
+not affect existing files.
 
 ### Verifying, without changing anything
 
 ```sh
-exportfs -v                  # the new client lines appear, existing ones unchanged
-zfs get sharenfs Storage/plex Storage/nextcloud
+cat /proc/fs/nfsd/versions          # +4.1 must be present
+exportfs -v                         # the two shares, three nodes each, flags as tabulated above
+showmount -e 192.168.2.30           # NOT localhost -- the service is bound to one address
+zfs get -r -o name,property,value,source sharenfs Storage    # every row off or -
 ```
 
----
-
 ## 6. uid / gid — the failure this design exists to avoid
 
-The export uses `sec=sys` and `no_root_squash`, so **the client's numeric uid is passed through
-verbatim** and the server's on-disk ownership is the entire access story. The media tree is
+The export uses `sec=sys` and `no_all_squash`, so **the client's numeric uid is passed through
+verbatim** and the server's on-disk ownership is the entire access story. (It no longer carries
+`no_root_squash`, which the Proxmox export did — see §5. Nothing here needs it, and the uid that
+matters is 1000, not 0.) The media tree is
 `0755 1000:3000` throughout (some directories `1000:1000`), so **only uid 1000 can write it**.
 
 **`fsGroup` cannot fix this, and reaching for it is how this gets misdiagnosed.** The in-tree NFS
@@ -233,24 +278,27 @@ Nextcloud needs no uid work at all, which is one more reason its mount is read-o
 
 ---
 
-## 7. The prod placeholder does NOT fail at apply time — read this before merging
+## 7. Why this knob is unvalidated at create AND immutable at update
+
+**The `TBD-PROD-BULK-NFS-SERVER` placeholder is gone** (2026-10-07) — prod now carries a real
+address, because the NAS is on prod's own subnet and there is no longer a decision to defer. The
+hazard that made it load-bearing is worth keeping on the page, because it is the half that explains
+why §10 exists.
 
 Every other `TBD-PROD-*` in `clusters/prod/kustomization.yaml` is rejected loudly by CRD validation
-(Cilium, LB-IPAM, CNPG, Velero) because it is not a valid IP or URL. **This one is not.**
+(Cilium, LB-IPAM, CNPG, Velero) because it is not a valid IP or URL. **This one was not.**
 
-A PersistentVolume's `spec.nfs.server` is an **unvalidated string**. The API server accepts
-`TBD-PROD-BULK-NFS-SERVER`, the PV is created, and the PVC **binds** to it — binding is pure
-control-plane and never touches the server. The failure appears only when a pod mounts:
+A PersistentVolume's `spec.nfs.server` is an **unvalidated string**. The API server accepted
+`TBD-PROD-BULK-NFS-SERVER`, the PV was created, and the PVC **bound** to it — binding is pure
+control-plane and never touches the server. The failure appeared only when a pod mounted:
 
 ```
 MountVolume.SetUp failed ... Failed to resolve server TBD-PROD-BULK-NFS-SERVER
 ```
 
-so **plex, sonarr, radarr and nextcloud would sit in ContainerCreating on prod.**
-
-**Consequently: set `BULK_NFS_SERVER` in `clusters/prod/kustomization.yaml` BEFORE prod reconciles a
-revision that carries these bulk volumes.** Dev renders a working value and is where these manifests
-are meant to be proven first.
+**The same field is immutable on UPDATE** (§10), which is the other half of the same awkwardness: a
+wrong value is accepted silently at create, and then cannot be corrected in place. Together they are
+why repointing this knob is a delete-and-recreate rather than a reconcile.
 
 ### The static guard this convention leans on is missing from `main` — but the gate now refuses the value anyway
 
@@ -287,10 +335,9 @@ Without the arm, HALF A is what holds the line, by design.
    EPHEMERAL layer"). Plex was fixed; sonarr, radarr, qbittorrent and jackett were not. Fixing it
    means those apps start from an empty config volume, so it belongs in its own change with the
    operator present. **This change does not make it worse** — that is why the media mount is `/media`.
-2. **qbittorrent has no bulk mount.** Today the arrs' import source is written by the **legacy
-   non-k3s** download client, which is coherent while the legacy stack stays live. Giving the k3s
-   qbittorrent the same export at the same `/media` path is the natural follow-up, and the paths were
-   chosen so it needs no Remote Path Mapping.
+2. **RESOLVED — qbittorrent now has the bulk mount** (`subPath: downloads` at `/downloads`), so the
+   import source is written by the k3s download client rather than the legacy non-k3s one. See
+   `catalog/qbittorrent.yaml`.
 3. **`/mnt/Storage/plex/downloads/downloads` is a symlink to `/shared/downloads`**, which is outside
    the export. Over NFS it is a dangling symlink on every client. Harmless, but expect Sonarr to log
    it while scanning `downloads/`.
@@ -300,10 +347,10 @@ Without the arm, HALF A is what holds the line, by design.
 5. **`nconnect` was not set.** It would likely help throughput on the media tree, but it is a tuning
    change that should be measured, not guessed, and it changes client behaviour under server
    failure.
-6. **Nothing was verified against a live cluster.** No `kubectl` context is usable from this
-   workstation (browser-OIDC only), and this change is manifest-authoring under an explicit
-   do-not-apply constraint. Every claim about the *host* is measured; every claim about *binding and
-   mounting* is reasoned from the manifests.
+6. **Every claim about the NAS in §5 is measured (2026-10-07); the post-move MOUNT is not.** The PVs
+   still have to be recreated (§10) before any pod mounts 192.168.2.30, so the §9 probes are the
+   outstanding verification, not a formality. The pre-move equivalents did pass against the old host
+   on 2026-09-12.
 7. **The bulk-tier requirement ID is deliberately not typed on any non-`.md` line of this change.**
    `ci/touched-requirements.sh` attributes requirements from R-IDs appearing in added lines, and the
    in-cluster conformance test for the bulk tier is still a gap, so typing it would select a
@@ -312,36 +359,95 @@ Without the arm, HALF A is what holds the line, by design.
 
 ---
 
-## 9. Validating on dev, once the export is extended
+## 9. Validating, after the PVs are recreated
 
-Dev can prove all of this before prod's network path exists. Read-only checks first:
+These run against **prod**; dev is decommissioned. They are the outstanding verification for the
+move — nothing below has been run against 192.168.2.30. Read-only checks first:
 
 ```sh
-infra/devcontainer/dev.sh run kubectl --context k3s-dev get pv
-infra/devcontainer/dev.sh run kubectl --context k3s-dev get pvc -A | grep -E 'media|legacy-archive'
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr get pod -o wide
+./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pv -o custom-columns=\
+NAME:.metadata.name,SERVER:.spec.nfs.server,PATH:.spec.nfs.path,STATUS:.status.phase'
 ```
 
-Expect **four** PVs — `plex-media`, `sonarr-media`, `radarr-media`, `nextcloud-legacy-archive` (three
-of them naming the *same* export, one per namespace, which is the cross-namespace model) — each
-`Bound` to the same-named PVC, with `RECLAIM POLICY Retain` and an empty `STORAGECLASS`.
+Expect **five** NFS PVs — `plex-media`, `sonarr-media`, `radarr-media`, `qbittorrent-downloads`,
+`nextcloud-legacy-archive` (four of them naming the *same* export, one per namespace, which is the
+cross-namespace model) — each `Bound`, `RECLAIM POLICY Retain`, empty `STORAGECLASS`, and
+**`SERVER 192.168.2.30`**. A PV still showing `192.168.2.102` means the recreate in §10 did not
+happen; Flux will NOT have updated it in place.
 
-Then the two things that actually matter, from inside a pod:
+Then the four things that actually matter, from inside a pod:
 
 ```sh
-# (a) the writer can write — this is the uid check, and it is the one that fails in practice
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr exec deploy/sonarr -- \
-  sh -c 'id; touch /media/tv/.k3s-write-probe && rm /media/tv/.k3s-write-probe && echo WRITE-OK'
+# (a) numeric ids survived the wire -- the idmapping check, new since the TrueNAS move
+kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- sh -c 'ls -n /media | head -3'
+#     expect uid 1000 / gid 3000 on the entries. 65534 / nobody means ids are being mapped, not
+#     passed through, and every import will fail EACCES while the pod still reports Ready.
 
-# (b) hardlinks work BETWEEN the library and the download path — the root-mount check
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr exec deploy/sonarr -- \
+# (b) the writer can write -- the uid check, and the one that fails in practice
+kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- \
+  sh -c 'touch /media/tv/.k3s-write-probe && rm /media/tv/.k3s-write-probe && echo WRITE-OK'
+
+# (c) hardlinks work BETWEEN the library and the download path -- the root-mount check
+kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- \
   sh -c 'set -e; d=/media/downloads/complete/tv-sonarr/.probe; t=/media/tv/.probe-link;
          echo x > "$d"; ln "$d" "$t"; ls -i "$d" "$t"; rm -f "$d" "$t"; echo HARDLINK-OK'
 
-# (c) the reader really is read-only
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n plex exec deploy/plex -- \
-  sh -c 'touch /media/.probe 2>&1 | head -1'      # expect "Read-only file system"
+# (d) the reader really is read-only
+kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c 'touch /media/.probe 2>&1 | head -1'
+#     expect "Read-only file system"
 ```
 
-`(b)` printing two identical inode numbers is the proof that §4's argument holds in practice. If it
+`(c)` printing two identical inode numbers is the proof that §4's argument holds in practice. If it
 prints `EXDEV`, something has split the single mount back into two.
+
+Reading the failures together is what localises the fault: **(b) EACCES with (a) showing `nobody`**
+is idmapping (§5 service settings). **(b) EACCES with (a) showing `1000`** is on-disk ownership —
+stop and check `ls -ldn` on the NAS before changing anything else.
+
+---
+
+## 10. Repointing BULK_NFS_SERVER means DELETING AND RECREATING THE PVs
+
+**A bound PersistentVolume's NFS source is immutable.** Measured by server-side dry-run on
+2026-10-07:
+
+```
+$ kubectl patch pv sonarr-media --type=json --dry-run=server \
+    -p '[{"op":"replace","path":"/spec/nfs/server","value":"192.168.2.30"}]'
+The PersistentVolume "sonarr-media" is invalid: spec.persistentvolumesource: Forbidden:
+  spec.persistentvolumesource is immutable after creation
+```
+
+Note the pairing with §7: `spec.nfs.server` is **unvalidated at create** (any string is accepted and
+the PVC binds) and **immutable at update**. So editing `BULK_NFS_SERVER` alone does not fail loudly
+and does not work either — Flux's apply errors on the five PVs and the pods keep their previous,
+now-dead mount.
+
+This is safe to do because `persistentVolumeReclaimPolicy: Retain` means deleting a PV never touches
+the data, and `catalog/render.py` emits a **pre-set `claimRef`** on each one, so a recreated PV
+rebinds its own PVC with no manual step.
+
+**Operator sequence** (mutating `kubectl` is an operator action, not an agent action):
+
+```sh
+# 1. Stop the consumers. This also releases the hung hard-mounts against the old server:
+#    the pods report Ready, but any real I/O on /media blocks indefinitely once 2049 stops
+#    answering, because the PVs mount `hard`.
+for ns in plex sonarr radarr qbittorrent nextcloud; do
+  kubectl --context k3s-prod -n $ns scale deploy --all --replicas=0
+done
+
+# 2. Delete the five PVs. Retain + no live claim means nothing is reclaimed or erased.
+kubectl --context k3s-prod delete pv plex-media sonarr-media radarr-media \
+  qbittorrent-downloads nextcloud-legacy-archive
+
+# 3. Let Flux reconcile the revision carrying BULK_NFS_SERVER=192.168.2.30, which recreates them.
+#    Confirm .status.lastAppliedRevision matches the GitRepository artifact before continuing.
+
+# 4. Bring the consumers back, then run every probe in §9.
+```
+
+The PVCs pass through `Lost` between steps 2 and 3; that is expected and resolves when the PV with
+the matching `claimRef` reappears. If a PVC is still `Lost` after Flux reconciles, the PV name or
+its `claimRef` namespace does not match — do **not** delete the PVC to "fix" it.
+

docs/plans/pm-agent-issue-to-pr.md

@@ -233,9 +233,21 @@ about runtime behaviour in this document is a design claim, not a measurement.
 
 ## Follow-ups, in the order they are worth doing
 
-1. **Build, push, pin.** Operator runs `infra/agent/build-image.sh`, pushes, and one PR
-   replaces both placeholder digests. Expect the build to need iteration: the nodejs module
-   stream and the npm install have never run against this base.
+1. **Make the registry pullable, THEN build, push, pin.** Measured after the merge, and
+   neither half is declared anywhere in this repository
+   (`decisions/2026-10-05-pm-agent-registry-pull-prereq.md`):
+   (i) the nodes resolved the registry hostname to the WAN IP and failed TLS against the
+   router's `CN=%D` placeholder certificate — **diagnosed and fixed** by pinning
+   `gitea.{{ k3s_base_domain }}` to the Gateway VIP in `node_baseline_hosts_pins`, the same
+   pin the Keycloak issuer already carries, applied by
+   `ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`;
+   (ii) the pull is still unauthenticated and no `imagePullSecret` exists in the tree — with
+   the credential decision now being local Gitea accounts with PATs, that means a
+   `dockerconfigjson` for an account whose token carries `read:package`. Only then does the
+   digest matter. Operator runs `infra/agent/build-image.sh`, pushes, and one PR replaces both
+   placeholder digests. Expect the build itself to need iteration too: the nodejs module
+   stream and the npm install have never run against this base. The same two blockers sit in
+   front of `conformance-runner`, which shares the registry path and has never been pulled.
 2. **Create the credentials** per `pm-agent-credentials.sops.yaml.example`, and **verify the
    bot's permission is `write`, not `admin`** — that measurement is the control that keeps
    the agent out of its own approval gate.

infra/agent/pm-worker

@@ -90,9 +90,19 @@ esac
 # pointing at an external host would be REACHABLE: it could clone a repository that is not
 # this one and push this repository's work somewhere else. The allowlist is baked in here,
 # where changing it requires rebuilding the image whose digest admission pins.
+#
+# ONE ENTRY, AND IT IS THE CLUSTER-LOCAL SERVICE NAME. This agent runs INSIDE the cluster, so
+# it reaches Gitea the way every other in-cluster consumer does — `*.svc.cluster.local` —
+# which is by design a different path from how a client outside the cluster reaches the same
+# service. The external hostname `gitea.home.keithjackson.dev` used to be a second accepted
+# value here; it is removed deliberately. It was never needed (nothing sets it), it routes
+# in-cluster traffic out through the Gateway and back for no reason, and as an ACCEPTED value
+# it widened this allowlist to a host reachable over the `world:443` egress rule — which is
+# the exact shape of injection the allowlist exists to refuse. The cluster-local name also
+# depends on none of public DNS, Cloudflare, the Gateway or the certificate.
 case "${PM_GITEA_URL}" in
-  http://gitea.gitea.svc.cluster.local:3000|https://gitea.home.keithjackson.dev) ;;
-  *) die "PM_GITEA_URL '${PM_GITEA_URL}' is not an allowed git host (see this script's header)" ;;
+  http://gitea.gitea.svc.cluster.local:3000) ;;
+  *) die "PM_GITEA_URL '${PM_GITEA_URL}' is not the allowed in-cluster git host (see this script's header)" ;;
 esac
 [[ "${PM_GITEA_OWNER}" =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,38}$ ]] || die "PM_GITEA_OWNER is malformed"
 [[ "${PM_GITEA_REPO}"  =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,99}$ ]] || die "PM_GITEA_REPO is malformed"

infra/agent/probe-validation.sh

@@ -86,12 +86,22 @@ refuses "branch equal to the base branch"       "must not be the base branch" \
 # --- where code comes FROM and goes TO -------------------------------------
 # The namespace's egress policy permits tcp/443 to the world (the model needs it), so an
 # injected host would be REACHABLE. These are the cases that matter most.
-refuses "a redirected git host"                 "is not an allowed git host" \
+refuses "a redirected git host"                 "is not the allowed in-cluster git host" \
         PM_GITEA_URL=https://evil.invalid
-refuses "a real, reachable, WRONG git host"     "is not an allowed git host" \
+refuses "a real, reachable, WRONG git host"     "is not the allowed in-cluster git host" \
         PM_GITEA_URL=https://github.com
-refuses "the right host on a smuggled path"     "is not an allowed git host" \
+refuses "the right host on a smuggled path"     "is not the allowed in-cluster git host" \
         PM_GITEA_URL=http://gitea.gitea.svc.cluster.local:3000/../..
+# This agent is IN the cluster, so it reaches Gitea by its cluster-local service name. The
+# EXTERNAL hostname is refused too — it used to be a second accepted value, and as an
+# accepted value it widened this allowlist to a host reachable over the `world:443` egress
+# rule, which is the injection shape the allowlist exists to refuse.
+refuses "this Gitea's EXTERNAL hostname"        "is not the allowed in-cluster git host" \
+        PM_GITEA_URL=https://gitea.home.keithjackson.dev
+refuses "the cluster-local name over https"     "is not the allowed in-cluster git host" \
+        PM_GITEA_URL=https://gitea.gitea.svc.cluster.local:3000
+refuses "the cluster-local name, wrong port"    "is not the allowed in-cluster git host" \
+        PM_GITEA_URL=http://gitea.gitea.svc.cluster.local:8080
 refuses "a malformed owner"                     "PM_GITEA_OWNER is malformed" \
         'PM_GITEA_OWNER=../other-org'
 refuses "a malformed repository"                "PM_GITEA_REPO is malformed" \

infra/ansible/inventory/group_vars/k3s_prod/vars.yml

@@ -327,6 +327,29 @@ node_baseline_hosts_pins:
   - ip: "192.168.2.20"
     names:
       - "keycloak.{{ k3s_base_domain }}"
+      # gitea: the SECOND node process to need this pin, for the identical reason, found
+      # 2026-10-06. containerd pulls images as a NODE process, so it resolves through the
+      # public upstreams above and gets the CNAME to wan.home.keithjackson.dev — the WAN IP,
+      # which this router does not hairpin. The pull then fails on the router's own
+      # placeholder certificate rather than on anything to do with the image:
+      #
+      #   failed to resolve reference "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:…":
+      #   tls: failed to verify certificate: x509: certificate is not valid for any names,
+      #   but wanted to match gitea.home.keithjackson.dev
+      #
+      # Measured from the workstation the same day, against the WAN address with that SNI:
+      #   subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D
+      #   No extensions in certificate          <- no SANs at all, hence "not valid for ANY names"
+      # while the same name via PiHole answers 192.168.2.20 and serves the real
+      # Let's Encrypt *.home.keithjackson.dev. So the certificate and the Gateway are fine;
+      # only the node's ANSWER is wrong. Same root cause, same fix, as the issuer pin above.
+      #
+      # WHO NEEDS IT: any workload whose image lives in the in-cluster Gitea registry —
+      # today platform/pm-agent (agent-runner) and conformance/incluster
+      # (conformance-runner), neither of which has ever been pulled. A pull is still
+      # UNAUTHENTICATED after this pin, which is a separate thing to settle; this only makes
+      # the node reach the right endpoint.
+      - "gitea.{{ k3s_base_domain }}"
 
 # --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) --------------------
 # PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/
@@ -397,3 +420,34 @@ flux_bootstrap_env_path: "clusters/prod"
 # rather than leaving an inert drop-in behind, so this group-wide `true` is still correct
 # for it.
 etcd_io_priority_enabled: true
+
+# --- automatic security patching (roles/unattended_upgrades) -------------------------
+# TURNS ON roles/unattended_upgrades FOR PROD ONLY. Role default is false. Same seam as
+# node_baseline_backports_kernel and etcd_io_priority_enabled above.
+#
+# WHAT WAS THERE BEFORE: nothing at all. Measured on all three prod nodes 2026-10-17 —
+# `unattended-upgrades`, `update-notifier-common` and `needrestart` were ALL absent,
+# `apt-mark showhold` was empty on every node, /etc/apt/preferences.d/ was empty, and
+# nothing on the node could create /var/run/reboot-required. These boxes have had ZERO
+# automatic patching since provisioning, and no way to signal that a reboot was needed.
+#
+# WHAT ENABLING THIS DOES *NOT* DO: nothing, until the operator runs
+# playbooks/unattended-upgrades.yml. It is not applied by site.yml (deliberately — see
+# that playbook's header). And once run, it still never upgrades the KERNEL, never
+# restarts k3s, and never reboots:
+#   * kernel — blacklisted `^linux-`, AND the trixie-backports archive the running
+#     kernel comes from (o=Debian Backports, measured) is not an allowed origin. Two
+#     independent exclusions, because this is the EFI-memattr constraint: these HP minis
+#     need >= 6.13 or they can boot with a fraction of their RAM
+#     (PROD-PROVISIONING.md §0b), and 6.12 kernels are still installed and bootable here.
+#     The kernel continues to move ONLY under a deliberate
+#     `node-baseline.yml --tags kernel` run, exactly as before.
+#   * k3s — needrestart blacklist. k3s is not even an apt package on these nodes
+#     (measured: no `k3s`, no `containerd` in dpkg), so the control plane and runtime are
+#     outside apt's reach regardless.
+#   * reboots — Automatic-Reboot "false", asserted by the role against apt's own
+#     effective config. Reboots belong to kured (platform/kured): one node at a time,
+#     under a cluster-wide lock, inside 03:00-06:00, after cordon + drain. Three
+#     control-plane nodes with etcd quorum 2/3 must never reboot themselves
+#     independently, which is what Automatic-Reboot "true" would mean.
+unattended_upgrades_enabled: true

infra/ansible/playbooks/unattended-upgrades.yml

@@ -0,0 +1,36 @@
+---
+# Automatic Debian SECURITY patching for the k3s nodes — the package half of the automated upgrade
+# process. The reboot half is kured (platform/kured), reconciled by Flux; the two are designed
+# together and neither is useful alone. Full rationale for every decision:
+# roles/unattended_upgrades/defaults/main.yml and that role's README.md.
+#
+# TARGET (R-ENV-02): `k3s_hosts` defaults to dev's `k3s_servers`; prod is
+#   ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod
+# Same pattern node-baseline.yml and node-health.yml established — `hosts:` is templated before
+# group_vars exist, so the target is an extra var, never a group_vars value.
+#
+# The role is a NO-OP unless `unattended_upgrades_enabled` is true, which is set for prod only in
+# inventory/group_vars/k3s_prod/vars.yml. Running this against dev changes nothing.
+#
+# DELIBERATELY NOT IN site.yml. site.yml is the bring-up chain (node-baseline -> k3s ->
+# etcd-io-priority -> kube-vip -> cilium -> verify -> flux-bootstrap) and is "not run against prod
+# casually". Automatic patching is not a prerequisite for anything in that chain and nothing in the
+# chain depends on it, so adding it there would only widen what a cluster rebuild touches. It is
+# applied on its own, once, and re-applied when its configuration changes.
+#
+# SAFE TO RUN ON A LIVE CLUSTER, AT ANY TIME. This playbook installs and configures the upgrader. It
+# never runs `apt upgrade`, never restarts k3s, and never reboots — the role asserts the last of
+# those against apt's own effective config rather than trusting the file it just wrote. The first
+# unattended upgrade happens at the next timer firing (04:00 +/-60m), and any reboot that upgrade
+# implies is kured's to schedule, one node at a time, inside 03:00-06:00.
+#
+# TAGS. `--tags verify` re-runs only the read-only assertions and the per-node summary, which is the
+# way to answer "is automatic patching actually working on these boxes, and is anything waiting on a
+# reboot" without rewriting any configuration:
+#   ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify
+- name: Automatic security upgrades on the k3s nodes (kernel excluded; reboots delegated to kured)
+  hosts: "{{ k3s_hosts | default('k3s_servers') }}"
+  become: true
+  gather_facts: true
+  roles:
+    - unattended_upgrades

infra/ansible/roles/unattended_upgrades/README.md

@@ -0,0 +1,195 @@
+# roles/unattended_upgrades
+
+Automatic Debian **security** patching on the k3s nodes. The kernel is excluded, k3s is never
+restarted, and reboots are delegated to **kured** (`platform/kured`).
+
+- **Playbook:** `playbooks/unattended-upgrades.yml`
+- **Enabled by:** `unattended_upgrades_enabled` — role default `false`, set `true` for prod in
+  `inventory/group_vars/k3s_prod/vars.yml`
+- **Companion component:** `platform/kured` (the reboot half). **Neither is useful alone** — see
+  "The two halves" below.
+- **Every decision and its reasoning:** `defaults/main.yml`. That file is the argument; this one is
+  the orientation.
+
+## The two halves, and why they ship together
+
+| | what it does | what it must NOT do |
+|---|---|---|
+| **this role** (Ansible, host) | installs security updates continuously; leaves `/var/run/reboot-required` behind when one needs a reboot | upgrade the kernel; restart k3s; reboot |
+| **kured** (Flux, cluster) | cordons, drains, reboots and uncordons **one node at a time** when that file exists | decide *whether* a reboot is needed |
+
+Deploying either alone produces a mechanism that looks healthy and does nothing:
+
+- **kured alone** watches a file that **nothing on Debian ever creates**. Measured 2026-10-18,
+  after a first attempt to install Ubuntu's `update-notifier-common` failed on all three nodes
+  with *"No package matching 'update-notifier-common' is available"*:
+  - `apt-cache policy update-notifier-common` → `Candidate: (none)` — the package does not exist
+    in Debian; it is Ubuntu-only.
+  - `/usr/share/update-notifier/notify-reboot-required` → absent.
+  - **No `linux-image-*` postinst references `notify-reboot-required`** — calling it is an Ubuntu
+    patch, so even dropping a script at that path would never be invoked.
+  - No hook under `/etc/apt/apt.conf.d` or `/etc/kernel/` mentions `reboot-required`.
+
+  So **this role ships the writer itself** (`/usr/local/sbin/k3s-notify-reboot-required` plus a
+  `DPkg::Post-Invoke` hook) rather than installing a package. Without it kured would sit at 1/1
+  Running reporting `kured_reboot_required=0` forever.
+- **this role alone** installs patches that a long-running process never picks up, and when a
+  reboot *is* needed there is nothing safe to take it. `Automatic-Reboot "true"` is not the
+  answer: three control-plane nodes with etcd quorum 2/3, each rebooting at its own time, is how
+  the cluster is lost.
+
+## What it was replacing: nothing
+
+Measured on all three prod nodes, 2026-10-17:
+
+| | |
+|---|---|
+| `unattended-upgrades` | **not installed** |
+| `update-notifier-common` | **not installable** — Ubuntu-only, `Candidate: (none)` on trixie |
+| `needrestart` | **not installed** |
+| `apt-mark showhold` | **empty** — no package held on any node |
+| `/etc/apt/preferences.d/` | **empty** — no apt pin declares the backports-kernel intent |
+| `/var/run/reboot-required` | absent, and uncreatable |
+
+These nodes had received **no automatic patching since provisioning**.
+
+## The three decisions
+
+### 1. Security only
+
+Allowed origins are `label=Debian-Security` and nothing else. `trixie-updates` is excluded
+deliberately (not overlooked) — it carries non-security fixes that are useful but are the archive
+through which an unreviewed change would most plausibly arrive on a control-plane node. The knob to
+add it is `unattended_upgrades_include_updates`.
+
+### 2. The kernel is excluded twice over
+
+**This is the highest-risk part of the change.** These HP EliteDesk 800 G5 minis publish a corrupt
+EFI Memory Attributes Table on R21 firmware; Linux `< 6.13` does not validate it before reserving
+memory from its contents, and `k3s-prod-2` twice booted with **2.6 GiB of 16 GB**. Upstream
+`8fbe4c49c0cc` (first in v6.13) is the only fix — there is no kernel cmdline workaround. Full
+write-up: `infra/PROD-PROVISIONING.md` §0b and `roles/node_baseline/defaults/main.yml:21-51`.
+
+Measured 2026-10-17: all three nodes run `7.1.8+deb13-amd64` (`7.1.8-1~bpo13+1`, i.e. the backports
+metapackage), comfortably above the floor — **but `linux-image-6.12.107+deb13-amd64` and
+`linux-image-6.12.94+deb13-amd64` are also still installed, below the floor, bootable, and held by
+nothing.**
+
+So an unattended process must not be able to touch the kernel at all:
+
+1. **Origin exclusion (structural).** The running kernel comes from `o=Debian Backports` (measured
+   via `apt-cache policy`, pin priority 100 — the `NotAutomatic` + `ButAutomaticUpgrades`
+   signature). That origin is not in `Origins-Pattern`, so unattended-upgrades cannot see a kernel
+   upgrade at all.
+2. **Package blacklist (belt and braces).** `^linux-` is blacklisted, so even if the origin list is
+   later widened — the most likely future edit to this role, and the one that would silently re-arm
+   the hazard — the kernel is still untouchable.
+
+The role **asserts both** against `apt-config dump` (apt's own parse of the whole
+`/etc/apt/apt.conf.d/` directory) rather than trusting the file it just wrote.
+
+**This does not stop the kernel moving.** The operator still advances it deliberately with
+
+```
+ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel --limit <host>
+```
+
+which is the only thing that maintains the backports source, the metapackage and the on-node
+provenance stamp. `node_baseline`'s own note that `apt upgrade` keeps the backports kernel current
+still holds for an operator-run apt. What changed is that a **new unattended actor** is forbidden
+from it.
+
+Also blacklisted, for the same "operator-only" reason: `intel-microcode`, `amd64-microcode`,
+`grub-*`. Microcode and bootloader changes take effect on reboot and alter boot behaviour on the
+very firmware that is already misbehaving here.
+
+`Remove-Unused-Kernel-Packages` is **false**, against the Debian default: those older kernels are
+the documented rollback path, and an unattended process pruning them could remove the fallback an
+operator needs in front of a box that will not boot.
+
+### 3. needrestart restarts services — but never k3s
+
+A security-only policy that restarts nothing patches bytes on disk that no running process reads.
+`needrestart` in mode `a` restarts services still mapped to a deleted library, which is what makes
+the policy actually effective.
+
+`k3s.service` is blacklisted. Restarting the Kubernetes control plane unattended at 04:00 is a
+worse outcome than a delayed library fix; that fix lands at the next kured reboot instead. Measured
+2026-10-17, `k3s.service` is the **only** matching unit — k3s bundles its own containerd and
+kubelet, and neither `containerd` nor `containerd.io` is an apt package on these nodes — so this
+one entry is complete today.
+
+## Schedule
+
+`apt-daily` at 03:30 and `apt-daily-upgrade` at 04:00, both `±60m` randomised, via systemd drop-ins.
+Chosen to land **inside** kured's 03:00–06:00 reboot window so that a sentinel written by an upgrade
+can be acted on the same night rather than ~24h later.
+
+Two traps the role handles explicitly:
+
+- `/etc/apt/apt.conf.d/20auto-upgrades` must set the four `APT::Periodic::*` counters. Without
+  them the apt-daily timers fire and **do nothing** — installed, configured, green, and inert.
+- The timer drop-ins clear `OnCalendar=` before setting it. systemd **accumulates** `OnCalendar`
+  entries across drop-ins, so omitting the reset keeps the shipped schedule *in addition* to ours.
+
+## Why a new role rather than extending `node_baseline`
+
+`node_baseline` is the repo's only other apt surface and would have been a defensible home. Two
+reasons against:
+
+1. **Single responsibility.** `node_baseline` is the *first-boot floor* — it runs before k3s exists
+   and is the thing that makes a node installable. Continuous patching policy is a different
+   lifecycle with a different blast radius, and it wants to be runnable (and re-runnable) without
+   going anywhere near that role's data-disk tasks.
+2. **Concurrent work.** Other open branches touch `roles/node_baseline/`. A new role has nothing to
+   merge.
+
+Nothing here duplicates `node_baseline`: it does not add apt sources, does not touch the kernel, and
+does not manage disks.
+
+## Post-merge verification
+
+Running the role is safe on a live cluster at any time — it installs and configures the upgrader, and
+never runs `apt upgrade`, restarts k3s, or reboots.
+
+```bash
+# Apply (prod). Idempotent; re-runnable.
+cd infra/ansible && ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod
+
+# Re-run ONLY the read-only assertions + the per-node pending-reboot summary.
+cd infra/ansible && ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify
+```
+
+Then, on a node, confirm the three properties that matter (all read-only, no sudo needed for the
+first two):
+
+```bash
+ssh [email protected] 'apt-config dump | grep -iE "unattended-upgrade::(automatic-reboot|package-blacklist|origins)"'
+ssh [email protected] 'systemctl list-timers apt-daily-upgrade.timer --no-pager'
+ssh [email protected] 'ls -la /var/run/reboot-required* 2>&1 | head -2
+  ls -l /usr/local/sbin/k3s-notify-reboot-required /etc/apt/apt.conf.d/99-k3s-reboot-required 2>&1
+  dpkg -l needrestart unattended-upgrades | tail -3'
+```
+
+Expect: `Automatic-Reboot "false"`, a blacklist containing `^linux-`, an `Origins-Pattern`
+containing `Debian-Security` and **not** `Backports`, a next-elapse around 04:00, and all three
+packages `ii`.
+
+A dry run of what the upgrader would actually do (needs root, so this one is an operator step):
+
+```bash
+sudo unattended-upgrade --dry-run --debug
+```
+
+The kernel must appear in its blacklist output, not its upgrade list.
+
+## What this role cannot tell you
+
+- **That a security upgrade has ever succeeded.** That is in
+  `/var/log/unattended-upgrades/unattended-upgrades.log` on each node, and nothing in this repo or
+  in Prometheus reads it. There is no alert for "automatic patching has silently stopped working".
+  That is a real, known gap — see the handoff entry
+  `decisions/2026-10-17-automated-upgrades.md`.
+- **That a reboot actually happened.** That is kured's half, and it *is* alerted on
+  (`KuredRebootPending`, `KuredNodeCoverageGap`, `KuredNotReporting` in
+  `platform/kured/alerts.yaml`).

infra/ansible/roles/unattended_upgrades/defaults/main.yml

@@ -0,0 +1,199 @@
+---
+# roles/unattended_upgrades — automatic Debian SECURITY patching on the k3s nodes, with the kernel
+# deliberately excluded and with reboots delegated to kured (platform/kured).
+#
+# OFF BY DEFAULT, ON FOR PROD. Same seam as node_baseline_backports_kernel / etcd_io_priority_enabled:
+# a plain boolean, default false, turned on in inventory/group_vars/k3s_prod/vars.yml. Not a hostname
+# test — that would encode the environment into the role (see node_baseline's defaults for the full
+# argument, which this role follows rather than restates).
+#
+# ═════════════════════════════════════════════════════════════════════════════════════════════════
+# STATE BEFORE THIS ROLE (measured on all three prod nodes, 2026-10-17, via `dpkg-query -W`)
+# ═════════════════════════════════════════════════════════════════════════════════════════════════
+#   unattended-upgrades      NOT INSTALLED
+#   update-notifier-common   NOT INSTALLED
+#   needrestart              NOT INSTALLED
+#   apt-mark showhold        EMPTY (no package is held on any node)
+#   /var/run/reboot-required ABSENT, and nothing on the node can create it
+#   /etc/apt/preferences.d/  EMPTY (no apt pin declares the backports-kernel intent)
+# So: zero automatic patching, and no mechanism by which a reboot could ever be signalled. That last
+# point is why this role and platform/kured must land together — kured watches
+# /var/run/reboot-required, and on these nodes today NOTHING WRITES IT. Deploying kured alone would
+# produce a daemon that is permanently, silently idle while looking perfectly healthy.
+#
+# ═════════════════════════════════════════════════════════════════════════════════════════════════
+# DECISION 1 — SECURITY ONLY, NOT FULL UPGRADES
+# ═════════════════════════════════════════════════════════════════════════════════════════════════
+# Measured `apt-cache policy` on k3s-prod-1, 2026-10-17, gives the four archives and their pins:
+#   trixie            o=Debian,            a=stable,            l=Debian          pin 500
+#   trixie-updates    o=Debian,            a=stable-updates,    l=Debian          pin 500
+#   trixie-security   o=Debian,            a=stable-security,   l=Debian-Security pin 500
+#   trixie-backports  o=Debian Backports,  a=stable-backports,  l=Debian Backports pin 100
+#
+# Only `Debian-Security` is allowed below. The reasoning, in order of weight:
+#
+#   * Security updates are the class where the cost of NOT applying them is unbounded and the
+#     change itself is minimal by Debian policy (targeted fixes against the frozen stable version,
+#     no feature churn). It is the one class worth applying with nobody watching.
+#   * A full `apt upgrade` on these nodes would also pull trixie point-release changes, which is a
+#     larger surface than anyone has reviewed for a 3-node etcd cluster, for no urgency.
+#   * `trixie-updates` is DELIBERATELY EXCLUDED, not overlooked. It carries non-security fixes that
+#     are too urgent to wait for a point release — useful, but it is the archive through which an
+#     unreviewed change would most plausibly arrive, and it is not what "unattended" should mean on
+#     a control-plane node. Add it consciously later if the operator wants it; the knob is below.
+#   * `trixie-backports` is EXCLUDED, AND THAT IS THE KERNEL ARGUMENT — see DECISION 2.
+#
+# ═════════════════════════════════════════════════════════════════════════════════════════════════
+# DECISION 2 — THE KERNEL IS EXCLUDED TWICE OVER. THIS IS THE HIGHEST-RISK PART OF THE CHANGE.
+# ═════════════════════════════════════════════════════════════════════════════════════════════════
+# The constraint: HP's R21 firmware on these EliteDesk 800 G5 minis publishes a corrupt EFI Memory
+# Attributes Table, and Linux < 6.13 does not validate it before reserving memory from its contents
+# — k3s-prod-2 twice booted with 2.6 GiB of 16 GB. Upstream 8fbe4c49c0cc (first in v6.13) is the
+# ONLY fix; there is no kernel cmdline workaround. So these nodes run the backports metapackage
+# `linux-image-amd64` (measured 2026-10-17: version 7.1.8-1~bpo13+1 installed and running on all
+# three, well above the floor). Full write-up: infra/PROD-PROVISIONING.md §0b, and the rationale
+# block in roles/node_baseline/defaults/main.yml:21-51.
+#
+# A NODE THAT COMES BACK FROM AN UNATTENDED UPGRADE ON A 6.12 KERNEL IS A NODE THAT MAY COME BACK
+# WITH A FRACTION OF ITS RAM. Note that 6.12 kernels are not hypothetical here — measured
+# 2026-10-17, `linux-image-6.12.107+deb13-amd64` and `linux-image-6.12.94+deb13-amd64` are BOTH
+# still installed on all three nodes, below the floor, bootable, and nothing holds them.
+#
+# Two independent mechanisms keep unattended-upgrades away from the kernel, and the role asserts
+# both rather than trusting either:
+#
+#   1. ORIGIN EXCLUSION (structural). The running kernel comes from `o=Debian Backports`, which is
+#      not in Allowed-Origins. u-u only ever considers packages from allowed origins, so it cannot
+#      see a backports kernel upgrade at all. This also means the backports kernel receives no
+#      automatic patching — accepted, and in any case backports carries no security guarantees
+#      (node_baseline's defaults state that cost plainly: "no security-tracker commitment, no DSAs").
+#   2. PACKAGE BLACKLIST (belt and braces). `linux-*` is blacklisted below, so even if the origin
+#      list is later widened — the most likely future edit to this file, and the one that would
+#      silently re-arm the hazard — the kernel is still untouchable. Without this, widening origins
+#      to include backports would hand an unattended process the EFI-memattr-critical package.
+#
+# WHAT THIS DOES *NOT* DO, AND MUST NOT BE READ AS DOING: it does not stop the kernel moving. The
+# operator still advances it deliberately with
+#   ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel --limit <host>
+# which is the only thing that maintains the backports source, the metapackage and the on-node
+# provenance stamp. node_baseline's own comment ("`ButAutomaticUpgrades: yes` ... means `apt upgrade`
+# keeps it current once installed") still holds for an operator-run apt; the measured pin priority of
+# 100 is exactly the NotAutomatic + ButAutomaticUpgrades signature that produces that behaviour.
+# Nothing here changes it. What changes is that a NEW unattended actor is forbidden from the kernel.
+#
+# NOT BLACKLISTED, BECAUSE THEY ARE NOT APT PACKAGES AT ALL — measured 2026-10-17: `dpkg-query` finds
+# no `k3s`, no `containerd` and no `containerd.io`. k3s is a single static binary at
+# /usr/local/bin/k3s installed by roles/k3s_server, and it bundles its own containerd and kubelet.
+# So the entire Kubernetes control plane and container runtime are OUTSIDE apt's reach and cannot be
+# upgraded by this role even in principle. That is worth knowing before worrying about it.
+unattended_upgrades_enabled: false
+
+# The packages this role installs. Each one is load-bearing; see the notes.
+unattended_upgrades_packages:
+  # The upgrader itself.
+  - unattended-upgrades
+  # NOTE: `update-notifier-common` was listed here and REMOVED (2026-10-18). It is an UBUNTU
+  # package and does not exist in Debian at all -- `apt-cache policy update-notifier-common` on
+  # trixie returns `Candidate: (none)`, so the play failed outright with "No package matching
+  # 'update-notifier-common' is available" on all three nodes. The sentinel writer is now provided
+  # by this role itself; see unattended_upgrades_sentinel_* below and tasks/main.yml.
+  # Restarts services still mapped to a deleted library after a security upgrade. Without it a
+  # security-only policy patches bytes on disk that no running process reads — see DECISION 3.
+  - needrestart
+
+# --- Allowed origins -----------------------------------------------------------------------------
+# Verbatim apt origin expressions, substituted into 50unattended-upgrades. The ${distro_*} tokens
+# here are UNATTENDED-UPGRADES' OWN placeholders, expanded by it at runtime — they are not Ansible
+# or Flux substitutions, which is why they are single-quoted and why this role's files never travel
+# through Flux envsubst.
+#
+# `l=Debian-Security` plus the `-security` codename is the form Debian's own shipped config uses, and
+# both halves were confirmed against the measured archive metadata above (l=Debian-Security,
+# n=trixie-security). Matching on the LABEL rather than the suite name is what makes this survive the
+# next Debian release.
+unattended_upgrades_allowed_origins:
+  - 'origin=Debian,codename=${distro_codename},label=Debian-Security'
+  - 'origin=Debian,codename=${distro_codename}-security,label=Debian-Security'
+# Set this to add 'origin=Debian,codename=${distro_codename}-updates,label=Debian' — a deliberate
+# widening, kept as a knob so that turning it on is a reviewable one-line change rather than an edit
+# to a heredoc. See DECISION 1 for why it is off.
+unattended_upgrades_include_updates: false
+
+# --- Package blacklist ---------------------------------------------------------------------------
+# Regular expressions matched against package names. See DECISION 2: `linux-` is the EFI-memattr
+# guard and must not be removed without re-reading PROD-PROVISIONING.md §0b.
+unattended_upgrades_blacklist:
+  # Every kernel package: linux-image-*, linux-headers-*, linux-image-amd64 (the metapackage that
+  # actually tracks backports), plus linux-kbuild / linux-libc-dev. Anchored at the start so it
+  # cannot accidentally match an unrelated package containing the substring.
+  - '^linux-'
+  # Microcode changes take effect only on reboot and can alter boot behaviour on the very firmware
+  # that is already misbehaving here. Operator-only, same argument as the kernel.
+  - '^intel-microcode$'
+  - '^amd64-microcode$'
+  # The bootloader. A grub upgrade rewrites boot configuration on a machine with a documented
+  # warm-reboot hang (k3s-prod-2, 2026-09-03). Never unattended.
+  - '^grub-'
+  - '^grub2-common$'
+
+# --- Reboot policy -------------------------------------------------------------------------------
+# FALSE, AND THIS IS THE DIVISION OF LABOUR WITH kured, NOT AN OVERSIGHT.
+# unattended-upgrades can reboot a node itself (Automatic-Reboot "true"). It must not here: it has
+# no notion of a cluster, so it would not cordon, would not drain, would not evict the 25 attached
+# Longhorn volumes cleanly, would not check whether another node is already down, and — on three
+# control-plane nodes where etcd quorum is 2 of 3 — three nodes each deciding independently to
+# reboot at their own `Automatic-Reboot-Time` is precisely how you lose the cluster.
+# u-u therefore only ever leaves the sentinel behind; kured owns every reboot, one node at a time,
+# under a cluster-wide lock. See platform/kured/helmrelease.yaml.
+unattended_upgrades_automatic_reboot: false
+
+# --- needrestart (DECISION 3) --------------------------------------------------------------------
+# See the role README for the full argument. Summary: 'a' (restart services automatically) is what
+# makes a security-only policy actually effective — a patched libssl on disk does nothing for a
+# process still mapping the old one — but k3s.service is EXCLUDED, because restarting the
+# Kubernetes control plane unattended at 06:00 is a worse outcome than a delayed library fix.
+# Measured 2026-10-17: the only matching unit on these nodes is `k3s.service` (no separate
+# containerd or kubelet unit exists; k3s bundles both), so this blacklist is complete today.
+unattended_upgrades_needrestart_mode: a
+unattended_upgrades_needrestart_blacklist:
+  - '^k3s'
+
+# --- Schedule ------------------------------------------------------------------------------------
+# apt's own systemd timers (apt-daily, apt-daily-upgrade) drive u-u; these set their windows.
+# 04:00 with a 1h randomised delay, so the upgrade lands INSIDE kured's 03:00-06:00 reboot window
+# (platform/kured/helmrelease.yaml) rather than hours away from it: if an upgrade ever does set the
+# sentinel, the reboot can be taken the same night instead of waiting ~24h. The randomised delay
+# keeps the three nodes from hitting the mirror simultaneously; it does NOT serialise anything, and
+# it does not need to — u-u never reboots, so three concurrent upgrades are harmless.
+unattended_upgrades_download_time: "03:30"
+unattended_upgrades_upgrade_time: "04:00"
+unattended_upgrades_randomised_delay: 60m
+
+
+# ── THE REBOOT SENTINEL, PROVIDED BY THIS ROLE ──────────────────────────────────────────────────
+# /var/run/reboot-required is the ONLY thing kured watches, and on Debian trixie **nothing creates
+# it**. Measured on all three nodes 2026-10-18:
+#   * `update-notifier-common` (which owns that hook on Ubuntu) has `Candidate: (none)` — the
+#     package does not exist in Debian. Installing it is what broke the first run of this play.
+#   * `/usr/share/update-notifier/notify-reboot-required` is absent.
+#   * NO `linux-image-*` postinst references `notify-reboot-required` — calling it is an Ubuntu
+#     patch, not Debian behaviour, so merely dropping a script at that path would never be invoked.
+#   * No hook under /etc/apt/apt.conf.d or /etc/kernel/ mentions reboot-required.
+# So the writer has to be ours. Without it kured is a permanent, INVISIBLE no-op: it stays 1/1
+# Running and reports kured_reboot_required=0 forever, which is the "failure mode is a green tick"
+# shape this repo keeps getting burned by.
+#
+# MECHANISM: a dpkg Post-Invoke hook runs a script that compares the RUNNING kernel against the
+# newest INSTALLED one and touches the sentinel when they differ. Deliberately kernel-only, which
+# matches the design: unattended-upgrades is security-only with `^linux-` blacklisted, so it will
+# essentially never set this — the sentinel appears when the OPERATOR deliberately runs the
+# node_baseline kernel play, and kured then automates only the mechanical reboot.
+#
+# It uses `linux-version` (from `linux-base`, a dependency of every linux-image package, so
+# guaranteed present — measured `linux-base 4.12.1` installed) rather than parsing `dpkg -l`
+# output, because kernel version ordering is not string ordering.
+#
+# /run is a tmpfs (measured), so the sentinel self-clears on boot: after a reboot the running
+# kernel IS the newest and the script writes nothing. No cleanup task is needed, and none exists.
+unattended_upgrades_sentinel_script: /usr/local/sbin/k3s-notify-reboot-required
+unattended_upgrades_sentinel_hook: /etc/apt/apt.conf.d/99-k3s-reboot-required

infra/ansible/roles/unattended_upgrades/handlers/main.yml

@@ -0,0 +1,29 @@
+---
+# Re-arm apt's periodic timers after a config or drop-in change.
+#
+# WHY A daemon_reload IS REQUIRED AND NOT OPTIONAL. The timer windows are set by drop-ins under
+# /etc/systemd/system/apt-daily{,-upgrade}.timer.d/. systemd does not re-read unit files on its own,
+# so until it is told to, the RUNNING manager keeps the shipped schedule and the new window is
+# ignored — the files look right in git and on disk, `systemctl cat` shows them, and the upgrade
+# still fires at the old time. That is a silent config no-op, which is the failure mode this whole
+# role is written against.
+#
+# `daemon_reload` re-parses unit files and re-runs generators. It starts and stops nothing, so it
+# cannot disturb k3s, containerd or Longhorn — the same reasoning node_baseline's "Reload systemd
+# unit files" handler records.
+#
+# RESTARTING THE TIMERS IS ALSO SAFE, and is not the same as running the upgrade. `*.timer` units
+# only hold the schedule; restarting one recomputes its next elapse and does NOT trigger its
+# `*.service`. With `Persistent=true` set in the drop-in, a window missed while the node was down is
+# caught up on the next boot rather than skipped.
+#
+# Both fire only when a task actually changed, so a re-run of the role is silent.
+- name: Restart unattended-upgrades timers
+  ansible.builtin.systemd_service:
+    name: "{{ item }}"
+    state: restarted
+    enabled: true
+    daemon_reload: true
+  loop:
+    - apt-daily.timer
+    - apt-daily-upgrade.timer

infra/ansible/roles/unattended_upgrades/tasks/main.yml

@@ -0,0 +1,369 @@
+---
+# roles/unattended_upgrades — automatic Debian security patching, kernel excluded, reboots delegated
+# to kured. Every decision and its reasoning lives in defaults/main.yml; this file is the mechanism.
+#
+# TAGS — following roles/node_baseline, every task carries one of `packages`, `config` or `verify`.
+# An untagged run does all three in that order. The split exists because `verify` is the half an
+# operator will want to re-run on its own to answer "is automatic patching actually working on this
+# box", without re-writing any configuration:
+#   ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify
+#
+# IT DOES NOT UPGRADE ANYTHING, AND IT DOES NOT REBOOT. This role installs and CONFIGURES the
+# upgrader; it never runs `apt upgrade` itself and never takes a reboot. Both of those are deliberate
+# and both follow roles/node_baseline/tasks/main.yml:61-67 ("IT DOES NOT REBOOT", after k3s-prod-2
+# hung on a warm reboot on 2026-09-03 and needed a physical power button). Applying this role is
+# therefore a SAFE operation on a live cluster at any time of day: the first unattended upgrade
+# happens at the next timer firing, and any reboot it implies is kured's to schedule inside the
+# 03:00-06:00 window.
+- name: Automatic security upgrades (unattended-upgrades; kernel excluded, reboots via kured)
+  when: unattended_upgrades_enabled | bool
+  tags: [packages, config, verify]
+  block:
+
+    # ── PREFLIGHT ───────────────────────────────────────────────────────────────────────────────
+    - name: Assert this is a Debian-family host
+      # The config file layout, the origin expressions and the package names below are all
+      # Debian-specific. On anything else this role would write files that are silently ignored,
+      # which is the failure mode this whole PR exists to avoid.
+      ansible.builtin.assert:
+        that:
+          - ansible_facts['os_family'] == 'Debian'
+        fail_msg: >-
+          REFUSING TO CONFIGURE unattended-upgrades on {{ inventory_hostname }}: os_family is
+          '{{ ansible_facts["os_family"] }}', not 'Debian'. The allowed-origin expressions and
+          config paths in this role are Debian-specific.
+      tags: [packages, config, verify]
+
+    # ── PACKAGES ────────────────────────────────────────────────────────────────────────────────
+    - name: Install unattended-upgrades and needrestart
+      # The reboot-sentinel writer is NOT a package here. On Debian nothing ships one -- see
+      # defaults/main.yml -- so this role installs its own script and dpkg hook, two tasks below.
+      ansible.builtin.apt:
+        name: "{{ unattended_upgrades_packages }}"
+        state: present
+        update_cache: true
+        cache_valid_time: 3600
+      tags: [packages]
+
+    # ── THE REBOOT SENTINEL WRITER (ours, because Debian ships none) ────────────────────────────
+    # Full rationale and the four measurements behind it: defaults/main.yml. Short version: kured
+    # watches /var/run/reboot-required and nothing on Debian creates it, so without these two files
+    # the entire reboot half is an invisible no-op.
+    - name: Install the reboot-sentinel writer
+      ansible.builtin.copy:
+        dest: "{{ unattended_upgrades_sentinel_script }}"
+        owner: root
+        group: root
+        mode: "0755"
+        content: |
+          #!/bin/sh
+          # MANAGED BY ANSIBLE -- infra/ansible/roles/unattended_upgrades. Local edits are reverted.
+          #
+          # Touch /var/run/reboot-required when the RUNNING kernel is not the newest INSTALLED one.
+          # kured watches that path; see the role's defaults/main.yml for why this is ours to ship.
+          #
+          # `linux-version` comes from linux-base (a dependency of every linux-image package), and
+          # is used rather than sorting `dpkg -l` output because kernel version ordering is not
+          # string ordering. If it is somehow missing, exit 0 and write nothing: a dpkg hook must
+          # never fail an apt transaction, and a missing sentinel is caught by the verify assert.
+          set -eu
+          command -v linux-version >/dev/null 2>&1 || exit 0
+
+          running="$(uname -r)"
+          newest="$(linux-version list 2>/dev/null | linux-version sort --reverse 2>/dev/null | head -n1)"
+
+          # No newest (nothing installed we can read) -> nothing to say.
+          [ -n "$newest" ] || exit 0
+
+          if [ "$running" != "$newest" ]; then
+            touch /var/run/reboot-required 2>/dev/null || exit 0
+            # kured only needs the file, but the .pkgs companion is what an operator reads to find
+            # out WHY a node is waiting, so write a reason rather than an empty marker.
+            printf 'linux-image-%s\n' "$newest" > /var/run/reboot-required.pkgs 2>/dev/null || true
+          fi
+          exit 0
+      tags: [packages, config]
+
+    - name: Install the dpkg hook that runs the sentinel writer
+      # Post-Invoke, so it runs after dpkg has finished and a newly installed kernel is already
+      # visible to `linux-version list`. `|| true` so a failure here can never abort an apt
+      # transaction -- an unwritten sentinel is a missed reboot, but a broken apt is a broken node.
+      ansible.builtin.copy:
+        dest: "{{ unattended_upgrades_sentinel_hook }}"
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          // MANAGED BY ANSIBLE -- infra/ansible/roles/unattended_upgrades. Local edits reverted.
+          // Writes /var/run/reboot-required when the running kernel is not the newest installed.
+          DPkg::Post-Invoke { "{{ unattended_upgrades_sentinel_script }} || true"; };
+      tags: [packages, config]
+
+    # ── CONFIG ──────────────────────────────────────────────────────────────────────────────────
+    # Written as a whole file rather than patched in place, for the reason node_baseline gives for
+    # its own apt source: an idempotent full-file write with a managed-by header is reviewable and
+    # re-runnable, whereas line-editing a conffile fights dpkg on every package upgrade.
+    #
+    # 52- RATHER THAN 50-: Debian ships /etc/apt/apt.conf.d/50unattended-upgrades as a CONFFILE of
+    # the unattended-upgrades package. Overwriting it would make every future apt upgrade of that
+    # package prompt about a locally-modified conffile (or, under the non-interactive defaults a
+    # scripted run uses, silently keep one side and discard the other). apt reads the directory in
+    # lexical order and later files win, so a separate 52- file overrides the shipped defaults
+    # without ever touching them, and `apt-config dump` in the verify block below proves which value
+    # actually won rather than assuming this ordering holds.
+    - name: Configure unattended-upgrades (52-k3s-unattended-upgrades)
+      ansible.builtin.copy:
+        dest: /etc/apt/apt.conf.d/52-k3s-unattended-upgrades
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          // MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades. Local edits are reverted
+          // on the next run. Rationale for every value: that role's defaults/main.yml.
+          //
+          // Overrides the shipped 50unattended-upgrades by lexical ordering (52 > 50). The shipped
+          // file is a dpkg conffile and is deliberately left untouched.
+
+          // SECURITY ONLY. Deliberately excludes trixie-updates (non-security fixes) and
+          // trixie-backports (o=Debian Backports) -- and excluding backports is what makes it
+          // structurally impossible for this upgrader to move the EFI-memattr-critical kernel,
+          // which is installed from there. See defaults/main.yml DECISION 2.
+          Unattended-Upgrade::Origins-Pattern {
+          {% for origin in unattended_upgrades_allowed_origins %}
+                  "{{ origin }}";
+          {% endfor %}
+          {% if unattended_upgrades_include_updates | bool %}
+                  "origin=Debian,codename=${distro_codename}-updates,label=Debian";
+          {% endif %}
+          };
+
+          // NEVER AUTOMATICALLY UPGRADED, whatever the origin list later says. `^linux-` is the
+          // EFI-memattr guard: these HP minis need kernel >= 6.13 or they can boot with a fraction
+          // of their RAM, and 6.12 kernels are still installed and bootable on all three nodes.
+          // Do not remove without re-reading infra/PROD-PROVISIONING.md §0b.
+          Unattended-Upgrade::Package-Blacklist {
+          {% for pkg in unattended_upgrades_blacklist %}
+                  "{{ pkg }}";
+          {% endfor %}
+          };
+
+          // REBOOTS BELONG TO kured (platform/kured), NOT TO apt. All three nodes are
+          // control-plane + etcd with quorum 2 of 3, so three hosts independently deciding to
+          // reboot at their own Automatic-Reboot-Time is how the cluster is lost. u-u leaves
+          // /var/run/reboot-required behind; kured takes a cluster-wide lock and reboots exactly
+          // one node at a time, after cordon + drain, inside 03:00-06:00, gated on Longhorn volume
+          // health.
+          Unattended-Upgrade::Automatic-Reboot "{{ unattended_upgrades_automatic_reboot | bool | lower }}";
+          Unattended-Upgrade::Automatic-Reboot-WithUsers "false";
+
+          // Keep the node's disk from filling with superseded .debs, and keep a log an operator can
+          // read after the fact (/var/log/unattended-upgrades/). MinAge guards against removing a
+          // package still needed by a rollback.
+          Unattended-Upgrade::Remove-Unused-Kernel-Packages "false";
+          Unattended-Upgrade::Remove-New-Unused-Dependencies "true";
+          Unattended-Upgrade::Remove-Unused-Dependencies "false";
+          Unattended-Upgrade::AutoFixInterruptedDpkg "true";
+          Unattended-Upgrade::MinimalSteps "true";
+          Unattended-Upgrade::SyslogEnable "true";
+
+          // Remove-Unused-Kernel-Packages is FALSE on purpose, against the Debian default. These
+          // nodes deliberately carry more than one kernel: the backports kernel they boot and the
+          // distro kernels that are the documented rollback path (node_baseline writes the rollback
+          // procedure onto the node itself). An unattended process pruning kernels could remove the
+          // fallback an operator needs at 3am in front of a box that will not boot.
+      notify: Restart unattended-upgrades timers
+      tags: [config]
+
+    - name: Enable the apt periodic timers that actually run unattended-upgrades
+      # Installing the package is NOT sufficient to make it run on a schedule: without these
+      # counters apt's daily units do nothing, and the whole mechanism is inert while looking
+      # installed. `1` means "every day".
+      ansible.builtin.copy:
+        dest: /etc/apt/apt.conf.d/20auto-upgrades
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          // MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades.
+          // Without these four counters the apt-daily timers fire and do nothing, so
+          // unattended-upgrades would be installed, configured, and never once invoked.
+          APT::Periodic::Update-Package-Lists "1";
+          APT::Periodic::Download-Upgradeable-Packages "1";
+          APT::Periodic::Unattended-Upgrade "1";
+          APT::Periodic::AutocleanInterval "7";
+      tags: [config]
+
+    - name: Ensure the systemd timer drop-in directories exist
+      # `copy` does not create parent directories, and these two .timer.d/ dirs do not exist on a
+      # stock Debian node — without this task the next task fails with "Destination directory does
+      # not exist" on a first run.
+      ansible.builtin.file:
+        path: "/etc/systemd/system/{{ item }}.timer.d"
+        state: directory
+        owner: root
+        group: root
+        mode: "0755"
+      loop:
+        - apt-daily
+        - apt-daily-upgrade
+      tags: [config]
+
+    - name: Pin the apt timer windows into kured's reboot window
+      # Overriding the shipped timers rather than editing them: a drop-in survives package upgrades.
+      # OnCalendar is cleared first (`OnCalendar=` with no value) because systemd ACCUMULATES
+      # OnCalendar entries across drop-ins — omit the reset and the unit keeps its shipped schedule
+      # IN ADDITION to ours, which would run the upgrade at both times and look like it worked.
+      ansible.builtin.copy:
+        dest: "/etc/systemd/system/{{ item.unit }}.timer.d/override.conf"
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          # MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades.
+          # Lands the upgrade inside kured's 03:00-06:00 reboot window so a sentinel written by an
+          # upgrade can be acted on the same night rather than ~24h later.
+          [Timer]
+          OnCalendar=
+          OnCalendar={{ item.time }}
+          RandomizedDelaySec={{ unattended_upgrades_randomised_delay }}
+          Persistent=true
+        force: true
+      loop:
+        - { unit: apt-daily, time: "{{ unattended_upgrades_download_time }}" }
+        - { unit: apt-daily-upgrade, time: "{{ unattended_upgrades_upgrade_time }}" }
+      loop_control:
+        label: "{{ item.unit }} at {{ item.time }}"
+      notify: Restart unattended-upgrades timers
+      tags: [config]
+
+    - name: Configure needrestart (restart services, but never k3s)
+      # See defaults/main.yml DECISION 3. Auto-restart is what makes a security-only policy real --
+      # a patched library on disk does nothing for a process still mapping the deleted one -- but
+      # k3s.service is excluded because restarting the control plane unattended is worse than a
+      # delayed library fix. `$nrconf{restart}` is Perl config read by needrestart itself.
+      ansible.builtin.copy:
+        dest: /etc/needrestart/conf.d/50-k3s.conf
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          # MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades.
+          # Restart mode: '{{ unattended_upgrades_needrestart_mode }}'  (a = automatically, l = list only, i = interactive)
+          $nrconf{restart} = '{{ unattended_upgrades_needrestart_mode }}';
+          # Units matching these patterns are NEVER restarted automatically. k3s bundles containerd
+          # and the kubelet, so this one entry covers the entire control plane and runtime on these
+          # nodes (measured 2026-10-17: k3s.service is the only matching unit). A library fix that
+          # needs k3s restarted therefore waits for the reboot kured will take.
+          $nrconf{blacklist_rc} = [
+          {% for pat in unattended_upgrades_needrestart_blacklist %}
+              qr({{ pat }}),
+          {% endfor %}
+          ];
+      tags: [config]
+
+    # ── VERIFY ──────────────────────────────────────────────────────────────────────────────────
+    # These tasks exist because every component of this mechanism fails SILENTLY. An installed
+    # package, a written config file and a green play are all compatible with nothing ever being
+    # upgraded and no reboot ever being signalled.
+
+    - name: Ask apt-config what the effective settings actually are
+      # `apt-config dump` is apt's OWN parse of the whole /etc/apt/apt.conf.d/ directory, so it
+      # proves what WON after lexical ordering rather than what this role wrote. If the shipped
+      # 50unattended-upgrades were ever renamed above 52-, this is the task that would catch it.
+      ansible.builtin.command: apt-config dump
+      register: unattended_upgrades_aptconfig
+      changed_when: false
+      check_mode: false
+      tags: [verify]
+
+    - name: Assert the kernel is blacklisted and automatic reboots are off
+      # The two properties that must never silently regress. Asserted against apt's own view.
+      ansible.builtin.assert:
+        that:
+          # apt-config renders each list entry as `Unattended-Upgrade::Package-Blacklist:: "^linux-";`
+          # so the anchored pattern itself is the thing to look for. Checking for the bare string
+          # `linux-` would pass on almost any dump and prove nothing.
+          - "'\"^linux-\"' in unattended_upgrades_aptconfig.stdout"
+          - "'Unattended-Upgrade::Automatic-Reboot \"false\"' in unattended_upgrades_aptconfig.stdout"
+          # The origin list must be non-empty, or u-u runs nightly and upgrades nothing — installed,
+          # scheduled, green, and inert.
+          - "'Debian-Security' in unattended_upgrades_aptconfig.stdout"
+        fail_msg: >-
+          EFFECTIVE apt CONFIG ON {{ inventory_hostname }} IS NOT WHAT THIS ROLE INTENDED.
+          Either the `^linux-` kernel blacklist or `Automatic-Reboot "false"` is not winning.
+          Both are safety properties, not preferences: without the blacklist an unattended process
+          can move a kernel these HP minis must keep >= 6.13 (PROD-PROVISIONING.md §0b), and
+          without Automatic-Reboot=false three control-plane nodes can each reboot themselves
+          independently and take etcd below quorum. Inspect with `apt-config dump | grep -i
+          unattended` and check for a file sorting after 52- in /etc/apt/apt.conf.d/.
+        success_msg: >-
+          Effective apt config confirmed: kernel blacklisted, automatic reboot disabled.
+      tags: [verify]
+
+    # THE SENTINEL IS THE SINGLE POINT OF FAILURE FOR THE ENTIRE REBOOT HALF OF THIS DESIGN.
+    # This was originally a `find` over /etc/kernel/postinst.d and /etc/apt/apt.conf.d, on the
+    # theory that the writer came from a package and the role could only discover whatever that
+    # package happened to ship. That theory was wrong: no such package exists on Debian (see
+    # defaults/main.yml), so the writer is now OURS and lives at two known paths. Checking two
+    # exact paths is strictly stronger than a glob -- a glob would have matched Debian's own
+    # unattended-upgrades apt conffile and passed while no sentinel writer existed at all.
+    - name: Stat the sentinel writer script
+      ansible.builtin.stat:
+        path: "{{ unattended_upgrades_sentinel_script }}"
+      register: unattended_upgrades_sentinel_script_stat
+      changed_when: false
+      tags: [verify]
+
+    - name: Stat the sentinel dpkg hook
+      ansible.builtin.stat:
+        path: "{{ unattended_upgrades_sentinel_hook }}"
+      register: unattended_upgrades_sentinel_hook_stat
+      changed_when: false
+      tags: [verify]
+
+    - name: Assert something on this node can write the reboot sentinel
+      # If this fails, kured is a no-op: it watches /var/run/reboot-required forever, the file is
+      # never created, no reboot is ever taken, and NOTHING GOES RED -- the DaemonSet stays 1/1
+      # Running and `kured_reboot_required` reports a steady 0 on every node. That is the exact
+      # shape of failure (a check whose failure mode is a green tick) this repo has been burned by
+      # before, so it is an assert and not a debug message.
+      ansible.builtin.assert:
+        that:
+          - unattended_upgrades_sentinel_script_stat.stat.exists
+          - unattended_upgrades_sentinel_script_stat.stat.executable | default(false)
+          - unattended_upgrades_sentinel_hook_stat.stat.exists
+        fail_msg: >-
+          NOTHING ON {{ inventory_hostname }} CAN CREATE /var/run/reboot-required.
+          {{ unattended_upgrades_sentinel_script }} (executable) and
+          {{ unattended_upgrades_sentinel_hook }} must both exist -- this role ships both, because
+          Debian provides no sentinel writer of its own (update-notifier-common is an UBUNTU
+          package and is not installable here). kured watches that path and reboots only when it
+          exists, so with no writer the reboot automation is permanently and INVISIBLY idle --
+          kured stays 1/1 Running and reports kured_reboot_required=0 on every node forever.
+          Re-run this role without `--tags verify` to install them.
+        success_msg: >-
+          Reboot sentinel writer present ({{ unattended_upgrades_sentinel_script }} + dpkg hook)
+          -- a kernel change will signal kured on the next apt transaction.
+      tags: [verify]
+
+    - name: Report the pending-reboot state of this node
+      # Read-only. Tells the operator whether this node is currently waiting on kured for a reboot,
+      # which is the question `--tags verify` is usually being run to answer.
+      ansible.builtin.stat:
+        path: /var/run/reboot-required
+      register: unattended_upgrades_reboot_required
+      changed_when: false
+      tags: [verify]
+
+    - name: Summary
+      ansible.builtin.debug:
+        msg: >-
+          {{ inventory_hostname }}: security-only unattended upgrades ACTIVE
+          (upgrade timer {{ unattended_upgrades_upgrade_time }} +/-{{ unattended_upgrades_randomised_delay }}),
+          kernel and microcode blacklisted, automatic reboot DISABLED.
+          Reboot currently pending: {{ unattended_upgrades_reboot_required.stat.exists }}.
+          {% if unattended_upgrades_reboot_required.stat.exists %}
+          This node is waiting for kured to reboot it in the next 03:00-06:00 window; it will be
+          cordoned and drained first, and no other node will be rebooted at the same time.
+          {% endif %}
+      tags: [verify]

infra/devcontainer/Containerfile

@@ -32,9 +32,23 @@ ARG TARGETARCH=amd64
 # Distro packages: libvirt client libs (harmless disk-image utilities), ansible, ssh,
 # disk-image + cloud-init seed tooling, python3-pip (for the pip-installed test tools),
 # diffutils (provides `diff`, used by L1 golden render checks), and common utilities.
+#
+# wireguard-tools + qrencode serve the REMOTE-ACCESS VPN (platform/vpn-ingress):
+#   * wireguard-tools gives `wg genkey` / `wg pubkey`, needed to mint the server key and to derive
+#     every client profile's `[Peer] PublicKey` (infra/vpn-ingress/new-peer.sh).
+#   * qrencode renders a client profile as a QR code, which is how a PHONE gets enrolled.
+# They are here rather than on the operator's host because this image is THE toolchain
+# (CLAUDE.md §5; README "Don't install the toolchain on the host"). That rule was briefly broken
+# by a runbook that told the operator to run `wg` on the host and `sops` in here, and then split
+# every step across the boundary to work around it — a procedure nobody should have to follow, and
+# one that cannot be reproduced in CI. Both tools in one image means `sops -d … | wg pubkey` is
+# just a pipeline again.
+# Unpinned on purpose, like every other dnf package here: the Fedora base is what is pinned
+# (FEDORA_VERSION), and these are not gate tools whose version changes what green means.
 RUN dnf -y install --setopt=install_weak_deps=False \
       libvirt-client qemu-img cloud-utils genisoimage openssh-clients \
       ansible-core python3-libvirt python3-pip diffutils \
+      wireguard-tools qrencode \
       git make jq unzip tar gzip xz which ca-certificates curl \
  && dnf clean all
 

infra/devcontainer/README.md

@@ -14,6 +14,7 @@ baseline, and the *same* binaries CI uses.
 | Provisioning | OpenTofu (`bpg/proxmox` provider), `ansible-core`, `openssh-clients` |
 | GitOps / cluster | `flux`, `kubectl`, `helm` |
 | Secrets | `sops`, `age` + `age-keygen` |
+| VPN (`platform/vpn-ingress`) | `wg` (wireguard-tools) — mint the server key, derive client public keys; `qrencode` — render a client profile as a phone-scannable QR |
 | **CI gate** | `yamllint`, `check-jsonschema`, `kyverno` CLI, `kyverno-json`, `chainsaw`, `yq`, `diffutils`, `make` |
 | Misc | `qemu-img`, `cloud-utils`/`genisoimage`, `libvirt-client` (disk-image utilities only — leftovers from the libvirt era, harmless) |
 

infra/vpn-ingress/new-peer.sh

@@ -2,10 +2,24 @@
 # =================================================================================================
 # new-peer.sh — enrol one device on the remote-access VPN (platform/vpn-ingress).
 #
-# Generates a WireGuard keypair ON THIS MACHINE, renders the device's client profile, optionally
-# prints a QR code for a phone, and prints the one `[Peer]` stanza to paste into
-# platform/vpn-ingress/peers.yaml. The private half NEVER leaves this machine: it is not sent to the
-# cluster, not committed, and not written to any log.
+# Generates a WireGuard keypair, renders the device's client profile, prints a QR code for a phone,
+# and prints the one `[Peer]` stanza to paste into platform/vpn-ingress/peers.conf. The device's
+# private half is written to ONE 0600 file and nowhere else: not sent to the cluster, not committed,
+# never logged.
+#
+# RUN IT IN THE DEV CONTAINER — that is where the toolchain lives (CLAUDE.md §5):
+#
+#     ./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh pixel-8 2'
+#
+# It needs `wg` (wireguard-tools), `qrencode` and `sops`, all three baked into the image. It will
+# run on a host that happens to have them too, but the container is the supported way and the only
+# one that is reproducible.
+#
+# THE SERVER'S PUBLIC KEY IS DERIVED FROM THE ENCRYPTED FILE, not from a copy you keep lying around:
+# `sops -d` the private half out of platform/vpn-ingress/wg-server-key.sops.yaml and pipe it
+# straight into `wg pubkey`. The plaintext never touches disk and never reaches a terminal. (This
+# is why both tools being in ONE image matters — see the Containerfile comment.) Override with
+# WG_SERVER_PUBKEY or WG_SERVER_KEY_FILE if you are working off-tree.
 #
 # WHY A SCRIPT AND NOT A RUNBOOK STEP. Three of these fields are easy to get wrong in ways that are
 # either silently insecure or silently broken, and all three are mechanical:
@@ -25,24 +39,30 @@
 #   <last-octet>    2-254, the device's address inside WG_CLIENT_CIDR. Must be unused.
 #   --split         split tunnel: route ONLY the LAN and the tunnel subnet. Default is a FULL
 #                   tunnel (0.0.0.0/0) so the device also egresses the house WAN.
-#   --out DIR       where to write the profile (default: a 0700 dir under $TMPDIR).
+#   --out DIR       where to write the profile. Default: `.cache/wg-peers/` in the repo, which is
+#                   GITIGNORED (`.gitignore` `.cache/`) and, unlike /tmp inside the container,
+#                   survives the container exiting — the repo is its only persistent read-write
+#                   mount, so a profile written anywhere else is gone before you can move it to a
+#                   desktop. The script refuses any destination git does not ignore.
 #
 # ENVIRONMENT
-#   WG_SERVER_KEY_FILE  server PRIVATE key (default ~/.config/wireguard-home/server.key); its public
-#                       half is derived from it. Alternatively set WG_SERVER_PUBKEY directly.
+#   WG_SERVER_PUBKEY    the server's public key, if you want to skip the SOPS decrypt entirely.
+#   WG_SERVER_KEY_FILE  a local copy of the server PRIVATE key to derive the public half from,
+#                       instead of reading the SOPS file.
+#   WG_SERVER_SOPS_FILE default platform/vpn-ingress/wg-server-key.sops.yaml
 #   WG_ENDPOINT_HOST    default wan.home.keithjackson.dev
 #   WG_CLIENT_CIDR      default 10.90.0.0/24   (must match clusters/base/cluster-vars.yaml)
 #   WG_LAN_CIDR         default 192.168.2.0/24 (prod LAN_CIDR)
 #   WG_DNS              default 192.168.2.23   (prod PIHOLE_DNS_VIP)
 #
-# REQUIRES: wireguard-tools (`wg`). `qrencode` is optional and only used for the QR code.
+# REQUIRES: `wg`, `qrencode`, `sops` — all three in the dev container.
 # =================================================================================================
 set -euo pipefail
 
 die() { printf 'new-peer.sh: %s\n' "$*" >&2; exit 1; }
 
 REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
-PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.yaml"
+PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.conf"
 
 # --- defaults, each mirroring a value that is DECLARED elsewhere in this repo -------------------
 # These are duplicated here rather than parsed out of the YAML on purpose: this script is meant to
@@ -51,7 +71,7 @@ PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.yaml"
 : "${WG_CLIENT_CIDR:=10.90.0.0/24}"                 # clusters/base/cluster-vars.yaml
 : "${WG_LAN_CIDR:=192.168.2.0/24}"                  # clusters/prod LAN_CIDR
 : "${WG_DNS:=192.168.2.23}"                         # clusters/prod PIHOLE_DNS_VIP
-: "${WG_SERVER_KEY_FILE:=${HOME}/.config/wireguard-home/server.key}"
+: "${WG_SERVER_SOPS_FILE:=${REPO_ROOT}/platform/vpn-ingress/wg-server-key.sops.yaml}"
 # wan.<base-domain> is an UNPROXIED A record that platform/externaldns/ddns.yaml keeps pointed at
 # the current WAN IP (checked every 5 min). Use it, never a literal address: this house has a
 # DYNAMIC WAN IP, and a profile pinned to today's address stops working without warning and with no
@@ -83,14 +103,28 @@ printf '%s' "$NAME" | grep -Eq '^[a-z0-9][a-z0-9-]{0,30}$' \
 printf '%s' "$OCTET" | grep -Eq '^[0-9]+$' || die "octet must be a number: got '$OCTET'"
 [ "$OCTET" -ge 2 ] && [ "$OCTET" -le 254 ] || die "octet must be 2-254 (.1 is the server): got '$OCTET'"
 
-command -v wg >/dev/null || die "wireguard-tools not installed ('wg' not on PATH). Debian/Ubuntu: apt install wireguard-tools"
+command -v wg >/dev/null || die "'wg' not on PATH. Run this inside the dev container:
+  ./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh $NAME $OCTET'"
 
 # --- the server's public key --------------------------------------------------------------------
+# Order: an explicit pubkey, then a local private-key file, then the SOPS file in the tree. The
+# last is the normal path and needs no local copy of anything.
 if [ -z "${WG_SERVER_PUBKEY:-}" ]; then
-  [ -r "$WG_SERVER_KEY_FILE" ] || die "cannot read server key '$WG_SERVER_KEY_FILE'.
-  Either point WG_SERVER_KEY_FILE at it, or set WG_SERVER_PUBKEY to the public half.
-  If the server key does not exist yet, see runbooks/vpn-remote-access.md step 1."
-  WG_SERVER_PUBKEY="$(tr -d ' \r\n' < "$WG_SERVER_KEY_FILE" | wg pubkey)"
+  if [ -n "${WG_SERVER_KEY_FILE:-}" ]; then
+    [ -r "$WG_SERVER_KEY_FILE" ] || die "cannot read WG_SERVER_KEY_FILE '$WG_SERVER_KEY_FILE'"
+    WG_SERVER_PUBKEY="$(tr -d ' \r\n' < "$WG_SERVER_KEY_FILE" | wg pubkey)"
+  else
+    command -v sops >/dev/null || die "'sops' not on PATH. Run this inside the dev container (see --help)."
+    [ -r "$WG_SERVER_SOPS_FILE" ] || die "cannot read '$WG_SERVER_SOPS_FILE'"
+    # The decrypted private key goes straight into `wg pubkey` — never to a file, never to a tty.
+    # `wg pubkey` fails on the committed REPLACE-ME placeholder, which is the useful error here.
+    WG_SERVER_PUBKEY="$(sops -d --extract '["stringData"]["privatekey"]' "$WG_SERVER_SOPS_FILE" \
+                          | tr -d ' \r\n' | wg pubkey 2>/dev/null)" || WG_SERVER_PUBKEY=""
+    [ -n "$WG_SERVER_PUBKEY" ] || die "could not derive the server public key from
+  $WG_SERVER_SOPS_FILE
+  Either it still holds the REPLACE-ME placeholder (see runbooks/vpn-remote-access.md step 2),
+  or this shell cannot decrypt it (the age keyring is mounted at ~/.config/sops in the container)."
+  fi
 fi
 
 CLIENT_BASE="${WG_CLIENT_CIDR%/*}"; CLIENT_BASE="${CLIENT_BASE%.*}"
@@ -99,22 +133,41 @@ CLIENT_ADDR="${CLIENT_BASE}.${OCTET}"
 # --- refuse a collision, because the failure is otherwise invisible -----------------------------
 # Two devices sharing an address does not error anywhere: the server routes that address to
 # whichever peer handshook most recently, so BOTH devices appear to work intermittently.
-if [ -r "$PEERS_FILE" ] && grep -q "AllowedIPs *= *${CLIENT_ADDR}/32" "$PEERS_FILE"; then
-  die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.yaml. Pick another octet."
+# COMMENT LINES ARE NOT ALLOCATIONS. This used to grep the whole file and so matched the
+# COMMENTED-OUT EXAMPLE STANZA in the register, which made `.2` — the first address anyone would
+# ask for — permanently un-allocatable with a message claiming it was already taken. Strip
+# comments first: a real stanza's `AllowedIPs` line is never commented, while both the template
+# and every peer's `# <name> — <owner>` label are.
+# FAIL if the register is missing rather than skipping the check. The path was silently wrong once
+# (it still said peers.yaml after the file became peers.conf), and because the old guard was
+# `[ -r "$PEERS_FILE" ] && ...` the only symptom would have been duplicate detection quietly
+# never firing again — the exact silent-skip shape this repo treats as a defect.
+[ -r "$PEERS_FILE" ] || die "cannot read the peer register '$PEERS_FILE'.
+  Duplicate-address detection cannot run, so this script refuses to issue a profile."
+if grep -vE '^[[:space:]]*#' "$PEERS_FILE" | grep -q "AllowedIPs *= *${CLIENT_ADDR}/32"; then
+  die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.conf. Pick another octet."
 fi
-if [ -r "$PEERS_FILE" ] && grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then
-  printf 'new-peer.sh: WARNING a peer comment already mentions "%s" in peers.yaml.\n' "$NAME" >&2
+if grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then
+  printf 'new-peer.sh: WARNING a peer comment already mentions "%s" in peers.conf.\n' "$NAME" >&2
 fi
 
 # --- output location: 0700, and never inside the repo -------------------------------------------
-if [ -z "$OUT_DIR" ]; then
-  OUT_DIR="$(mktemp -d "${TMPDIR:-/tmp}/wg-peer-XXXXXX")"
-else
-  mkdir -p "$OUT_DIR"
-fi
+[ -n "$OUT_DIR" ] || OUT_DIR="${REPO_ROOT}/.cache/wg-peers"
+mkdir -p "$OUT_DIR"
 chmod 700 "$OUT_DIR"
-case "$(cd "$OUT_DIR" && pwd)" in
-  "$REPO_ROOT"|"$REPO_ROOT"/*) die "refusing to write a private key inside the git checkout ($OUT_DIR)" ;;
+OUT_ABS="$(cd "$OUT_DIR" && pwd)"
+# A profile holds a private key, so the destination must not be committable. Inside the repo that
+# means git must IGNORE it — test the real property rather than a proxy, because the repo is the
+# container's only persistent read-write mount and so the only sane place to write. Outside the
+# repo, anywhere goes.
+case "$OUT_ABS" in
+  "$REPO_ROOT"|"$REPO_ROOT"/*)
+    if ! git -C "$REPO_ROOT" check-ignore -q "$OUT_ABS/probe.conf" 2>/dev/null; then
+      die "refusing to write a private key to '$OUT_ABS': it is inside the git checkout and NOT
+  gitignored, so the profile would be committable. Use the default (.cache/wg-peers) or add an
+  ignore rule first. Verify with: git check-ignore -v <path>"
+    fi
+    ;;
 esac
 PROFILE="${OUT_DIR}/${NAME}.conf"
 
@@ -162,7 +215,7 @@ cat <<EOF
   Profile     : ${PROFILE}   (0600; contains the private key)
 
   ------------------------------------------------------------------------------------------
-  STEP 1 — add this to platform/vpn-ingress/peers.yaml, under data.peers.conf, then PR it:
+  STEP 1 — append this to platform/vpn-ingress/peers.conf, then PR it:
 
     # ${NAME} — <owner>, enrolled $(date -u +%Y-%m-%d)
     [Peer]
@@ -181,7 +234,7 @@ cat <<EOF
   STEP 3 — delete the profile from this machine once the device has it:
     shred -u ${PROFILE} 2>/dev/null || rm -f ${PROFILE}
 
-  The tunnel will not work until the peers.yaml PR is MERGED and Flux has rolled the DaemonSet
+  The tunnel will not work until the peers.conf PR is MERGED and Flux has rolled the DaemonSet
   (~1 min). Until then the device handshakes and gets silence, exactly like an unknown key.
   ------------------------------------------------------------------------------------------
 EOF
@@ -190,6 +243,8 @@ if command -v qrencode >/dev/null; then
   printf '  QR code (this IS the private key — your terminal scrollback now holds a credential):\n\n'
   qrencode -t ansiutf8 < "$PROFILE"
 else
-  printf '  qrencode not installed, so no QR code was printed (Debian/Ubuntu: apt install qrencode).\n'
-  printf '  Transfer %s to the device by another route instead.\n' "$PROFILE"
+  # Should not happen in the dev container: qrencode is baked into the image. Kept as a real
+  # fallback for a host run, rather than failing after the profile has already been written.
+  printf '  qrencode not on PATH, so no QR code was printed. It is in the dev container —\n'
+  printf '  rerun there, or transfer %s to the device by another route.\n' "$PROFILE"
 fi

platform/kured/README.md

@@ -0,0 +1,127 @@
+# platform/kured
+
+**kured (Kubernetes Reboot Daemon)** — coordinated, one-node-at-a-time OS reboots of the three prod
+control-plane nodes.
+
+- **Flux tier:** `clusters/base/kured.yaml` (its own Kustomization, `dependsOn: [platform]`,
+  `wait: true`)
+- **Chart:** `kured` 6.1.0 → appVersion 1.23.0, from `https://kubereboot.github.io/charts`
+- **The other half:** `infra/ansible/roles/unattended_upgrades` — the package upgrades. **Neither
+  half is useful alone.**
+- **Runbook:** `/runbooks/node-reboot-automation`
+- **Every tuned value and its reasoning:** `helmrelease.yaml`. That file is the argument; this one is
+  the orientation.
+- **Design record:** `decisions/2026-10-17-automated-upgrades.md`
+
+## What this component owns
+
+The *mechanics* of a reboot: take a cluster-wide lock, cordon, drain, reboot, uncordon, hold the
+lock a while longer. Nothing else.
+
+**It does not decide whether to reboot.** kured reboots a node if and only if
+`/var/run/reboot-required` exists there. On these nodes that file is written by a deliberate
+operator kernel upgrade (`playbooks/node-baseline.yml --tags kernel`) — unattended-upgrades is
+security-only with the kernel blacklisted, so it essentially never produces one. **The human gate is
+the sentinel, not the daemon.**
+
+## The problem this fixes
+
+Rebooting these three nodes was a manual, serialized, supervised procedure, and
+`roles/node_baseline` has to end its kernel run by shouting the rules at the operator: *"Reboot prod
+nodes ONE AT A TIME, verify each returns before touching the next, and NEVER node 2 first"*. That is
+a correct procedure and an error-prone one, and nothing enforced it.
+
+What it has to get right, measured on prod 2026-10-17:
+
+- **etcd quorum is 2 of 3** — all three nodes are control-plane+etcd. Two down is a lost cluster.
+- **Every drain degrades all 25 Longhorn volumes to 2/3**, because 3 nodes × 3 replicas ×
+  `replica-soft-anti-affinity: false` leaves nowhere to place a replacement, and rebuild is
+  serialized behind a 600s wait. Reboot the next node too early and volumes hit 1/3, where
+  `block-if-contains-last-replica` can block a drain forever.
+- **Six PodDisruptionBudgets allow zero voluntary disruptions** — 3 Longhorn `instance-manager-*`
+  (structurally 0 by design; this is how Longhorn gates the drain) and 3 CNPG `*-primary`.
+
+## Placement — why its own Kustomization
+
+`platform` is `wait: true` over twelve components and gates `platform-config` → every apps wave. A
+reboot daemon that failed to go healthy there would stall **every application in the cluster** on
+"dependency not ready". A component added to make maintenance safer must not be able to cause an
+outage. `wait: true` here is a health **signal** for something nothing depends on — and nothing
+should ever `dependsOn: [kured]`.
+
+The `dependsOn: [platform]` edge is load-bearing, not conventional: it supplies the
+`monitoring.coreos.com` CRDs this component's `PrometheusRule` needs (F-MON1 — a pass containing an
+unknown kind aborts), plus the Prometheus that kured **queries** before every reboot.
+
+## How concurrent reboots are made impossible
+
+| mechanism | value | why |
+|---|---|---|
+| cluster-wide lock | one annotation on kured's own DaemonSet | a node reboots only while holding it; enforced by API-server optimistic concurrency, not timing |
+| `concurrency` | `1` | one lock holder at a time |
+| **`lockTtl`** | **`0` (disabled)** | a TTL lets reboots continue after the holder dies — on 3 etcd nodes that is the catastrophe. A hung node holds the lock **forever** and nothing cascades. |
+| `lockReleaseDelay` | `30m` | the next node waits out Longhorn's 600s replenishment even if no alert latched |
+| alert gate | `alertFilterMatchOnly: true` + anchored allow-list | refuses to proceed while a volume is degraded/faulted or a node is down |
+| window | 03:00–06:00 daily, America/New_York | matches `renovate.json`'s window; daily (not weekly) is what makes `KuredRebootPending`'s `for: 36h` meaningful |
+
+## Deliberate non-choices
+
+- **No `--ignore-daemonsets` / `--delete-emptydir-data` / `--force`.** Those flags **do not exist**
+  in kured 1.23.0 — it hardcodes all three in the drain helper (`cmd/kured/main.go:438-449`).
+  Passing them would CrashLoop the DaemonSet. All three are genuinely required here (12 DaemonSets,
+  135 emptyDir pods).
+- **`rebootMethod: command`, not `signal`.** `signal` would allow `privileged: false` + `CAP_KILL`,
+  but needs a host-side `kured-reboot.service` that does not exist on these nodes. A real future
+  hardening step; explicitly not taken here.
+- **No tolerations.** No prod node is tainted (measured). A future tainted node would be silently
+  uncovered — hence `KuredNodeCoverageGap`.
+- **`NodeEtcdRequestsSlow` / `NodeIOPressure*` are NOT in the alert gate**, despite being exactly
+  what you would want to pause for. prod-1 has a documented etcd WAL I/O starvation history and
+  including them risks a permanent wedge that looks like safety.
+- **No Slack/shoutrrr notifications.** Would require a SOPS Secret and a decryption patch; the three
+  alerts cover the same ground through Alertmanager.
+
+## Versioning
+
+Chart pinned `>=6.1.0 <6.2.0`. The minor ceiling is **deliberate here**, unlike elsewhere in this
+repo where it is a main cause of pin rot: a kured minor may change drain or lock semantics, and this
+is the one component where "a human read the changelog" should be a precondition. `renovate.json`'s
+`blast-radius` rule additionally forbids auto-merging it, so the ceiling is a review prompt, not a
+wall.
+
+## What breaks silently, and what catches it
+
+Every failure mode of this component leaves a cluster that looks healthy — kured stays 1/1 Running
+while rebooting nothing. `alerts.yaml` ships three rules over kured's single metric, with a
+companion promtool suite:
+
+| alert | catches |
+|---|---|
+| `KuredNotReporting` | the scrape pipeline is dead (chart defaults `service.create`/`metrics.create` to **false**; a NetworkPolicy can time the scrape out) — and the other two are inert while this is true |
+| `KuredNodeCoverageGap` | kured missing on *some* node, which `absent()` cannot see |
+| `KuredRebootPending` | a reboot was offered and blocked for >36h |
+
+## Post-merge verification
+
+```bash
+infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds,pod,svc,servicemonitor -o wide'
+infra/devcontainer/dev.sh run bash -c 'flux --context k3s-prod get kustomization kured'
+infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured logs ds/kured --tail=40'
+```
+
+Expect `3/3` ready, the Kustomization `Ready=True`, and kured logging a reboot-check decision each
+period. Then confirm the scrape is real — this is the step that proves the alerts are not inert:
+
+```bash
+infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n monitoring exec \
+  sts/prometheus-kube-prometheus-stack-prometheus -c prometheus -- \
+  wget -qO- "http://localhost:9090/api/v1/query?query=count(kured_reboot_required)"'
+```
+
+**Expect `3`.** Anything else (and especially an empty result) means the Service, ServiceMonitor or
+NetworkPolicy is wrong and all three alerts are silently dead.
+
+**Before trusting the timer with a real kernel upgrade**, do one watched drain to settle the open
+question the design record flags: whether CNPG actually switches its primary away in response to a
+drain-induced eviction. If it does not, draining `k3s-prod-1` blocks on three 0-disruption PDBs at
+once.

platform/kured/alerts.promtool-test.yaml

@@ -0,0 +1,282 @@
+---
+# promtool unit tests for the three rules in platform/kured/alerts.yaml.
+#
+# WHY THIS FILE EXISTS. The rules it covers are the ONLY thing that can tell an operator that the
+# reboot automation has stopped working, and every failure mode they describe is silent — kured stays
+# 1/1 Running while rebooting nothing. An alert that cannot fire is therefore not a cosmetic defect
+# here, it is the removal of the only signal. This repo has shipped that bug in both directions
+# already (a permanently-firing VPNTunnelDown; a structurally inert VeleroNoRecentBackup over an
+# absent series), which is why the harness exists at all. Each rule below is pinned BOTH ways: quiet
+# on the healthy state, firing on the constructed bad one, with the `for:` boundary asserted rather
+# than assumed.
+#
+# HOW TO RUN. `platform/monitoring/tests/run.sh` — the one repo-wide rule-unit harness. It finds this
+# file by the COMPANION convention (`<manifest>.promtool-test.yaml` beside `<manifest>.yaml`), so the
+# manifest under test is exactly platform/kured/alerts.yaml and nothing else. Run just this file with
+#   platform/monitoring/tests/run.sh platform/kured/alerts.promtool-test.yaml
+# The harness flattens the PrometheusRule CR's `.spec.groups` into the bare rules file promtool wants,
+# `promtool check rules` it first so a non-parsing file cannot pass vacuously, and runs it all in a
+# temp dir. It leaves Flux's `${BASE_DOMAIN}` in the runbook_url annotations verbatim, which is safe
+# because the assertions below query the `ALERTS` series and never compare annotations.
+#
+# WHY THE ASSERTIONS QUERY `ALERTS` RATHER THAN USING `exp_alerts`. promtool's alert_rule_test
+# compares annotations VERBATIM, which would mean pasting each rule's description prose in here and
+# re-pasting it on every wording change. Querying the synthetic `ALERTS` series proves what actually
+# matters — the alert reaches `pending` and then `firing` across its `for:` boundary, with the right
+# labels — and stays decoupled from annotation text. Same choice as
+# platform/cnpg/alerts.promtool-test.yaml.
+#
+# THE SERIES SHAPES ARE REAL. `kured_reboot_required{node=...}` is a GaugeVec with exactly one label,
+# `node` — read from kured's cmd/kured/main.go:92-96 (the only registered metric in the project).
+# `kube_node_info{node=...}` was measured on prod Prometheus on 2026-10-17 returning
+# `count(kube_node_info) == 3`, matching the three nodes. The extra kube_node_info labels prod
+# carries are omitted deliberately: every rule aggregates with `count()`, so they cannot affect the
+# result, and inventing plausible-but-wrong values would be worse than leaving them out.
+#
+# evaluation_interval is 5m, not the usual 1m: KuredRebootPending has `for: 36h`, and at a 1m
+# interval promtool would evaluate that test over 2000+ steps for no added coverage.
+rule_files:
+  - alerts.rules.yaml
+evaluation_interval: 5m
+
+tests:
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  # 1. THE HEALTHY STEADY STATE. Three nodes, kured running on all three, nothing pending.
+  #    ALL THREE RULES MUST BE SILENT. This is the case that a badly-written `absent()` or a
+  #    `count() <` comparison with an off-by-one gets wrong, and it is also the state the cluster
+  #    is in ~100% of the time, so a false positive here would train the operator to ignore the
+  #    whole group.
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  - interval: 5m
+    name: healthy cluster - no reboots pending, kured on every node, all three rules quiet
+    input_series:
+      - series: 'kured_reboot_required{node="k3s-prod-1"}'
+        values: '0x60'
+      - series: 'kured_reboot_required{node="k3s-prod-2"}'
+        values: '0x60'
+      - series: 'kured_reboot_required{node="k3s-prod-3"}'
+        values: '0x60'
+      - series: 'kube_node_info{node="k3s-prod-1"}'
+        values: '1x60'
+      - series: 'kube_node_info{node="k3s-prod-2"}'
+        values: '1x60'
+      - series: 'kube_node_info{node="k3s-prod-3"}'
+        values: '1x60'
+    promql_expr_test:
+      # Nothing from this group may be pending or firing, at any point in the window.
+      - expr: 'count(ALERTS{alertname=~"Kured.*"})'
+        eval_time: 4h
+        exp_samples: []
+      # And the underlying expressions must each be genuinely false, not merely unevaluated:
+      # absent() yields no series because the vector is populated...
+      - expr: 'absent(kured_reboot_required)'
+        eval_time: 4h
+        exp_samples: []
+      # ...coverage is exact (3 == 3), so the `<` comparison drops the sample...
+      - expr: 'count(kured_reboot_required) < count(kube_node_info)'
+        eval_time: 4h
+        exp_samples: []
+      # ...and no node reports a pending reboot.
+      - expr: 'max by (node) (kured_reboot_required) == 1'
+        eval_time: 4h
+        exp_samples: []
+
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  # 2. KuredNotReporting — THE WHOLE PIPELINE IS DEAD. No kured series at all, which is what the
+  #    cluster looks like if the chart's `service.create`/`metrics.create` regress to their FALSE
+  #    defaults, if the ServiceMonitor is dropped, or if a NetworkPolicy silently times out the
+  #    scrape (the Longhorn precedent). The nodes still exist, so kube_node_info is present —
+  #    proving the rule keys on kured's absence and not on a dead cluster.
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  - interval: 5m
+    name: no kured series anywhere - KuredNotReporting fires after 30m
+    input_series:
+      - series: 'kube_node_info{node="k3s-prod-1"}'
+        values: '1x60'
+      - series: 'kube_node_info{node="k3s-prod-2"}'
+        values: '1x60'
+      - series: 'kube_node_info{node="k3s-prod-3"}'
+        values: '1x60'
+    promql_expr_test:
+      # The expression itself yields 1 immediately.
+      - expr: 'absent(kured_reboot_required)'
+        eval_time: 10m
+        exp_samples:
+          - labels: '{}'
+            value: 1
+      # `for: 30m` is honoured — still only pending at 25m.
+      - expr: 'ALERTS{alertname="KuredNotReporting"}'
+        eval_time: 25m
+        exp_samples:
+          - labels: 'ALERTS{alertname="KuredNotReporting", alertstate="pending", severity="warning"}'
+            value: 1
+      # ...and firing once past it.
+      - expr: 'ALERTS{alertname="KuredNotReporting", alertstate="firing"}'
+        eval_time: 35m
+        exp_samples:
+          - labels: 'ALERTS{alertname="KuredNotReporting", alertstate="firing", severity="warning"}'
+            value: 1
+      # THE GUARD CLAUSE: with kured absent, the other two rules are inert — exactly the reason
+      # this rule has to exist. KuredRebootPending cannot fire (no series to be 1) and
+      # KuredNodeCoverageGap cannot either, because `count()` over an empty vector returns NO
+      # SAMPLE rather than 0, so the `<` comparison has nothing to compare. Asserting that here
+      # pins the dependency: if a future edit made coverage-gap fire in this state, the overlap
+      # would be deliberate rather than accidental.
+      - expr: 'ALERTS{alertname="KuredRebootPending"}'
+        eval_time: 35m
+        exp_samples: []
+      - expr: 'ALERTS{alertname="KuredNodeCoverageGap"}'
+        eval_time: 35m
+        exp_samples: []
+      - expr: 'count(kured_reboot_required) < count(kube_node_info)'
+        eval_time: 35m
+        exp_samples: []
+
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  # 3. KuredNodeCoverageGap — kured COVERS ONLY TWO OF THREE NODES. k3s-prod-3's series stops at
+  #    1h and never returns (a node taint this DaemonSet does not tolerate, a node that never came
+  #    back from a reboot, a DaemonSet nodeSelector edit). This also pins the `for: 2h` choice,
+  #    which exists SO THAT AN ORDINARY REBOOT DOES NOT TRIP IT: the gap is real from ~1h05 but
+  #    the alert is only pending an hour later, and a healthy reboot is minutes.
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  - interval: 5m
+    name: kured missing on one node - KuredNodeCoverageGap fires after 2h, not before
+    input_series:
+      - series: 'kured_reboot_required{node="k3s-prod-1"}'
+        values: '0x60'
+      - series: 'kured_reboot_required{node="k3s-prod-2"}'
+        values: '0x60'
+      # Ends after 1h (13 samples at 5m) and is stale thereafter.
+      - series: 'kured_reboot_required{node="k3s-prod-3"}'
+        values: '0x12'
+      - series: 'kube_node_info{node="k3s-prod-1"}'
+        values: '1x60'
+      - series: 'kube_node_info{node="k3s-prod-2"}'
+        values: '1x60'
+      - series: 'kube_node_info{node="k3s-prod-3"}'
+        values: '1x60'
+    promql_expr_test:
+      # Before the gap opens, coverage is exact and nothing is pending.
+      - expr: 'ALERTS{alertname="KuredNodeCoverageGap"}'
+        eval_time: 30m
+        exp_samples: []
+      # The shortfall is real well before the alert fires — 2 < 3.
+      - expr: 'count(kured_reboot_required)'
+        eval_time: 2h
+        exp_samples:
+          - labels: '{}'
+            value: 2
+      # A SHORT OUTAGE DOES NOT ALERT: still only pending at 2h, an hour after the gap opened.
+      - expr: 'ALERTS{alertname="KuredNodeCoverageGap"}'
+        eval_time: 2h
+        exp_samples:
+          - labels: 'ALERTS{alertname="KuredNodeCoverageGap", alertstate="pending", severity="warning"}'
+            value: 1
+      # Firing once the gap has persisted past `for: 2h`.
+      - expr: 'ALERTS{alertname="KuredNodeCoverageGap", alertstate="firing"}'
+        eval_time: 3h15m
+        exp_samples:
+          - labels: 'ALERTS{alertname="KuredNodeCoverageGap", alertstate="firing", severity="warning"}'
+            value: 1
+      # KuredNotReporting must stay QUIET: two nodes are still publishing, so the vector is not
+      # absent. This is the distinction between "kured is gone" and "kured is incomplete", and
+      # getting it wrong in either direction would collapse two different diagnoses into one.
+      - expr: 'ALERTS{alertname="KuredNotReporting"}'
+        eval_time: 3h15m
+        exp_samples: []
+
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  # 4. KuredRebootPending — A REBOOT IS BLOCKED. k3s-prod-1 reports a pending reboot continuously:
+  #    the sentinel was written (an operator kernel upgrade) and the reboot has not happened across
+  #    more than one 03:00-06:00 window. Asserts the 36h boundary in both directions, which is the
+  #    assertion that makes the DAILY-window decision load-bearing rather than cosmetic: on a
+  #    weekly window this threshold could not distinguish "blocked" from "waiting for Sunday".
+  #    Series run to 38h20m (456 samples at 5m) so the post-boundary evaluation has real data.
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  - interval: 5m
+    name: reboot pending on one node for over 36h - KuredRebootPending fires, per-node label kept
+    input_series:
+      - series: 'kured_reboot_required{node="k3s-prod-1"}'
+        values: '1x456'
+      - series: 'kured_reboot_required{node="k3s-prod-2"}'
+        values: '0x456'
+      - series: 'kured_reboot_required{node="k3s-prod-3"}'
+        values: '0x456'
+      - series: 'kube_node_info{node="k3s-prod-1"}'
+        values: '1x456'
+      - series: 'kube_node_info{node="k3s-prod-2"}'
+        values: '1x456'
+      - series: 'kube_node_info{node="k3s-prod-3"}'
+        values: '1x456'
+    promql_expr_test:
+      # The expression selects ONLY the pending node — the operator needs to know which box.
+      - expr: 'max by (node) (kured_reboot_required) == 1'
+        eval_time: 1h
+        exp_samples:
+          - labels: '{node="k3s-prod-1"}'
+            value: 1
+      # A NORMAL WAIT IS NOT AN ALERT. At 24h the node is still inside its legitimate
+      # once-per-day window wait, so the alert may be pending but must NOT be firing.
+      - expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}'
+        eval_time: 24h
+        exp_samples: []
+      # Still not firing just short of the boundary.
+      - expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}'
+        eval_time: 35h30m
+        exp_samples: []
+      # Firing past it, carrying the node label through to the notification.
+      - expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}'
+        eval_time: 36h30m
+        exp_samples:
+          - labels: 'ALERTS{alertname="KuredRebootPending", alertstate="firing", node="k3s-prod-1", severity="warning"}'
+            value: 1
+      # The two healthy nodes must NOT be alerted on. A rule written with `kured_reboot_required`
+      # unaggregated, or compared with `>= 0`, would raise all three and bury the real one.
+      - expr: 'count(ALERTS{alertname="KuredRebootPending", alertstate="firing"})'
+        eval_time: 36h30m
+        exp_samples:
+          - labels: '{}'
+            value: 1
+      # Liveness rules stay quiet: kured is present and covering all three nodes. A pending reboot
+      # is NOT a wiring fault and must not be reported as one.
+      - expr: 'ALERTS{alertname="KuredNotReporting"}'
+        eval_time: 36h30m
+        exp_samples: []
+      - expr: 'ALERTS{alertname="KuredNodeCoverageGap"}'
+        eval_time: 36h30m
+        exp_samples: []
+
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  # 5. THE RULE MUST STOP. A reboot that SUCCEEDS clears the sentinel, and the alert must clear
+  #    with it — the other half of the failure space this harness exists for (a permanently-firing
+  #    rule is how VPNTunnelDown taught this repo the lesson). k3s-prod-1 is pending for ~40h,
+  #    long enough to fire, then the reboot happens and the gauge returns to 0.
+  # ─────────────────────────────────────────────────────────────────────────────────────────────
+  - interval: 5m
+    name: reboot completes after firing - KuredRebootPending resolves
+    input_series:
+      # 1 for 480 samples (40h), then 0 for the remainder.
+      - series: 'kured_reboot_required{node="k3s-prod-1"}'
+        values: '1x480 0x120'
+      - series: 'kured_reboot_required{node="k3s-prod-2"}'
+        values: '0x600'
+      - series: 'kured_reboot_required{node="k3s-prod-3"}'
+        values: '0x600'
+      - series: 'kube_node_info{node="k3s-prod-1"}'
+        values: '1x600'
+      - series: 'kube_node_info{node="k3s-prod-2"}'
+        values: '1x600'
+      - series: 'kube_node_info{node="k3s-prod-3"}'
+        values: '1x600'
+    promql_expr_test:
+      # Firing while the reboot is still outstanding.
+      - expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}'
+        eval_time: 39h
+        exp_samples:
+          - labels: 'ALERTS{alertname="KuredRebootPending", alertstate="firing", node="k3s-prod-1", severity="warning"}'
+            value: 1
+      # Cleared once the sentinel is gone. If this assertion ever fails, the rule has become
+      # self-sustaining and the operator will start ignoring it.
+      - expr: 'ALERTS{alertname=~"Kured.*"}'
+        eval_time: 45h
+        exp_samples: []

platform/kured/alerts.yaml

@@ -0,0 +1,124 @@
+---
+# Alert rules for the kured reboot automation.
+#
+# WHY THESE THREE, AND WHY THEY ARE NOT OPTIONAL. Reboot automation's characteristic failure is
+# SILENCE, not noise. Every way this component can break leaves a cluster that looks perfectly
+# healthy: a sentinel nothing writes, a Service the chart did not create, a NetworkPolicy that drops
+# the scrape, a taint that excludes a node, a drain wedged on a PodDisruptionBudget, a cluster lock
+# held forever by a node that hung on warm reboot. In none of those cases does anything turn red on
+# its own. kured itself stays 1/1 Running throughout.
+#
+# kured EXPOSES EXACTLY ONE METRIC. Read from cmd/kured/main.go:92-96 and :113 on 2026-10-17 — the
+# only `prometheus.MustRegister` in the tree is a GaugeVec `Subsystem: "kured"`, `Name:
+# "reboot_required"`, labelled `{node}`. So the full vocabulary available is
+# `kured_reboot_required{node="..."}`, 1 when the sentinel exists and 0 when it does not. Each rule
+# below is built from that one series plus `kube_node_info` (kube-state-metrics), and nothing else.
+# There is deliberately no rule over an invented metric name: this repo has shipped a
+# structurally-inert alert before (VeleroNoRecentBackup over an absent series, which is why
+# platform/monitoring/tests/run.sh exists), and an alert that cannot fire is worse than no alert
+# because it also removes the suspicion that would have caught it.
+#
+# THESE RULES LIVE HERE, NOT IN platform/monitoring/alerts.yaml, for two reasons. (1) The unit-test
+# harness treats a `<manifest>.promtool-test.yaml` COMPANION as covering just its sibling, whereas a
+# `tests/*.test.yaml` SUITE covers every PrometheusRule in its directory — so shipping them beside
+# the component keeps the test obligation local and legible (the platform/cnpg precedent). (2)
+# platform/monitoring/alerts.yaml is concurrently edited by other open branches; adding 3 rules to
+# its tail would create a merge conflict for no benefit.
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: kured
+  namespace: kured
+  labels:
+    app.kubernetes.io/name: kured
+spec:
+  groups:
+    - name: platform-health.node-reboots
+      rules:
+        # ── 1. IS THE MECHANISM ALIVE AT ALL? ────────────────────────────────────────────────
+        - alert: KuredNotReporting
+          # `absent()` yields a series only when the ENTIRE vector is empty, so this fires when no
+          # node at all is publishing — kured gone, or its Service/ServiceMonitor/NetworkPolicy
+          # regressed so nothing is scraped. It stays quiet while a single node is mid-reboot
+          # (the other two still report), which is what makes it specific to a wiring failure
+          # rather than to a reboot in progress.
+          #
+          # THIS IS THE RULE THAT GUARDS THE OTHER TWO. Both of them read `kured_reboot_required`,
+          # so if that series stops existing they go quiet and look like good news. This one turns
+          # the absence itself into the signal. The specific regression it is aimed at: the kured
+          # chart defaults `service.create` and `metrics.create` to FALSE, and Longhorn has already
+          # demonstrated on this cluster (see the companion note in
+          # conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh) that a
+          # NetworkPolicy can let targets be DISCOVERED while every scrape times out and the TSDB
+          # stays empty.
+          expr: absent(kured_reboot_required)
+          # 30m absorbs a HelmRelease upgrade rolling the DaemonSet and a Prometheus restart,
+          # without tolerating a real wiring break for more than one quiet night's worth of time.
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: kured is publishing no reboot_required series on any node
+            description: >-
+              No node is reporting kured_reboot_required, so the reboot automation is either not
+              running or not being scraped. While this is true, OS reboots are not happening and
+              the two alerts that would say so (KuredRebootPending, KuredNodeCoverageGap) are
+              themselves inert. Nodes will silently accumulate pending kernel upgrades.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-reboot-automation"
+
+        # ── 2. IS IT ALIVE ON *EVERY* NODE? ──────────────────────────────────────────────────
+        - alert: KuredNodeCoverageGap
+          # A per-node agent that silently covers only some nodes is the subtler half of the
+          # liveness question, and `absent()` cannot see it. kured publishes one series per node it
+          # runs on; kube-state-metrics publishes one `kube_node_info` per node that exists
+          # (measured 2026-10-17: 3, matching the 3 nodes). A shortfall means at least one node has
+          # no reboot automation — most likely a new or re-provisioned node carrying a taint this
+          # DaemonSet does not tolerate (it declares none, because no prod node is tainted today).
+          # That node would then never reboot, with nothing else to indicate it.
+          expr: count(kured_reboot_required) < count(kube_node_info)
+          # 2h, chosen to sit comfortably ABOVE the duration of a healthy reboot. A node being
+          # rebooted stops publishing for a few minutes and legitimately creates a shortfall; a
+          # node that is still missing after two hours is either genuinely uncovered or has hung on
+          # reboot, and both of those want a human.
+          for: 2h
+          labels:
+            severity: warning
+          annotations:
+            summary: kured is not running on every node
+            description: >-
+              Fewer nodes are reporting kured_reboot_required than exist in the cluster, so at
+              least one node has no reboot automation and will never be rebooted by it. Most
+              likely a node carries a taint this DaemonSet does not tolerate, or has hung during
+              a reboot.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-reboot-automation"
+
+        # ── 3. IS A REBOOT STUCK? ────────────────────────────────────────────────────────────
+        - alert: KuredRebootPending
+          # A node has wanted a reboot for longer than a full maintenance window plus margin, so
+          # the reboot was OFFERED and did not happen. The causes are all real and all invisible
+          # otherwise: the Prometheus alert gate is blocking on a degraded Longhorn volume that is
+          # not recovering; the drain hit `drainTimeout` against a 0-disruption PDB; another node
+          # holds the cluster lock and never released it (`lockTtl: 0` means a lock held by a hung
+          # node is held FOREVER — that is deliberate, see helmrelease.yaml, and this is the alert
+          # that makes the resulting stall visible).
+          expr: max by (node) (kured_reboot_required) == 1
+          # 36h, and the window being DAILY is what makes this number meaningful — see the window
+          # block in helmrelease.yaml. With a 03:00-06:00 window every day, a pending reboot waits
+          # at most ~24h legitimately, so 36h means at least one full window elapsed without the
+          # reboot completing. On a weekly window this threshold would have to exceed 7 days, which
+          # is longer than this cluster's 7d Prometheus retention and is reset by any Prometheus
+          # restart past the 1h for-outage-tolerance — i.e. unreliable in the one direction that
+          # matters. Note this alert can still be LATE for that reason, but it cannot be spurious.
+          for: 36h
+          labels:
+            severity: warning
+          annotations:
+            summary: Node {{ $labels.node }} has needed a reboot for over 36h
+            description: >-
+              kured reports a pending reboot on {{ $labels.node }} that has survived at least one
+              full 03:00-06:00 maintenance window. The reboot is being blocked, not merely
+              waiting: likely a degraded Longhorn volume holding the Prometheus alert gate, a
+              drain timing out against a PodDisruptionBudget, or a stale cluster lock held by a
+              node that did not come back. The kernel or security update that needed the reboot is
+              installed but NOT running.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-reboot-automation"

platform/kured/helmrelease.yaml

@@ -0,0 +1,289 @@
+---
+# kured (Kubernetes Reboot Daemon) — coordinated, one-node-at-a-time reboots of the three prod
+# control-plane nodes.
+#
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# READ THIS FIRST: kured DOES NOT DECIDE *WHETHER* TO REBOOT. IT ONLY DECIDES *WHEN* AND *HOW*.
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# kured reboots a node if and only if the file `/var/run/reboot-required` exists on it. On these
+# nodes exactly one thing creates that file: a deliberate operator run of
+# `playbooks/node-baseline.yml --tags kernel` (which advances the backports kernel). The
+# unattended-upgrades role this PR also adds installs SECURITY updates only and BLACKLISTS
+# `linux-image-*`/`linux-headers-*`, so it can never produce a reboot-required sentinel on its own —
+# see infra/ansible/roles/unattended_upgrades/defaults/main.yml for that argument in full.
+#
+# That is the whole answer to this repo's two standing objections to reboot automation:
+#
+#   * infra/ansible/roles/node_baseline/tasks/main.yml:61-67 — "IT DOES NOT REBOOT." k3s-prod-2 hung
+#     on a warm reboot on 2026-09-03 (powered, off the LAN ~38 minutes, physical power button
+#     required), so rebooting was made "a SERIALIZED, SUPERVISED operator step ... and a role cannot
+#     enforce that." STILL TRUE, and nothing here changes it: node_baseline continues not to reboot.
+#   * docs/plans/cluster-upgrade-plan.md:291-299 — system-upgrade-controller was rejected because it
+#     "automates away the human gate between nodes", and the operator's hard requirement is an
+#     explicit PASS/STOP verification BETWEEN every node.
+#
+# THE HUMAN GATE IS PRESERVED, AND IT IS THE SENTINEL. The operator decides whether a kernel moves
+# and on which hosts (`--limit k3s-prod-1`), which is what writes the sentinel on those hosts and
+# only those hosts. kured then performs the cordon/drain/reboot/uncordon that the operator would
+# otherwise have typed by hand — the error-prone mechanical half, including the ordering rule
+# node_baseline's own message has to shout ("NEVER node 2 first"). Run the kernel play one host at a
+# time and the operator keeps per-node PASS/STOP control exactly as before; run it on all three and
+# kured still does them strictly one at a time, just in an order it chooses. Both are safe for
+# quorum; only the first preserves the ordering preference. See runbooks/node-reboot-automation.md.
+#
+# cluster-upgrade-plan.md:329-330 and :545-548 name the condition under which its own rejection
+# reverses — "SUC becomes viable once the cluster is on fast disks ... where drain-driven rebuild is
+# cheap" — and prod is that cluster. This component takes that clause up, but NOT its drain
+# assumption: see the Longhorn section below, where drain-driven rebuild is measurably NOT free even
+# on SSD, and is gated on explicitly rather than waved through.
+#
+# WHAT IS STILL UNMITIGATED, STATED PLAINLY: kured cannot press a power button. If a node hangs on
+# warm reboot the way k3s-prod-2 did in September, kured will not recover it. The blast radius is
+# bounded by design rather than by luck — `concurrency: 1` plus `lockTtl: 0` means the hung node
+# keeps the cluster lock forever, so NO second node is ever rebooted behind it; etcd keeps 2/3 quorum
+# and the cluster stays up, degraded, until a human attends. The KuredRebootPending alert in
+# alerts.yaml is what tells them. That is strictly better than the previous automation story (none)
+# but it is NOT the same as "safe to ignore at 3am on a Sunday".
+apiVersion: helm.toolkit.fluxcd.io/v2
+kind: HelmRelease
+metadata:
+  name: kured
+  namespace: kured
+spec:
+  interval: 30m
+  chart:
+    spec:
+      chart: kured
+      # Chart 6.1.0 -> appVersion 1.23.0 (published 2026-06-30; newest of the 80 entries in
+      # https://kubereboot.github.io/charts/index.yaml, read 2026-10-17).
+      #
+      # THE `<6.2.0` CEILING IS A DELIBERATE CHOICE HERE, NOT THE REFLEX IT IS ELSEWHERE. This PR's
+      # own audit found that 9 of 12 HelmReleases in this repo pin a minor-locking range, and that
+      # this is a MAIN CAUSE of pin rot: Renovate can then only ever offer patches inside a line it
+      # has been told not to leave (kube-prometheus-stack is stuck 4 chart majors back that way).
+      # For a component whose job is to reboot control-plane nodes, that ceiling is worth paying on
+      # purpose: a kured MINOR may change drain or lock semantics, and this is the one component
+      # where "a human looked at the changelog" should be a precondition, not a nice-to-have.
+      # Renovate will raise the minor as a non-automerged PR (see the kured packageRule in
+      # renovate.json), so the ceiling is a review prompt, not a wall.
+      version: ">=6.1.0 <6.2.0"
+      sourceRef:
+        kind: HelmRepository
+        name: kured
+        namespace: kured
+  install:
+    remediation:
+      retries: 3
+  upgrade:
+    remediation:
+      retries: 3
+  values:
+    # ── IDENTITY / PLACEMENT ───────────────────────────────────────────────────────────────────
+    # No tolerations. Measured 2026-10-17: all three prod nodes carry `spec.taints` EMPTY (k3s does
+    # not taint its servers), so a DaemonSet reaches every node untainted. If a future node IS
+    # tainted, kured silently stops covering it and that node then never reboots — invisible,
+    # because the other two keep reporting. KuredNodeCoverageGap in alerts.yaml is the detector: it
+    # compares kured's per-node series count against the node count rather than merely asking
+    # whether kured exists.
+    priorityClassName: system-node-critical
+    # The chart's defaults are kept: `hostPID: true` and `containerSecurityContext.privileged: true`
+    # with `readOnlyRootFilesystem: true`. Both are load-bearing and neither is reducible:
+    # `rebootMethod: command` runs the reboot by nsenter'ing /proc/1/ns/mnt, which needs the host PID
+    # namespace to see PID 1 and the privilege to enter its mount namespace. The documented
+    # lower-privilege alternative (privileged:false + CAP_KILL) applies ONLY to
+    # `rebootMethod: signal`, which signals a host-side kured-reboot.service that does not exist on
+    # these nodes. Choosing `signal` would mean shipping and maintaining that unit via Ansible for a
+    # capability reduction on a DaemonSet that is already cluster-admin-adjacent by function; it is a
+    # reasonable future hardening step and is explicitly NOT taken here.
+    resources:
+      requests:
+        cpu: 10m
+        memory: 32Mi
+      limits:
+        # No CPU limit on purpose (the repo's node-pressure work treats CPU limits on tiny system
+        # agents as a throttling hazard). Memory is capped: kured holds no state beyond one
+        # annotation and a gauge, and 128Mi is ~4x its idle RSS upstream reports.
+        memory: 128Mi
+
+    configuration:
+      # ── THE QUORUM CONSTRAINT: TWO NODES MUST NEVER BE DOWN AT ONCE ──────────────────────────
+      # All three prod nodes are control-plane + etcd (measured 2026-10-17: every node reports
+      # `control-plane,etcd`). etcd quorum is 2 of 3, so one node down is degraded-but-up and two
+      # nodes down is a LOST CLUSTER that does not self-heal. Everything in this block exists to
+      # make the second case structurally unreachable rather than merely unlikely.
+      #
+      # concurrency: 1 — kured's cluster-wide lock is a single annotation
+      # (`weave.works/kured-node-lock`) on kured's OWN DaemonSet object, and a node reboots only
+      # while it holds that lock. One lock + concurrency 1 = one reboot, enforced by the API server's
+      # optimistic concurrency rather than by timing. This is the chart default; it is written out
+      # explicitly because it is the single most important value in this file.
+      concurrency: 1
+      # lockTtl: 0 (DISABLED) — THIS IS A SAFETY DECISION, NOT A DEFAULT WE FORGOT TO TUNE.
+      # A TTL makes the lock EXPIRE, which exists for the case "the lock holder died and we want
+      # reboots to continue". On a 3-node etcd cluster that is precisely the catastrophe: if node A
+      # has hung on a warm reboot (which has happened here — see the header) and its lock expires,
+      # node B becomes eligible and takes the cluster from 2/3 to 1/3. We would rather the reboot
+      # cycle STOP FOREVER and page a human than have it step over a corpse. The cost is that a
+      # genuinely stale lock needs manual clearing; runbooks/node-reboot-automation.md has the
+      # procedure, and KuredRebootPending is the alert that surfaces it.
+      lockTtl: 0
+      # lockReleaseDelay: 30m — hold the lock for 30 minutes AFTER this node is back and uncordoned,
+      # before any other node may take it. See the Longhorn section: this is the STRUCTURAL half of
+      # the inter-node gate, deliberately not dependent on an alert's `for:` timer having latched.
+      lockReleaseDelay: 30m
+
+      # ── THE MAINTENANCE WINDOW: 03:00-06:00 EVERY DAY, America/New_York ──────────────────────
+      # The timezone and the 03:00-06:00 band match renovate.json's existing
+      # `"schedule": ["after 3am and before 6am"]` / `"timezone": "America/New_York"`, so every
+      # unattended change in this repo lands in the same quiet 3-hour window.
+      #
+      # DAILY, NOT WEEKLY, AND THE REASON IS THE ALERT — not impatience. A weekly (`- sun`) window
+      # was the first draft and it breaks the only alert that can tell you this mechanism has
+      # stalled. With a Sunday-only window, a kernel upgraded on a Monday legitimately waits six
+      # days, so "a reboot has been pending too long" cannot be distinguished from "it is Tuesday"
+      # until the threshold exceeds a full 7-day cycle. That forces `for: 8d`+, which is longer than
+      # this cluster's Prometheus retention (MONITORING_PROMETHEUS_RETENTION is 7d) and is reset by
+      # any Prometheus restart beyond the 1h default for-outage-tolerance — i.e. the alert would be
+      # unreliable in exactly the direction that matters. A daily window caps the legitimate wait at
+      # under 24h, which makes KuredRebootPending's `for: 36h` both robust and meaningful: 36 hours
+      # with a sentinel still set means at least one full window passed and SOMETHING BLOCKED IT.
+      #
+      # Daily does NOT weaken the quorum guarantee — that is `concurrency: 1` plus the lock, not the
+      # calendar. It does mean a 3-node kernel rollout can finish in one to three nights instead of
+      # three weeks, with each node still gated on the previous one's Longhorn rebuilds (below).
+      rebootDays:
+        - su
+        - mo
+        - tu
+        - we
+        - th
+        - fr
+        - sa
+      startTime: "03:00"
+      endTime: "06:00"
+      timeZone: America/New_York
+      # How often to re-evaluate (sentinel + window + alerts + lock). 1h is the chart default; at a
+      # 3h window that is three evaluation opportunities per node per night.
+      period: 1h
+
+      # ── THE SENTINEL ─────────────────────────────────────────────────────────────────────────
+      # Written ONLY by a deliberate operator kernel upgrade (see the header). `update-notifier-common`
+      # — the package that owns /etc/kernel/postinst.d and actually creates this file on Debian — was
+      # measured ABSENT on all three nodes on 2026-10-17, which is why the unattended_upgrades role
+      # installs it. Without that package nothing ever writes this path, kured correctly never
+      # reboots, every probe looks healthy, and the entire mechanism is a silent no-op. That failure
+      # mode is the reason KuredNotReporting exists.
+      #
+      # THE PATH kured LOGS IS NOT THIS PATH, and that trips people debugging it. The chart
+      # defaults `useRebootSentinelHostPath: true` (kept), which bind-mounts the DIRNAME of this
+      # value onto /sentinel read-only and renders the flag as
+      # `--reboot-sentinel=/sentinel/reboot-required` (daemonset.yaml:134). So the container watches
+      # /sentinel/reboot-required and the host file is /var/run/reboot-required. Same file, two
+      # names. Set this value as the HOST path, as below.
+      rebootSentinel: /var/run/reboot-required
+
+      # ── THE LONGHORN GATE: WHY POD READINESS IS NOT A SUFFICIENT INTER-NODE SIGNAL ────────────
+      # Measured on prod 2026-10-17, and this is the finding that shaped this file:
+      #   * 25 Longhorn volumes, ALL `numberOfReplicas: 3`, all healthy, with exactly 25 replicas on
+      #     each of the 3 nodes. So there is no single-replica volume to lose today.
+      #   * BUT `replica-soft-anti-affinity: false` (replicas may not share a node) and
+      #     `disable-scheduling-on-cordoned-node: true`. With 3 nodes and 3 replicas, cordoning ONE
+      #     node leaves nowhere to place a replacement — so every drain takes ALL 25 volumes to
+      #     degraded 2/3 for the whole reboot.
+      #   * Recovery is slow and serialized: `replica-replenishment-wait-interval: 600` (10 minutes
+      #     before a rebuild even starts) and `concurrent-replica-rebuild-per-node-limit: 1`.
+      # Therefore: if the next node is rebooted before rebuilds finish, those volumes go to 1/3, and
+      # the drain after that genuinely CAN hit Longhorn's `node-drain-policy:
+      # block-if-contains-last-replica` and block forever. The inter-node gate must be VOLUME
+      # ROBUSTNESS, not "are the pods Ready again".
+      #
+      # It is gated twice, on purpose, because each half covers the other's blind spot:
+      #   1. lockReleaseDelay: 30m (above) — unconditional. Covers the 600s replenishment wait with
+      #      margin and does not depend on any alert having fired.
+      #   2. The Prometheus alert gate below — semantic. Covers a rebuild that takes LONGER than 30
+      #      minutes (25 serialized rebuilds on one node can), which a fixed delay cannot.
+      prometheusUrl: http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090
+      # Only consider alerts that are FIRING, not merely pending. A `pending` alert is a 10-minute
+      # guess; blocking a weekly reboot on one would make the gate fire on noise.
+      alertFiringOnly: true
+      # ── alertFilterMatchOnly: true IS LOAD-BEARING. DO NOT DROP IT. ──────────────────────────
+      # kured's two alert modes are near-opposites and the flag name does not advertise it:
+      #   * false (the chart default): alertFilterRegexp is an IGNORE list — kured blocks on ANY
+      #     firing alert except those matching.
+      #   * true: alertFilterRegexp is an ALLOW list — kured blocks ONLY on alerts that match.
+      # We need `true`, because this cluster always has something firing. Measured via the Prometheus
+      # API on 2026-10-17, these five were firing with no reboot pending and no end in sight:
+      # CNPGPrimariesCoLocated, KubeDeploymentReplicasMismatch, KubeDeploymentRolloutStuck,
+      # KubePodNotReady, OAuth2ProxyGateUnobserved (the last three all trace to the pm-agent pod,
+      # ImagePullBackOff on a placeholder digest for 12 days). Under the default `false`, kured would
+      # have been blocked by those on day one and would never have rebooted anything — a silent
+      # no-op indistinguishable, from outside, from "no reboots were needed".
+      alertFilterMatchOnly: true
+      # The ALLOW list: the conditions under which rebooting ANOTHER node risks data or quorum.
+      # Anchored and explicit — a prefix match here would silently re-admit the chronic five above.
+      #   LonghornVolumeDegraded  — the inter-node gate (see above). `longhorn_volume_robustness
+      #                             {state="degraded"} == 1`, for: 10m, in platform/monitoring/alerts.yaml.
+      #   LonghornVolumeFaulted   — data already at risk; never reboot into this.
+      #   KubeNodeNotReady        — a node is already down. Rebooting a second one loses quorum. This
+      #                             is the single most important entry for the etcd constraint.
+      #   NodeExporterDown        — a node we cannot see the health of. Treated as "assume the worst".
+      # DELIBERATELY NOT INCLUDED: NodeEtcdRequestsSlow and the NodeIOPressure* family. They are
+      # exactly the conditions you would want to pause for, but prod-1 has a documented history of
+      # etcd WAL I/O starvation, so including them risks a permanent wedge that looks like safety.
+      # Revisit once those are quiet for a sustained period; KuredRebootPending makes a wedge visible
+      # either way.
+      alertFilterRegexp: ^(LonghornVolumeDegraded|LonghornVolumeFaulted|KubeNodeNotReady|NodeExporterDown)$
+
+      # ── THE DRAIN ────────────────────────────────────────────────────────────────────────────
+      # NOTE ON FLAGS THAT DO NOT EXIST: there is no `--ignore-daemonsets`, no
+      # `--delete-emptydir-data` and no `--force` in kured 1.23.0. Read from
+      # cmd/kured/main.go:438-449 on 2026-10-17, kured HARDCODES
+      # `IgnoreAllDaemonSets: true`, `DeleteEmptyDirData: true` and `Force: true` in the drain
+      # helper. Passing them as extraArgs would CrashLoop the DaemonSet on an unknown flag. This
+      # matters because all three are genuinely REQUIRED here and it is easy to assume they must be
+      # configured: 12 DaemonSets and 135 emptyDir-carrying pods were measured on 2026-10-17, and a
+      # drain without those three settings refuses to start.
+      #
+      # drainTimeout: 15m — the chart/kured default is 0, meaning INFINITE, and infinite is the wrong
+      # answer here. Measured 2026-10-17, six PodDisruptionBudgets currently allow ZERO voluntary
+      # disruptions, and a plain drain loops on them indefinitely:
+      #   * 3x longhorn-system/instance-manager-* — one per node, `minAvailable: 1` against a
+      #     selector pinned to a single node's single pod, so disruptionsAllowed is STRUCTURALLY 0
+      #     forever. This is the mechanism by which Longhorn's node-drain-policy blocks: the
+      #     longhorn-manager deletes the node's instance-manager PDB only once it judges the drain
+      #     safe. The drain is SUPPOSED to wait on this one.
+      #   * 3x CNPG `<cluster>-primary` (keycloak-db, nextcloud-db, semaphore-db) — `minAvailable: 1`
+      #     against a selector matching exactly one pod (the primary), so again structurally 0 until
+      #     the CNPG operator switches the primary away. All three primaries were on k3s-prod-1.
+      # With a timeout, a drain that is genuinely stuck ABORTS, the lock is released, and kured
+      # retries next period instead of holding the cluster lock forever on a wedged node. 15m is
+      # chosen to be longer than a CNPG switchover and longer than the `unifi` VM's eviction (below)
+      # usually needs, while still being well inside the 3h window.
+      drainTimeout: 15m
+      # Taint a node PreferNoSchedule while its reboot is pending, so the scheduler stops placing new
+      # pods on a node that is about to go away. With concurrency 1 this is a small win (there is no
+      # second rebooting node to shed pods onto) but it is free and it shortens the drain.
+      preferNoScheduleTaint: weave.works/kured-node-reboot
+      # Record reboot state on the Node objects (`weave.works/kured-reboot-in-progress`,
+      # `weave.works/kured-most-recent-reboot-needed`). Off by default. Turned on because the only
+      # other observability kured offers is ONE gauge, and during an incident `kubectl get node -o
+      # yaml` is the thing an operator can reach without Prometheus.
+      annotateNodes: true
+      logFormat: json
+
+    # ── METRICS: BOTH OF THESE MUST BE true OR THE ALERTS IN alerts.yaml CANNOT FIRE ───────────
+    # The chart ships `service.create: false` AND `metrics.create: false`. Leaving either at its
+    # default means no Service, or a ServiceMonitor with nothing to select, so `kured_reboot_required`
+    # is never scraped — and a PrometheusRule over a series that no target produces is INERT: quiet
+    # forever, indistinguishable from healthy. This repo has shipped that exact bug twice
+    # (VeleroNoRecentBackup over an absent series; the inert gotk_* rules in
+    # runbooks/image-automation.md), which is why platform/monitoring/tests/run.sh exists at all.
+    # KuredNotReporting (alerts.yaml) is the standing detector for this pair regressing.
+    # Prometheus selects ServiceMonitors cluster-wide: measured 2026-10-17, `serviceMonitorSelector`
+    # and `serviceMonitorNamespaceSelector` are both `{}`, so no label plumbing is needed here.
+    service:
+      create: true
+      port: 8080
+    metrics:
+      create: true
+      interval: 60s

platform/kured/helmrepository.yaml

@@ -0,0 +1,12 @@
+---
+# HelmRepository for the official kured chart (github.com/kubereboot/charts). Renovate maintains the
+# pinned chart version in helmrelease.yaml via its `flux` manager, whose managerFilePatterns already
+# cover `platform/**` (renovate.json). Same Flux source pattern as platform/headlamp.
+apiVersion: source.toolkit.fluxcd.io/v1
+kind: HelmRepository
+metadata:
+  name: kured
+  namespace: kured
+spec:
+  interval: 1h
+  url: https://kubereboot.github.io/charts

platform/kured/kustomization.yaml

@@ -0,0 +1,36 @@
+---
+# platform/kured — the Kubernetes Reboot Daemon.
+#
+# Rendered by its OWN Flux Kustomization (clusters/base/kured.yaml, dependsOn platform), NOT by a
+# line in platform/kustomization.yaml. The reasoning is the same one the Cilium split-out wrote down
+# and intel-gpu-plugin reused: inside the shared `wait: true` platform tier, this component's health
+# would be indistinguishable from eleven others, the only emergency valve would be suspending the
+# whole tier, and that tier's 5m timeout is wrong for a 3-node DaemonSet rollout. It matters more
+# than usual here, because `platform` gates platform-config which gates every apps wave — so a wedged
+# reboot daemon placed there could stall every application in the cluster. A component whose entire
+# purpose is to make maintenance safer must not be able to cause an outage.
+#
+# `namespace: kured` applies to every resource below, so none of them repeats it. Keep it: the
+# CiliumNetworkPolicy in networkpolicy.yaml is a CR whose namespace kustomize would otherwise leave
+# to chance.
+#
+# NO SECRET OF ANY KIND. kured authenticates to the API server with its own ServiceAccount token and
+# talks to nothing off-cluster — no registry credential, no webhook URL, no notification token. So
+# neither clusters/dev nor clusters/prod adds a SOPS decryption patch for this tier (same as
+# homepage/runbooks/intel-gpu-plugin). If Slack or shoutrrr notifications are ever wanted, the chart
+# takes them via `configuration.notifyUrl`, and THAT would need a SOPS-encrypted Secret plus a
+# decryption patch — do not inline a webhook URL here.
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: kured
+resources:
+  - namespace.yaml
+  - helmrepository.yaml
+  - helmrelease.yaml     # chart installs the DaemonSet / RBAC / SA / Service / ServiceMonitor
+  # default-deny baseline + the Prometheus scrape ingress + the apiserver/Prometheus egress.
+  # The scrape ingress is NOT optional decoration: without it the ServiceMonitor's targets are
+  # discovered and every scrape times out, leaving the TSDB empty and all three rules in alerts.yaml
+  # permanently quiet. See the header in that file for the measured Longhorn precedent.
+  - networkpolicy.yaml
+  # Three rules over kured's single metric, with a companion promtool unit test beside them.
+  - alerts.yaml

platform/kured/namespace.yaml

@@ -0,0 +1,34 @@
+---
+# kured — namespace for the Kubernetes Reboot Daemon DaemonSet.
+#
+# PLATFORM INFRA, NOT AN APPLICATION NAMESPACE, and here that distinction is the difference between
+# a working component and an admission rejection. kured runs `hostPID: true` with a `privileged: true`
+# container (it `nsenter`s /proc/1/ns/mnt to run `/bin/systemctl reboot` on the host). Both are
+# forbidden by platform/kyverno/policies/restrict-host-namespaces.yaml — but that policy, like every
+# other policy in this repo, is scoped by
+#   `namespaceSelector: matchLabels: homelab.keithjackson.dev/app-namespace: "true"`
+# and its own header states the carve-out: "platform components that do [need these] live in platform
+# namespaces that do NOT carry the app-namespace label, so they are out of scope."
+# Verified by reading the policies on this branch, not assumed:
+#   * restrict-host-namespaces.yaml rule `disallow-host-namespaces`     (hostPID)
+#   * restrict-host-namespaces.yaml rule `disallow-privileged-containers` (privileged)
+#
+# We set the label EXPLICITLY to "false" rather than leaving it unset, following
+# platform/intel-gpu-plugin/namespace.yaml and platform/vpn-egress: an unset label and a deliberate
+# "false" are identical to the admission engine but not to a reviewer, and this component's
+# correctness depends entirely on the answer.
+#
+# NO PodSecurity labels. There is no cluster-wide PSA baseline in this repo — `pod-security.*`
+# appears only where a namespace opts IN (conformance/incluster, platform/kyverno/admission,
+# platform/pm-agent), all at `enforce: restricted`. Adding them here would reject this DaemonSet for
+# the same reasons Kyverno would. Leaving them off is the convention for a privileged platform
+# namespace (platform/vpn-ingress/namespace.yaml sets none either).
+#
+# NO `gateway-access` label. kured serves no HTTP to users and attaches no HTTPRoute; its only
+# listener is the :8080 metrics endpoint, reached in-cluster by Prometheus. See networkpolicy.yaml.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: kured
+  labels:
+    homelab.keithjackson.dev/app-namespace: "false"

platform/kured/networkpolicy.yaml

@@ -0,0 +1,125 @@
+---
+# Default-deny baseline (R-NET-01): deny all ingress + egress, allow only DNS. `kured` is a PLATFORM
+# namespace, so Kyverno's generator — scoped to `app-namespace=true` — does not produce this; we ship
+# it explicitly, as crafty/headlamp/homepage/runbooks all do.
+#
+# THESE POLICIES DO APPLY TO kured, UNLIKE platform/vpn-ingress'. That component's identical-looking
+# baseline is honest-but-inert because its DaemonSet runs `hostNetwork: true` and therefore has no
+# Cilium endpoint for any policy to select. kured's chart defaults to `hostNetwork: false` and we
+# keep that default (it needs the host PID namespace to reboot, not the host network stack), so it
+# has a normal pod identity and every rule here is enforced.
+#
+# Cilium enforces the UNION of these policies (allow if ANY policy allows), so the CiliumNetworkPolicy
+# below adds the kube-apiserver path on top of this deny-all floor.
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: default-deny
+  namespace: kured
+  labels:
+    homelab.keithjackson.dev/baseline: default-deny
+spec:
+  podSelector: {}
+  policyTypes:
+    - Ingress
+    - Egress
+  ingress: []
+  egress:
+    - to:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: kube-system
+      ports:
+        - protocol: UDP
+          port: 53
+        - protocol: TCP
+          port: 53
+---
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# PROMETHEUS MUST REACH :8080, OR EVERY RULE IN alerts.yaml IS INERT.
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# This is not a theoretical hazard in this repo, it is a measured one. From the Longhorn entry in
+# conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh: with the ServiceMonitor
+# applied and no NetworkPolicy allowance, all three targets were DISCOVERED and every scrape failed
+# with "context deadline exceeded" after 10.0s — "a drop, not a refusal" — and the TSDB stayed
+# empty. Target discovery succeeding is therefore NOT evidence the scrape works, and the alerts
+# would have been quiet rather than broken. KuredNotReporting is the standing detector for exactly
+# this regression, and it can only work if this policy is correct.
+#
+# THE WHOLE `monitoring` NAMESPACE, NOT THE PROMETHEUS POD LABELS — following the same choice
+# longhorn/networkpolicy-metrics.yaml made and for the same reason: the prometheus-operator owns
+# those pod labels and rewrites them on upgrade (measured 2026-10-17, the pod carries
+# `app.kubernetes.io/name: prometheus` plus five operator-managed labels including a
+# controller-revision-hash), so selecting on them makes the scrape silently fail on a chart bump.
+# The namespace holds only the monitoring stack.
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: kured-allow-prometheus-scrape
+  namespace: kured
+  labels:
+    app.kubernetes.io/name: kured
+spec:
+  podSelector:
+    matchLabels:
+      # Set by the chart's `kured.matchLabels` helper (_helpers.tpl:70-71) with release name `kured`.
+      app.kubernetes.io/name: kured
+  policyTypes:
+    - Ingress
+  ingress:
+    - from:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: monitoring
+      ports:
+        - protocol: TCP
+          # The container port the chart names `metrics` (daemonset.yaml:211-213), from
+          # `configuration.metricsPort`, which we leave at the chart default of 8080 and mirror in
+          # the HelmRelease's `service.port`.
+          port: 8080
+---
+# Egress allows. kured talks to exactly two things and nothing else; both are required and the
+# component is useless without either.
+#
+#   1. THE KUBE-APISERVER — the lock annotation on its own DaemonSet (`weave.works/kured-node-lock`),
+#      cordon/uncordon (`nodes` patch), the node annotations, listing pods, and creating
+#      `pods/eviction` objects during the drain. Reached via the `kube-apiserver` Cilium entity
+#      rather than an IP: on this cluster the API is fronted by a kube-vip VIP that moves between
+#      nodes on every reboot, so any CIDR-based rule here would break during the exact operation
+#      this component exists to perform. Same construct as platform/headlamp/networkpolicy.yaml.
+#
+#   2. PROMETHEUS, on 9090 — the alert gate. The HelmRelease sets
+#      `prometheusUrl: http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090`
+#      and `alertFilterMatchOnly: true`, so before taking the cluster lock kured asks Prometheus
+#      whether any of LonghornVolumeDegraded / LonghornVolumeFaulted / KubeNodeNotReady /
+#      NodeExporterDown is firing. IF THIS EGRESS IS BLOCKED, kured cannot evaluate the gate.
+#      It treats the query error as "do not proceed" and skips the reboot — which is the correct
+#      fail-safe direction, but it fails SILENTLY and indefinitely: reboots simply stop happening
+#      and nothing goes red except, eventually, KuredRebootPending. Do not remove this rule
+#      thinking it is only for metrics; metrics ingress is the policy above, this is the gate.
+#
+# NOTHING ELSE. No egress to the internet (kured pulls no content at runtime), no DNS beyond the
+# baseline above, no Longhorn or etcd access — kured reads Longhorn's health THROUGH Prometheus
+# alerts, never from Longhorn's own API, which is why that allow is absent here.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: kured-allow-egress
+  namespace: kured
+  labels:
+    app.kubernetes.io/name: kured
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: kured
+  egress:
+    - toEntities:
+        - kube-apiserver
+    - toEndpoints:
+        - matchLabels:
+            io.kubernetes.pod.namespace: monitoring
+            app.kubernetes.io/name: prometheus
+      toPorts:
+        - ports:
+            - port: "9090"
+              protocol: TCP

platform/monitoring/node-pressure-dashboard.yaml

@@ -199,7 +199,7 @@ data:
           "id": 15,
           "type": "timeseries",
           "title": "CAUSE 4 — NFS RPC retransmission ratio, per node",
-          "description": "node_nfs_rpc_retransmissions_total / node_nfs_rpcs_total, from /proc/net/rpc/nfs. Alert line at 1% (NodeNFSRetransmissionRateHigh). MEASURED BASELINE IS EXACTLY ZERO on all three nodes over 48h, including throughout the 2026-09-14 spike — so any visible line here means the export at 192.168.2.102, or the path to it, is dropping requests. A flat zero does NOT clear NFS: it only clears the TRANSPORT. A merely SLOW export needs per-mount RPC round-trip time, which arrives with the mountstats collector (see kube-prometheus-stack.yaml) and is the known gap recorded in runbooks/node-io.md.",
+          "description": "node_nfs_rpc_retransmissions_total / node_nfs_rpcs_total, from /proc/net/rpc/nfs. Alert line at 1% (NodeNFSRetransmissionRateHigh). MEASURED BASELINE IS EXACTLY ZERO on all three nodes over 48h, including throughout the 2026-09-14 spike — so any visible line here means the export at 192.168.2.30, or the path to it, is dropping requests. A flat zero does NOT clear NFS: it only clears the TRANSPORT. A merely SLOW export needs per-mount RPC round-trip time, which arrives with the mountstats collector (see kube-prometheus-stack.yaml) and is the known gap recorded in runbooks/node-io.md.",
           "gridPos": {"h": 8, "w": 12, "x": 12, "y": 39},
           "fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [
             {"color": "green", "value": null}, {"color": "red", "value": 0.01}

platform/oauth2-proxy/helmrelease.yaml

@@ -63,6 +63,75 @@ spec:
       # gets oauth2-proxy's HTML sign-in page (HTTP 403) instead of an immediate redirect (docs/15 UX).
       skip-provider-button: "true"
       scope: "openid email profile"
+    # ── Cold-start gate: do not start the proxy until Keycloak's discovery doc answers ──────────
+    # MEASURED on prod 2026-10-17, during the UPS move: oauth2-proxy performs OIDC discovery ONCE at
+    # startup and EXITS 1 if it fails. On a whole-cluster cold boot Keycloak is not serving yet, so
+    # the Gateway answers 503 and the proxy dies:
+    #
+    #   [provider.go:55] Performing OIDC Discovery...
+    #   [main.go:73] ERROR: Failed to initialise OAuth2 Proxy: ... failed to discover OIDC
+    #                configuration: unexpected status "503": no healthy upstream
+    #
+    # It then sits in exponential CrashLoopBackOff long after Keycloak is healthy -- one replica was
+    # still down minutes later and only a manual pod delete cleared it. There is no retry flag; the
+    # process simply does not survive a cold start. A readinessProbe cannot help (the container is
+    # already dead), and CrashLoopBackOff backoff is not tunable per-pod.
+    #
+    # `extraInitContainers` is the chart's OWN documented hook for this -- its commented example in
+    # values.yaml is literally `- name: wait-for-idp` -- and the chart already uses an alpine init
+    # container for its built-in waitForRedis gate, so this adds no new base image to the platform.
+    # A pod in Init does not accrue CrashLoopBackOff backoff and retries promptly, which is the whole
+    # point.
+    #
+    # BOUNDED, AND FAILS OPEN. After ~5 minutes it gives up and starts the proxy anyway: the worst
+    # case is then exactly today's behaviour, never something new. An unbounded wait would trade a
+    # CrashLoop for a pod wedged in Init forever, which is not an improvement.
+    #
+    # SHELL VARIABLES ARE WRITTEN BARE ($i, $url, $(( ))), WITH NO BRACED FORM ANYWHERE. This file is
+    # rendered through Flux envsubst (platform declares postBuild.substituteFrom cluster-vars), which
+    # CONSUMES braced expansions -- so a braced shell variable renders to the empty string and
+    # silently breaks the loop. BASE_DOMAIN below is braced ON PURPOSE: it is a real cluster-vars
+    # knob and must be substituted. Same hazard, and the same convention, as the gitea and nextcloud
+    # OIDC bootstrap hooks. (ci: flux-postbuild-substitution enforces it.)
+    extraInitContainers:
+      - name: wait-for-idp
+        image: docker.io/library/alpine:3.22
+        imagePullPolicy: IfNotPresent
+        command: ["/bin/sh", "-c"]
+        args:
+          - |
+            set -u
+            url=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
+            i=0
+            while [ "$i" -lt 60 ]; do
+              # busybox wget validates against alpine's ca-certificates-bundle, so a real TLS
+              # failure is a non-zero exit here rather than a false pass.
+              if wget -q -O /dev/null --timeout=5 "$url" 2>/dev/null; then
+                echo "[wait-for-idp] Keycloak discovery reachable; starting oauth2-proxy"
+                exit 0
+              fi
+              i=$(( i + 1 ))
+              sleep 5
+            done
+            echo "[wait-for-idp] discovery still unreachable after ~5m; starting anyway so this" >&2
+            echo "[wait-for-idp] gate can never be worse than the un-gated behaviour it replaces" >&2
+            exit 0
+        securityContext:
+          allowPrivilegeEscalation: false
+          runAsNonRoot: true
+          runAsUser: 65534
+          runAsGroup: 65534
+          readOnlyRootFilesystem: true
+          capabilities:
+            drop:
+              - ALL
+        resources:
+          requests:
+            cpu: 10m
+            memory: 16Mi
+          limits:
+            memory: 32Mi
+
     # Metrics for the ServiceMonitor (../oauth2-proxy servicemonitor in platform-config).
     metrics:
       enabled: true

platform/pm-agent/README.md

@@ -22,6 +22,25 @@ requires.
 
 ---
 
+## Why the agent authenticates to Gitea with a local account and a PAT
+
+Not Keycloak, and that is a decision rather than an omission (operator, 2026-10-06): the
+agents only ever talk to Gitea, so a local Gitea account with a scoped personal access token
+is the whole requirement, and it is drastically simpler than federating a machine identity.
+
+It is also what Gitea can actually do. Keycloak service accounts were considered and are not
+available for this: Gitea's API accepts basic auth, PATs, SSH signatures and **its own**
+OAuth2 bearer tokens, and as an OAuth2 provider Gitea implements **only the Authorization
+Code Grant** — there is no client-credentials grant, and nothing in Gitea validates a token
+issued by a third-party OIDC provider. So a Keycloak service-account token cannot
+authenticate to Gitea's API or to git-over-HTTP at all. Keycloak remains what it is for human
+access: the browser login source registered on Gitea (`catalog/gitea.yaml`).
+
+What still matters, and is enforced by the hand-off rather than by Keycloak: **the account
+must not be an administrator** — `infra/gitea/approval-policy.json` authorises
+`/approve-override` on live repository permission alone, so an admin-permission bot could
+waive the sensitive-path human review on its own pull request.
+
 ## It does nothing at all until an operator does two things
 
 Both are deliberate, and both fail **visibly** rather than silently:
@@ -31,6 +50,36 @@ Both are deliberate, and both fail **visibly** rather than silently:
    `infra/agent/build-image.sh` (which verifies it and refuses to push), push it as the
    operator, then update **both** places with the digest `podman push --digestfile` returned,
    in one pull request. Until then: `ImagePullBackOff`.
+
+   **Two things have to be true before any digest will pull, and neither is declared in this
+   repository.** Both were measured after PR #57 merged — this service is the first workload
+   in the cluster ever to reference the in-cluster registry, so nothing had exercised the
+   path. Details and the exact commands:
+   [`decisions/2026-10-05-pm-agent-registry-pull-prereq.md`](../../decisions/2026-10-05-pm-agent-registry-pull-prereq.md).
+
+   - **The nodes resolve the registry hostname to the WAN, so the TLS they fail to verify is
+     the router's — DIAGNOSED 2026-10-06, and fixed in this change.** containerd pulls as a
+     *node* process, so it resolves through the node's public upstreams
+     (`node_baseline_dns_servers` = the LAN gateway) rather than through Pi-hole. The public
+     answer for every name in this zone is a CNAME to `wan.home.keithjackson.dev` — the WAN
+     IP — which this router does not hairpin; it answers :443 with its own placeholder
+     certificate `CN=%D` carrying **no extensions, therefore no SANs**, which is precisely
+     `x509: certificate is not valid for any names`. The Gateway, the wildcard certificate
+     and the digest were never implicated. The fix is the pin this repository already uses
+     for the Keycloak issuer, for the identical reason: `gitea.{{ k3s_base_domain }}` is now
+     in `node_baseline_hosts_pins` — node configuration is Ansible's, declared in the
+     inventory, never typed on a node. Apply it scoped to that one task:
+     `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod
+     --tags hosts`.
+   - **The pull is still unauthenticated** once the node reaches the right endpoint. An
+     unauthenticated client gets nothing from this Gitea's registry, and there is no
+     `imagePullSecret` anywhere in the tree. Consistent with the agent credential decision
+     (local Gitea accounts with PATs), the fix is a `dockerconfigjson` for a local Gitea
+     account whose token carries `read:package`, referenced from the Deployment and the
+     committed worker Job template — not built yet, because minting that token is yours.
+
+   The same two apply to `conformance-runner`: same registry path, same push-then-pin model,
+   never pulled either.
 2. **The credentials are placeholders.** `pm-agent-credentials.sops.yaml` is encrypted to the
    repository's age recipient but holds `REPLACE-ME-…` strings. The hand-off —
    which Gitea account to create, which scopes, and why it must *not* be an administrator —
@@ -96,7 +145,7 @@ Job from a committed template. **The model's output is data, never a command.**
 | **cannot** | merge anything — the gate is the approver, and the service makes no merge call |
 | **cannot** | reach the Kubernetes API: no ServiceAccount token is mounted, the `pm-agent-worker` identity is bound to nothing, and the egress policy does not include the API server. `kubectl` is on its `PATH` (the gate's toolchain needs it) and is inert |
 | **cannot** | reach anything except this Gitea and tcp/443 — no package registry, no GitHub |
-| **cannot** | clone from or push to any host but this Gitea: `pm-worker` validates `PM_GITEA_URL` against an allowlist baked into the image |
+| **cannot** | clone from or push to anywhere but `gitea.gitea.svc.cluster.local:3000`: `pm-worker` validates `PM_GITEA_URL` against a one-entry allowlist baked into the image. In-cluster consumers reach a service by its cluster-local name — a different path by design from how a client outside the cluster reaches it — so the external hostname is not an accepted value |
 | **cannot** | authenticate a `git push` **during the model session**: `GIT_ASKPASS` is unset for its duration, so the restriction is structural rather than a request in a prompt |
 | **cannot** | be created in any shape but the committed one — see the admission policy |
 

platform/pm-agent/pm-agent-credentials.sops.yaml.example

@@ -51,6 +51,15 @@
 # (2) MINT THE GITEA TOKEN as that user (log in as `pm-agent`, or use the admin API to
 #     create a token for it): Settings -> Applications -> Generate New Token.
 #
+#     A LOCAL GITEA ACCOUNT WITH A PAT IS THE DECIDED MODEL (operator, 2026-10-06), not
+#     Keycloak. The agents only ever talk to Gitea, so this is the whole requirement and it
+#     is far simpler than federating a machine identity. It is also the only thing that
+#     works: Gitea's API accepts basic auth, PATs, SSH signatures and its OWN OAuth2 bearer
+#     tokens, and as a provider Gitea implements only the Authorization Code Grant — no
+#     client-credentials grant, and nothing that validates a third-party OIDC token. A
+#     Keycloak service-account token therefore cannot authenticate to Gitea's API or to
+#     git-over-HTTP. Do not "upgrade" this to Keycloak without re-reading that.
+#
 #       Scopes: `write:repository` and `write:issue`. Nothing else — in particular NOT
 #       `write:admin`, NOT `write:organization`, NOT `write:user`.
 #

platform/vpn-ingress/README.md

@@ -13,8 +13,8 @@ egress out the house WAN.
 | Transport | WireGuard, **udp/51820**, kernel implementation |
 | Where it runs | `hostNetwork` **DaemonSet** — one warm listener on every node |
 | Client subnet | `${WG_CLIENT_CIDR}` = `10.90.0.0/24`; `.1` is the server, `.2`–`.254` devices |
-| Authorisation | a device's **public key** is in [`peers.yaml`](./peers.yaml), or it cannot connect |
-| The one secret | the server's private key, Secret `wg-server-key`, **created out of band** |
+| Authorisation | a device's **public key** is in [`peers.conf`](./peers.conf), or it cannot connect |
+| The one secret | the server's private key — **SOPS-encrypted in this tree**, `wg-server-key.sops.yaml` |
 | Exposure | one hand-made WAN port-forward on the SOHO router, off-repo |
 | Enrolling a device | [`infra/vpn-ingress/new-peer.sh`](../../infra/vpn-ingress/new-peer.sh) |
 | Procedure / bring-up | [`runbooks/vpn-remote-access.md`](../../runbooks/vpn-remote-access.md) |
@@ -129,12 +129,18 @@ translations win here.
 
 | half | where | why |
 |---|---|---|
-| peer **public** keys + addresses | [`peers.yaml`](./peers.yaml), **plaintext in git** | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
-| the server's **private** key | Secret `wg-server-key`, **out of band** | the only real secret; an agent must not mint a long-lived private key (CLAUDE.md §6) |
+| peer **public** keys + addresses | [`peers.conf`](./peers.conf), **plaintext in git**, as a content-hashed `configMapGenerator` so an edit rolls the pods | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
+| the server's **private** key | `wg-server-key.sops.yaml`, **SOPS-encrypted in git** | the only real secret (R-SEC-01, spec docs/07). Ships with a `REPLACE-ME` placeholder: an agent must not mint a long-lived private key (CLAUDE.md §6), so the operator `sops set`s it in |
 
 The two are joined only at pod start, by the init container, into a **`medium: Memory` emptyDir** —
 so the assembled config holding the private key never touches any node's disk and dies with the pod.
 
+Both halves are therefore in git, which is the point: a cluster rebuild restores the server key with
+everything else. An out-of-band `kubectl create secret` — the first shape this component shipped in,
+corrected after review — would have put the one credential every enrolled device depends on *outside*
+the GitOps tree, and losing it means re-enrolling every device, because each client profile pins this
+key's public half.
+
 Encrypting the peer register would have hidden exactly the change a reviewer most needs to see: a
 new line granting a new device access to the whole LAN.
 
@@ -160,21 +166,40 @@ NetworkPolicy applies.
 ## Operating it
 
 * **Enrol a device:** `./infra/vpn-ingress/new-peer.sh <name> <octet>` → paste the printed `[Peer]`
-  stanza into `peers.yaml` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
-* **Revoke a device:** delete its stanza → PR → merge. No other step; the key stops working.
+  stanza into `peers.conf` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
+* **Revoke a device:** delete its stanza from `peers.conf` → PR → merge. No other step; the key
+  stops working as soon as the pods roll (the generator's content hash guarantees they do).
 * **Is it up?** `kubectl --context k3s-prod -n vpn-ingress get ds wireguard` — three Ready.
 * **Who is connected?** `kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show`
   (`latest handshake` under ~3 min means live). Never dump `wg showconf` — it prints the private key.
 * **Full bring-up and verification:** `runbooks/vpn-remote-access.md`.
 
+## Measured on prod, 2026-10-06 — what is now known rather than assumed
+
+The first real reconcile settled four things this component's design rested on. Three went the
+right way; one was a defect in my own manifest.
+
+| question | answer |
+|---|---|
+| Does the **kernel module** reach the pod? | **Yes.** `ip link add dev wg0 type wireguard` succeeds, and `/proc/modules` lists `wireguard`, after the `node_baseline` run. `CAP_SYS_MODULE` is correctly not needed. |
+| Does `allowPrivilegeEscalation: false` break the image's **s6 init**? | **No.** The container reaches `wg-quick` normally. The flag stays. |
+| Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
+| Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |
+
 ## Prerequisites outside this directory
 
 1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`
    (`infra/ansible/roles/node_baseline/defaults/main.yml`); the operator runs the role. The pod has
    `NET_ADMIN` but deliberately **not** `SYS_MODULE`, so it cannot load the module itself — it
    CrashLoops with a clear `wg-quick` error instead of quietly degrading to userspace WireGuard.
-2. **The `wg-server-key` Secret.** Runbook step 1. Until it exists the DaemonSet cannot start —
-   the correct fail-closed state for an unenrolled VPN.
+2. **The server private key.** `wg-server-key.sops.yaml` is committed, encrypted, with a
+   `REPLACE-ME` placeholder, and both overlays already carry its decryption patch — so the wiring
+   is complete and the only outstanding act is replacing the value (runbook step 2). Until then the
+   DaemonSet CrashLoops with an explicit FATAL naming the file: the init container validates the
+   value with `wg pubkey`, which rejects the placeholder, an empty value, a public key pasted by
+   mistake, **and** a Secret that arrived as undecrypted `ENC[...]` ciphertext. That last case is
+   the one worth having a guard for — a missing overlay decryption patch applies cleanly and leaves
+   every Flux condition green.
 3. **The WAN port-forward**, udp/51820 → one node address. Hand-made on the SOHO router, off-repo,
    and the half CI can never check.
 
@@ -187,7 +212,7 @@ NetworkPolicy applies.
   covered generically by `KubeDeploymentReplicasMismatch`/`KubePodNotReady` from
   kube-prometheus-stack. **This is the top follow-up.**
 * **No preshared keys.** They would add a symmetric post-quantum hedge on top of the Noise
-  handshake — and would be a real secret, forcing `peers.yaml` into SOPS and costing the
+  handshake — and would be a real secret, forcing `peers.conf` into SOPS and costing the
   reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the
   `wg-server-key` Secret.
 * **No IPv6.** The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6

platform/vpn-ingress/daemonset.yaml

@@ -37,6 +37,10 @@
 #     /etc/sysctl.d/99-k3s-cilium.conf), so no pod sysctl is needed — and `net.ipv4.ip_forward`
 #     is NOT in the kubelet's safe-sysctl allowlist, so a pod-netns version of this would have
 #     required `--allowed-unsafe-sysctls` on all three kubelets (a node-restart change).
+#     COROLLARY LEARNED THE HARD WAY (prod, 2026-10-06): this pod must not merely avoid NEEDING
+#     the sysctl, it must not ATTEMPT it. hostNetwork implies /proc/sys is mounted read-only, so
+#     `sysctl -w` fails with EPERM regardless of capabilities, and wg-quick treats a failed PostUp
+#     as fatal and deletes the interface. Verify the value; never write it.
 #   * no LoadBalancer VIP, so no CiliumL2AnnouncementPolicy lease lottery (the defect that cost
 #     this LAN an 8-day Minecraft outage) and no `externalTrafficPolicy` trade-off at all.
 #   * no double encapsulation. WireGuard's UDP lands on the node's own NIC, so the underlay is a
@@ -104,7 +108,7 @@ spec:
       dnsPolicy: ClusterFirstWithHostNet
       automountServiceAccountToken: false
       # ── Assemble the running config: PUBLIC peer register (git) + PRIVATE server key (Secret) ──
-      # The two halves are deliberately stored apart (peers.yaml explains why) and are only ever
+      # The two halves are deliberately stored apart (peers.conf explains why) and are only ever
       # joined here, at pod start, into a tmpfs that is destroyed with the pod. The server's private
       # key is therefore never written to any node's disk, never committed, and never logged.
       initContainers:
@@ -136,9 +140,51 @@ spec:
               # The key arrives as a Secret file; strip any trailing newline the operator's shell
               # added, or wg-quick rejects the config with "Key is not the correct length or format".
               key=$(tr -d ' \r\n' < /run/wg-server-key/privatekey)
-              if [ -z "$key" ]; then
-                echo "FATAL: /run/wg-server-key/privatekey is empty." >&2
-                echo "       See runbooks/vpn-remote-access.md step 1 (create the server key)." >&2
+              # VALIDATE, don't just check for empty. The Secret ships committed with a REPLACE-ME
+              # placeholder so the GitOps wiring is complete before the key exists, so the common
+              # failure is a value that is present and WRONG — a placeholder, a public key pasted by
+              # mistake, or a SOPS document that reached the cluster undecrypted (ENC[...], which
+              # happens if an overlay is missing its `vpn-ingress` decryption patch). `wg pubkey`
+              # accepts a value only if it is a real 32-byte curve25519 private key, so it rejects
+              # all four. Without this the bad value would be written into wg0.conf and the pod would
+              # fail later and less legibly, or come up with a server nobody can handshake with.
+              if ! printf '%s' "$key" | wg pubkey >/dev/null 2>&1; then
+                echo "FATAL: the server private key in Secret 'wg-server-key' is not a valid" >&2
+                echo "       WireGuard private key (wg pubkey rejected it)." >&2
+                if [ -z "$key" ]; then
+                  echo "       It is EMPTY." >&2
+                elif [ "$key" = "REPLACE-ME" ]; then
+                  echo "       It is still the committed REPLACE-ME PLACEHOLDER — the key has not" >&2
+                  echo "       been generated yet. This is the expected state on a fresh install." >&2
+                else
+                  # `case`, NOT a dollar-brace prefix-strip. This manifest is rendered through
+                  # envsubst, which consumes every dollar-brace expansion it sees, so a shell
+                  # parameter expansion here renders to the empty string and silently disables this
+                  # very branch. `case` uses no braces, so there is nothing to escape.
+                  # (ci: flux-postbuild-substitution caught both the original expansion AND the
+                  # first version of this comment, which spelled the hazard out literally.)
+                  case "$key" in
+                    ENC\[*)
+                      echo "       It is still SOPS CIPHERTEXT (starts with ENC[). The Secret" >&2
+                      echo "       reached the cluster undecrypted — check that this environment's" >&2
+                      echo "       overlay carries the 'vpn-ingress' decryption patch." >&2
+                      ;;
+                  esac
+                fi
+                echo "       Fix: runbooks/vpn-remote-access.md step 2." >&2
+                exit 1
+              fi
+
+              # The pod shares the HOST netns, so this reads the node's real setting. Reading
+              # /proc/sys is permitted even though writing is not (see the PostUp note below).
+              # Fail closed: a tunnel that comes up but cannot forward would look healthy to every
+              # probe here and simply drop every client's traffic.
+              if ! grep -qx 1 /proc/sys/net/ipv4/ip_forward; then
+                echo "FATAL: net.ipv4.ip_forward is 0 on this node, so forwarded VPN traffic would" >&2
+                echo "       be dropped. This pod CANNOT set it: with hostNetwork the runtime mounts" >&2
+                echo "       /proc/sys read-only, because it is the host's global setting." >&2
+                echo "       Fix on the NODE: run the node_baseline role (it writes" >&2
+                echo "       /etc/sysctl.d/99-k3s-cilium.conf) — runbooks/vpn-remote-access.md step 1." >&2
                 exit 1
               fi
 
@@ -155,9 +201,20 @@ spec:
                 echo 'MTU = 1420'
                 printf 'PrivateKey = %s\n' "$key"
                 # ── Forwarding + SNAT, applied in the HOST netns when wg0 comes up ──
-                # ip_forward is already 1 from node_baseline; re-asserted so this component does not
-                # silently depend on another role's sysctl file surviving.
-                echo 'PostUp = sysctl -q -w net.ipv4.ip_forward=1'
+                # NO `sysctl -w net.ipv4.ip_forward=1` HERE. It is IMPOSSIBLE from this pod and it was
+                # the only thing stopping the tunnel coming up — MEASURED on prod 2026-10-06:
+                #     [#] sysctl -q -w net.ipv4.ip_forward=1
+                #     sysctl: permission denied on key "net.ipv4.ip_forward"
+                #     [#] ip link delete dev wg0
+                # and wg-quick then tore the whole tunnel down. The cause is not a missing
+                # capability: with `hostNetwork: true` the runtime mounts /proc/sys READ-ONLY in the
+                # container (confirmed in /proc/mounts: `proc /proc/sys proc ro,...`), precisely
+                # because those sysctls are the HOST's globals and a container must not rewrite
+                # them. CAP_NET_ADMIN does not and cannot lift a read-only mount.
+                # It was never needed anyway: node_baseline writes net.ipv4.ip_forward=1 to
+                # /etc/sysctl.d/99-k3s-cilium.conf, and reading it from this pod returns 1. The
+                # init container now VERIFIES that (a read, which /proc/sys does allow) and fails
+                # closed with a pointed message, which is what the write was pretending to achieve.
                 # SNAT client traffic to the node's address. REQUIRED: without it a LAN host would
                 # reply to 10.90.0.x, find no route, hand the packet to the router, and the router
                 # would drop it — so every connection would hang rather than fail. Matching on
@@ -217,8 +274,10 @@ spec:
             # `kubectl get pod -o wide` shows what this listener owns.
             - { name: wireguard, containerPort: 51820, hostPort: 51820, protocol: UDP }
           securityContext:
-            # NET_ADMIN is the whole privilege budget: create the wg0 link, set the host sysctl, and
-            # write the two netfilter rules above. Everything else is dropped, and the container is
+            # NET_ADMIN is the whole privilege budget: create the wg0 link and write the netfilter
+            # rules above. (It does NOT let the pod set a host sysctl — /proc/sys is read-only under
+            # hostNetwork — which is why nothing here tries.) Everything else is dropped, and the
+            # container is
             # NOT privileged — which also keeps it clear of restrict-host-namespaces' second rule.
             # SYS_MODULE is deliberately NOT granted: the `wireguard` module is loaded by the node
             # (infra/ansible/roles/node_baseline, node_baseline_kernel_modules), so this container
@@ -232,6 +291,16 @@ spec:
                 - NET_ADMIN
             # readOnlyRootFilesystem intentionally UNSET: the image's s6 init writes under /run and
             # /var during startup. Same deferral as crafty/headlamp.
+            #
+            # EXPECTED, HARMLESS STARTUP NOISE — do not "fix" it. Because capabilities are dropped
+            # to NET_ADMIN only, the image's s6 init cannot chown /config (no CAP_CHOWN) and logs a
+            # run of `chown: ... Operation not permitted` plus "Permissions could not be set. This
+            # is probably because your volume mounts are remote or read-only" and a scary "we will
+            # not provide support for it". It is wrong about the cause and it does not matter:
+            # /config is a writable tmpfs emptyDir, everything here runs as root (uid 0) anyway, and
+            # wg-quick proceeds normally — verified on prod 2026-10-06, where the tunnel's only
+            # failure was the sysctl above, well past these warnings. Granting CAP_CHOWN to silence
+            # log noise would widen the privilege set of an internet-facing pod for nothing.
           resources:
             requests:
               cpu: 25m
@@ -272,14 +341,22 @@ spec:
             # during startup rather than at a point anyone would connect to the volume size. Still
             # counted against the container's 128Mi limit, and still far more than wg0.conf needs.
             sizeLimit: 8Mi
-        # The ONLY secret in this component. Created OUT OF BAND by the operator and referenced by
-        # name (R-IDP-04) — runbooks/vpn-remote-access.md step 1. It is NOT a *.sops.yaml in this
-        # tree on purpose: an agent must not mint a long-lived private key (CLAUDE.md §6), and a key
-        # that passed through a session transcript would have to be treated as compromised.
-        # DR: Velero's `daily` schedule excludes only `monitoring` and `unifi`, so this namespace
-        # and this Secret ARE captured off-cluster (platform/velero/helmrelease.yaml). Keep a copy
-        # in the operator's password manager as well — restoring it is what makes every already
-        # enrolled client work again without re-enrolling.
+        # The ONLY secret in this component. SOPS-encrypted IN THIS TREE (R-SEC-01, spec docs/07) as
+        # wg-server-key.sops.yaml, delivered by Flux, referenced here by name only (R-IDP-04).
+        #
+        # It ships committed with a REPLACE-ME placeholder: an agent must not mint a long-lived
+        # private key (CLAUDE.md §6), so the operator generates it and `sops set`s it in
+        # (runbooks/vpn-remote-access.md step 2). Shipping the encrypted placeholder rather than no
+        # file at all is what lets the resource line and both overlay decryption patches be present
+        # and GREEN from the first commit — the alternative, a commented-out resource to uncomment
+        # later, is exactly the "file in git with no generator entry: reviewed, merged, apparently
+        # done, still broken" shape this repo keeps getting bitten by.
+        #
+        # DR: in git means a cluster rebuild restores it with everything else — the property an
+        # out-of-band `kubectl create secret` would NOT have had, and losing this key means
+        # re-enrolling every device because each client profile pins its public half. Velero also
+        # captures this namespace (its `daily` schedule excludes only `monitoring` and `unifi`).
+        # Keep a password-manager copy too: that is the one that survives losing cluster AND backups.
         - name: wg-server-key
           secret:
             secretName: wg-server-key

platform/vpn-ingress/kustomization.yaml

@@ -4,27 +4,66 @@
 # same placement as crafty/headlamp/homepage/runbooks, for the same reason: a leaf whose pod can be
 # legitimately unhealthy must never sit in the gating `platform` tier, where one component's
 # unhealth stalls platform-config and every apps wave behind it (DECISIONS-LOG: "Headlamp gating
-# stall"). That rationale is unusually concrete here: until the operator creates the `wg-server-key`
-# Secret (runbooks/vpn-remote-access.md step 1) this DaemonSet CANNOT start, so putting it in the
-# gating tier would have taken the whole platform down on merge.
+# stall"). That rationale is unusually concrete here: until the operator replaces the `REPLACE-ME`
+# placeholder in wg-server-key.sops.yaml (runbooks/vpn-remote-access.md step 2) this DaemonSet
+# CANNOT start, so putting it in the gating tier would have taken the whole platform down on merge.
 #
 # NOT in platform/kustomization.yaml — see clusters/base/vpn-ingress.yaml.
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
+# REQUIRED for the configMapGenerator below to be wired up at all, not merely tidy. A generated
+# ConfigMap inherits NO namespace, every other object here declares `vpn-ingress`, and kustomize's
+# nameReference transformer will not rewrite a reference across a namespace boundary — so without
+# this the DaemonSet keeps mounting the unhashed name `wg-peers`, which does not exist, and the
+# pods wedge on a missing volume. Verified by building it both ways (kustomize v5.8.1).
+# Same placement as runbooks/kustomization.yaml, which uses `namespace: runbooks` for its own
+# content generator.
+namespace: vpn-ingress
 resources:
   - namespace.yaml
-  # The peer register: which devices may dial in. PLAINTEXT and reviewable on purpose (see the file).
-  - peers.yaml
   # The listener itself: hostNetwork DaemonSet, udp/51820 on every node.
   - daemonset.yaml
   # Namespace default-deny baseline. NOTE: does NOT constrain the hostNetwork DaemonSet — the file
   # says so at length rather than implying a protection it cannot provide.
-  - networkpolicy.yaml
+  - networkpolicy.yaml  # The server's WireGuard private key — the ONE secret here. SOPS-encrypted (R-SEC-01, spec
+  # docs/07), decrypted per-env by the `vpn-ingress` Kustomization, referenced by name only from
+  # daemonset.yaml (R-IDP-04). Its plaintext `.example` sibling carries the operator hand-off --
+  # deliberately NOT named here: conformance/tests/l0-static/no-plaintext-secrets fails any
+  # kustomization that mentions an unencrypted template, because naming one is how a plaintext
+  # Secret gets applied by accident. (It matches comments too. That is correct, not overzealous.)
+  #
+  # SHIPS WITH A `REPLACE-ME` PLACEHOLDER, already encrypted. An agent must not
+  # mint a long-lived private key (CLAUDE.md §6), and the init container validates the value with
+  # `wg pubkey`, so the unfilled state is a loud CrashLoop naming this file, never a server that
+  # quietly cannot handshake.
+  - wg-server-key.sops.yaml
 #
-# DELIBERATELY ABSENT: a Secret. The server's private key (`wg-server-key`) is created out of band
-# by the operator and referenced BY NAME from daemonset.yaml. There is no `*.sops.yaml` here, so
-# this component needs NO SOPS decryption stanza in clusters/dev or clusters/prod — unlike
-# crafty/headlamp/velero. If that key is ever migrated into SOPS, BOTH halves must land together:
-# the file here, and a `decryption:` patch naming the `vpn-ingress` Kustomization in each overlay.
-# Without the patch, kustomize hands Flux the raw ciphertext and the pod mounts a key full of
-# ENC[...] — which applies cleanly and fails only at handshake time.
+# THIS COMPONENT CARRIES AN ENCRYPTED SECRET, so BOTH overlays must patch the `vpn-ingress`
+# Kustomization with a `decryption:` stanza — clusters/dev and clusters/prod both do, added by the
+# same change as this file. That patch is as load-bearing as the file itself: without it kustomize
+# hands Flux the raw ciphertext, the Secret APPLIES CLEANLY full of ENC[...], and nothing in any
+# Flux condition looks wrong. The init container's `wg pubkey` guard exists partly to turn exactly
+# that into a named error rather than a silently non-working tunnel.
+
+# ── THE PEER REGISTER, as a CONTENT-HASHED ConfigMap ─────────────────────────────────────────
+# peers.conf (plaintext, reviewable — see the file) becomes the `wg-peers` ConfigMap. It is a
+# GENERATOR, not a plain resource, and that distinction is load-bearing rather than stylistic:
+# kustomize appends a hash of the content to the generated name and rewrites the DaemonSet's
+# volume reference to match, so adding or revoking a peer changes the pod template and ROLLS THE
+# PODS automatically.
+#
+# A fixed-name ConfigMap (which is what this component shipped first) does not do that. Flux
+# updates it in place, the pod template is byte-identical, no rollout is triggered — and because
+# the init container assembles wg0.conf ONCE at pod start into a tmpfs, the running tunnel keeps
+# the old peer list indefinitely. Measured on prod 2026-10-06: a merged peer reached the
+# ConfigMap while `wg show` listed no peers and the DaemonSet still read generation=1, having
+# never rolled. The runbook's "PR -> merge -> Flux rolls the DaemonSet" was false until this.
+#
+# DO NOT set `disableNameSuffixHash` here. The hash IS the mechanism.
+configMapGenerator:
+  - name: wg-peers
+    files:
+      - peers.conf
+    options:
+      labels:
+        app.kubernetes.io/name: vpn-ingress

platform/vpn-ingress/networkpolicy.yaml

@@ -19,7 +19,7 @@
 # protects this port, and it would be enforcing nothing. So there isn't one.
 #
 # WHAT ACTUALLY PROTECTS THE EXPOSED PORT, then, since the CNI does not:
-#   1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.yaml gets
+#   1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.conf gets
 #      NO REPLY — see README "Why this is safe to expose". This is the real control.
 #   2. The SOHO router's forward: only udp/51820 is forwarded inbound, to one node.
 #   3. The blast radius once authenticated is deliberately the whole LAN — that IS the requirement

platform/vpn-ingress/peers.conf

@@ -1,4 +1,16 @@
----
+# THIS FILE IS THE MOUNTED CONTENT. It is turned into the `wg-peers` ConfigMap by
+# `configMapGenerator` in kustomization.yaml, whose generated name carries a CONTENT HASH — so
+# editing this file changes the ConfigMap name, which changes the DaemonSet's pod template, which
+# ROLLS THE PODS. That is the whole reason it is a generator and not an inline ConfigMap manifest:
+# a fixed-name ConfigMap updates in place, the pod template never changes, no rollout happens, and
+# the init container's wg0.conf (built once at pod start, in tmpfs) keeps the OLD peer list
+# forever. Measured 2026-10-06: a merged peer sat in the ConfigMap while `wg show` listed none and
+# the DaemonSet reported generation=1 — it had never rolled. Same mechanism and same fix as
+# runbooks/kustomization.yaml.
+#
+# Every line here is WireGuard config. `#` is a legal comment in it, so the documentation below
+# travels inside the file that is actually mounted, rather than in a wrapper that gets stripped.
+#
 # ============================================================================================
 # THE PEER REGISTER — every device allowed to dial in, in PLAINTEXT, in git, on purpose.
 # ============================================================================================
@@ -41,33 +53,40 @@
 # the Noise handshake and would be a genuine secret — which would force this register into SOPS
 # and cost the reviewability above. Revisit only if the threat model changes; if it does, the PSK
 # belongs in the `wg-server-key` Secret, not here. See README "Hardening not taken".
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: wg-peers
-  namespace: vpn-ingress
-  labels:
-    app.kubernetes.io/name: vpn-ingress
-data:
-  # Appended VERBATIM to the generated [Interface] block, in this order, at pod start.
-  #
-  # *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. ***
-  # An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of
-  # authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the
-  # DaemonSet will run and listen, and every handshake will be refused because no key is trusted.
-  # The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible
-  # (`wg show` reports an interface with no peers).
-  #
-  # COPY THIS TEMPLATE, uncomment, one block per device:
-  #
-  #   # pixel-8 — Keith's phone, enrolled 2026-10-05
-  #   [Peer]
-  #   PublicKey = <the public key new-peer.sh printed>
-  #   AllowedIPs = 10.90.0.2/32
-  #
-  # Keep the `# <name> — <owner>, enrolled <date>` comment line. It is the only place a key is
-  # tied to a human-recognisable device; `wg show` can only ever print the key itself, so without
-  # it an incident responder cannot tell which stanza to delete.
-  peers.conf: |
-    # No peers enrolled yet. See the template in this file's comments and
-    # ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device.
+#
+# name: wg-peers
+# namespace: vpn-ingress
+# labels:
+# app.kubernetes.io/name: vpn-ingress
+# Appended VERBATIM to the generated [Interface] block, in this order, at pod start.
+#
+# *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. ***
+# An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of
+# authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the
+# DaemonSet will run and listen, and every handshake will be refused because no key is trusted.
+# The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible
+# (`wg show` reports an interface with no peers).
+#
+# COPY THIS TEMPLATE, uncomment, one block per device:
+#
+#   # <device-name> — <owner>, enrolled <YYYY-MM-DD>
+#   [Peer]
+#   PublicKey = <the public key new-peer.sh printed>
+#   AllowedIPs = 10.90.0.<octet>/32
+#
+# The placeholders above are deliberately NOT valid values. An earlier version of this template
+# used a real-looking `10.90.0.2/32`, and new-peer.sh's duplicate-address check — which grepped
+# the whole file — matched it, so `.2` could never be allocated and the script insisted it was
+# already taken. The check now ignores comments; keeping the example unmatchable too means a
+# future grep cannot be fooled either.
+#
+# Keep the `# <name> — <owner>, enrolled <date>` comment line. It is the only place a key is
+# tied to a human-recognisable device; `wg show` can only ever print the key itself, so without
+# it an incident responder cannot tell which stanza to delete.
+#
+# No peers enrolled yet. See the template in this file's comments and
+# ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device.
+# keith-phone — <owner>, enrolled 2026-10-06
+[Peer]
+PublicKey = pBRSHrRyG2hnsuZ6FvHfEzF8hOVTF5stHUb/7fW2Km8=
+AllowedIPs = 10.90.0.2/32

platform/vpn-ingress/wg-server-key.sops.yaml

@@ -0,0 +1,34 @@
+# WireGuard SERVER private key for platform/vpn-ingress — SOPS-ENCRYPTED (R-SEC-01, spec docs/07).
+# Keys stay cleartext for reviewable diffs; only `stringData` values are encrypted (.sops.yaml
+# `encrypted_regex`). Plaintext template + operator hand-off: wg-server-key.sops.yaml.example.
+apiVersion: v1
+kind: Secret
+metadata:
+    name: wg-server-key
+    namespace: vpn-ingress
+type: Opaque
+stringData:
+    #ENC[AES256_GCM,data:fUoPkfWvasisdTtwdhjeOVOlmdQkTl6Bp9YxYyji8g6k+ClATds3BnHJWOj2iplaGpT2XaNakq1+D4YyOi7QbFvNJui+e4qRpqcY8hWnazvZIHJYxjxfAfzW,iv:xtKd6oMZQumAQJAsNLHqJB+o0wdQJ+vL1kI1mc9jL7w=,tag:aImqLgbSmhsRI36/t5kpyg==,type:comment]
+    #ENC[AES256_GCM,data:npDsCW6rQleolIpur5gFkX2jSRrIQzBlGaIPs2Schv0YU3q8AanG39z4QqT3ASDmskV5YuH29Bo0ujEAgBf52YIjQCp58PDDETofrVKOJzhHMa7bV+HCxUFE,iv:7E7thyFBleyqZpqJRGApKvIq3ubCFOkshF/mxim3YPc=,tag:joIrhMe2I6kUnx1ouOPdZw==,type:comment]
+    #ENC[AES256_GCM,data:wHUFx09FgP/2+LTyw61EFbKA5UaqfjU7CeGk6Xc5av2y4r4AvQJRefDM5EGlahKecDFEFsf7fSQRFRHtUJqrQTtw+6tcj3ubSCPW4iJvhyJJv0xdb/0puRmcgj+2NfgF,iv:xgDOSDLqMQ1wcI9vrkKqfNMgIDFEusFnnO7Rzq/aOa4=,tag:m1m/fgjkBsG91c+3daqOpA==,type:comment]
+    privatekey: ENC[AES256_GCM,data:ML8CPNm4JwhQHUTUUxZO/I+9X15BCIGqOb1Uw9EmX5RTAsxQ9N97YvkYlbY=,iv:5ZUsFdMMmeGsy4fNbk6qbJkzweMVWXZ5g6E7NciAGUg=,tag:pAHv3ieLjoEyFbqbWXsVgg==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBMays2OXY0VkpRc2FQQmhU
+            Q0owM3N0VkhYR0lYTUdNZWRqeTFIMjNHOVJ3CnJSMnc5NFEyU0IzWTFtUkw2YVFS
+            MU02VFo1TE14YVpZRzNhVjZwRTZwd0UKLS0tIDFaMWF4NUZjZmhRdXRuZFJYNkg3
+            SUUzN1FnMlI4bkkzbVNlMWhUVnJZODAKQ+Iy86CIv/Oj9E9j4E/0y5hDbcctRuqB
+            OR825KRPRaTdF/G2hudE1R1JbM4wGbPLMzv078R4ZXHDPGvDttSEkA==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-10-06T06:10:31Z"
+    mac: ENC[AES256_GCM,data:6wAGAwWL7jI/NEeswBnUiWoRty+bb5DEsGEAPXajKNNYCwtm5yuXBrcOI/q4w6H8FG+nqVAECkdpAOI3S9mjDfc2zlWAzfCJJdDic92RKPXOC0tOH/M1BBSGJ1fVdJGAZZT1yAVxmtYy9WohJpawv3kmsVv+JnD+TIqmZ1CJHXo=,iv:srs4o0olSwmraQQgmdPxb+4h/t6fHHeFKJ2z914kGgY=,tag:jjoIKhU8Tu9i5SJkvkfcCg==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

platform/vpn-ingress/wg-server-key.sops.yaml.example

@@ -0,0 +1,76 @@
+# =============================================================================
+# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never
+# auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
+#
+# WHAT THIS IS: the WireGuard SERVER's long-term private key for the remote-access VPN
+# (platform/vpn-ingress). It is the ONE secret in the whole component. The init container in
+# daemonset.yaml reads it at `/run/wg-server-key/privatekey` and joins it to the PLAINTEXT peer
+# register (peers.conf) in a tmpfs, so the assembled wg0.conf never touches a node's disk.
+#
+# WHY IT IS IN GIT, SOPS-ENCRYPTED, AND NOT HAND-CREATED. R-SEC-01 / spec docs/07: secrets live in
+# source, encrypted, and reach the cluster only through Flux. An out-of-band `kubectl create secret`
+# would put the single credential every enrolled device depends on OUTSIDE the GitOps tree, where a
+# cluster rebuild silently loses it — and losing it means RE-ENROLLING EVERY DEVICE, because each
+# client profile pins this key's public half.
+#
+# *** THE REAL FILE IS COMMITTED WITH `privatekey: REPLACE-ME`, ALREADY ENCRYPTED. ***
+# That is deliberate, so the wiring is complete and green before the key exists: the resource is
+# already listed in kustomization.yaml and both overlays already carry the `vpn-ingress` decryption
+# patch. Your job is ONLY to replace the value (step 3 below). Until you do, the DaemonSet
+# CrashLoops with an explicit FATAL naming this file — the init container validates the value with
+# `wg pubkey` before writing wg0.conf, so a placeholder fails CLOSED and LOUDLY rather than
+# producing a server that silently cannot complete a handshake.
+#
+# ---------------------------------------------------------------------------------------------
+# EVERYTHING RUNS IN THE DEV CONTAINER. `wg`, `sops` and `qrencode` are all baked into the image
+# (infra/devcontainer/Containerfile) because that image IS the toolchain — CLAUDE.md §5, and the
+# devcontainer README's "Don't install the toolchain on the host". One shell, one toolchain, so
+# `wg genkey | sops set` is just a pipeline.
+# ---------------------------------------------------------------------------------------------
+#
+# HAND-OFF STEPS (operator — an agent must not mint a long-lived private key, CLAUDE.md §6).
+# Full procedure with verification: runbooks/vpn-remote-access.md step 2.
+#
+#  1. Generate the key STRAIGHT INTO this file. The plaintext never lands on disk and never
+#     reaches a terminal; `sops set` edits the already-encrypted document in place:
+#         ./infra/devcontainer/dev.sh run bash -c '
+#           cd /work
+#           umask 077
+#           sops set platform/vpn-ingress/wg-server-key.sops.yaml \
+#             "[\"stringData\"][\"privatekey\"]" "\"$(wg genkey)\""
+#         '
+#     (Or edit it: `./infra/devcontainer/dev.sh run sops platform/vpn-ingress/wg-server-key.sops.yaml`.)
+#
+#  2. Record BOTH halves in your PASSWORD MANAGER. The encrypted file is in git and Velero captures
+#     this namespace to R2, so this is belt-and-braces — but it is the only copy that survives
+#     losing the cluster AND the backups, and the key cannot be regenerated without invalidating
+#     every client profile.
+#         # public half (safe to display; it is what every client profile carries)
+#         ./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d \
+#           --extract "[\"stringData\"][\"privatekey\"]" \
+#           platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey'
+#
+#  3. Confirm the value is a real key without printing it — `wg pubkey` succeeds only on a valid
+#     32-byte curve25519 private key, which is exactly the check the pod's init container runs:
+#         ./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d \
+#           --extract "[\"stringData\"][\"privatekey\"]" \
+#           platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey >/dev/null \
+#           && echo "valid WireGuard key"'
+#
+#  4. PR it. Nothing else to wire: the resource line and both overlay decryption patches already
+#     exist. Flux rolls the DaemonSet and the pods leave CrashLoop.
+#
+# NOTE infra/vpn-ingress/new-peer.sh reads THIS FILE to derive each client profile's
+# `[Peer] PublicKey`, so there is no separate copy of the key to keep in step with anything.
+# =============================================================================
+apiVersion: v1
+kind: Secret
+metadata:
+  name: wg-server-key
+  namespace: vpn-ingress
+type: Opaque
+stringData:
+  # The server's WireGuard private key: `wg genkey` output — 44 chars of base64 ending in '='.
+  # NOT a passphrase, and NOT the public key. The init container validates it with `wg pubkey`
+  # before writing wg0.conf, so a placeholder or a mangled value fails closed with a clear message.
+  privatekey: REPLACE-ME

renovate.json

@@ -1,6 +1,6 @@
 {
   "$schema": "https://docs.renovatebot.com/renovate-schema.json",
-  "_comment": "Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). Renovate owns HELM CHART + DEPENDENCY bumps (opens PRs; allowed classes auto-merge on green CI). Flux image-automation owns per-service container IMAGE TAGS (the $imagepolicy setters). The two do NOT overlap: this config DISABLES image-tag updates so Renovate never fights Flux over a running version. Self-hosted against the bootstrap Gitea; platform: gitea set at runtime by the operator's renovate runner.",
+  "_comment": "Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). AS OF 2026-10-17 RENOVATE OWNS BOTH HALVES: Helm chart/dependency bumps AND per-service container image tags. It opens PRs for everything; allowed low-risk classes auto-merge on green CI. The ADR-0009 split that gave image tags to Flux image-automation is formally closed — see .gitea/workflows/renovate.yaml for the three measured reasons (it pushes commits rather than opening PRs, so it bypasses the CLAUDE.md §1 gate; `main` is protected with user_can_push:false and the flux-system GitRepository has no secretRef, so it cannot write anyway; and enabling it would need an in-cluster git WRITE credential for the repo that defines the cluster). image-automation has been commented out at platform/kustomization.yaml:52 since it was authored and the cluster holds zero ImagePolicy/ImageRepository/ImageUpdateAutomation objects, so nothing is being torn down. CONSEQUENCE: the `docker` and `kubernetes` managers, previously disabled ONLY to avoid fighting that controller, are now ENABLED — without them every plain `image: repo/name:tag` line in this repo was maintained by nothing at all, which is how the oauth2-proxy auth sidecars reached 7 minors behind the oauth2-proxy running in the same cluster. RUNNER: .gitea/workflows/renovate.yaml (scheduled daily); platform/endpoint are set there, not here.",
   "extends": [
     "config:recommended",
     ":dependencyDashboard",
@@ -15,13 +15,24 @@
   "labels": ["dependencies", "automated-update"],
   "ignoreUnstable": true,
   "respectLatest": true,
-  "enabledManagers": ["flux", "kustomize", "helmv3", "regex", "github-actions"],
+  "enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "regex", "github-actions"],
   "flux": {
     "managerFilePatterns": ["/(^|/)platform/.+\\.ya?ml$/", "/(^|/)apps/.+\\.ya?ml$/"]
   },
   "kustomize": {
     "managerFilePatterns": ["/(^|/)kustomization\\.ya?ml$/"]
   },
+  "_kubernetes_comment": "ADDED 2026-10-17. The `kubernetes` manager reads plain `image:` fields out of Kubernetes manifests, and it is NOT auto-detected — it only looks at files matched here, which is why this block is as broad as it is. Scope covers every place this repo writes a container image: platform components, the rendered app waves, the ServiceDefinition catalog that generates them, the runbooks web server and the pm-agent job template. `catalog/` matters as much as `apps/`: apps/wave-*/*.yaml are GENERATED by catalog/render.py, so a bump applied only to the rendered output is reverted by the next render and would also fail ci/render-golden.sh. Renovate will raise BOTH, and both must land together.",
+  "kubernetes": {
+    "managerFilePatterns": [
+      "/(^|/)platform/.+\\.ya?ml$/",
+      "/(^|/)platform-config/.+\\.ya?ml$/",
+      "/(^|/)apps/.+\\.ya?ml$/",
+      "/(^|/)catalog/.+\\.ya?ml$/",
+      "/(^|/)runbooks/serving/.+\\.ya?ml$/",
+      "/(^|/)infra/agent/.+\\.ya?ml$/"
+    ]
+  },
   "customManagers": [
     {
       "customType": "regex",
@@ -60,14 +71,29 @@
       "matchFileNames": ["ci/versions.env", "infra/devcontainer/versions.env"],
       "automerge": false,
       "addLabels": ["ci-tooling"]
+    },
+    {
+      "description": "ADDED 2026-10-17. NEVER AUTO-MERGE THE BLAST-RADIUS COMPONENTS, not even a patch. The default rule above auto-merges minor/patch/digest on a green gate, which is right for an app image: a bad Sonarr bump inconveniences one pod. It is wrong for these five, because the gate is a STATIC check — it renders manifests and lints them, it does not boot a node, drain a volume or form an etcd quorum (vars.CLUSTER_AVAILABLE is unset, so L2/L3 conformance does not run in CI at all). So for anything whose failure mode is the CLUSTER rather than a service, a green gate is not evidence and a human must read the changelog. kured is here because it reboots control-plane nodes; longhorn because a storage-engine bump migrates live volume data; cilium because it is the CNI and a bad bump is a total partition (and its version is DUAL-PINNED in infra/ansible/roles/cilium/defaults/main.yml, which Renovate does not see, so the two would silently desync); cloudnative-pg because it drives Postgres switchovers; kube-prometheus-stack because ci/versions.env asserts an exact promtool/Prometheus lockstep and a chart bump that moves Prometheus fails the gate until that file moves with it.",
+      "matchPackageNames": [
+        "kured",
+        "longhorn",
+        "cilium",
+        "cloudnative-pg",
+        "kube-prometheus-stack"
+      ],
+      "automerge": false,
+      "addLabels": ["needs-review", "blast-radius"]
     }
   ],
+  "_unmanageable_comment": "WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A FLOATING tag has no version to compare, so Renovate skips it silently — it does not pin it and does not warn. As of 2026-10-17 that is: plexinc/pms-docker:latest, lscr.io/linuxserver/jackett:latest, pihole/pihole (NO TAG AT ALL), quay.io/keycloak/keycloak (no tag), nextcloud:31-apache and docker:27-dind. Those must be pinned BY HAND to a concrete version before any automation can maintain them, and until then they drift on every pull with nothing recording what is running. Separately, the HelmRelease minor-locking ranges (9 of 12, e.g. `>=88.0.0 <89.0.0` on kube-prometheus-stack) cap Renovate at patches inside a line it has been told not to leave; raising those ceilings is deliberate per-component work and is NOT part of enabling the bot. Both lists are in decisions/2026-10-17-automated-upgrades.md.",
   "ignorePaths": [
     "**/*.example",
     "**/*.sops.yaml",
     "conformance/**",
-    "platform/image-automation/per-service/**"
+    "platform/image-automation/per-service/**",
+    "platform/kyverno/tests/**"
   ],
-  "docker": { "enabled": false },
-  "kubernetes": { "enabled": false }
+  "_ignorePaths_comment": "platform/kyverno/tests/** ADDED 2026-10-17: those Kyverno policy fixtures pin `alpine:latest` and a `conformance-runner:latest`, which are test inputs whose whole point is to be a fixed shape, not software to keep current. `conformance/**` was already ignored for the same reason; this closes the matching hole that enabling the `kubernetes` manager would otherwise have opened.",
+  "_docker_comment": "The `docker` manager (Dockerfiles, docker-compose) and the `kubernetes` manager above were BOTH previously `enabled: false`, solely so Renovate would not fight Flux image-automation over a running tag. That controller was never enabled and never will be here (see _comment), so the only thing those two flags were achieving was leaving every plain `image:` tag in the repo unmanaged. `docker` is left out of enabledManagers because this repo contains no Dockerfile that ships to the cluster; `kubernetes` is now ON and is the one that matters.",
+  "docker": { "enabled": true }
 }

runbooks/kustomization.yaml

@@ -85,6 +85,9 @@ configMapGenerator:
       - node-io=node-io.md
       - node-memory=node-memory.md
       - node-memory-swap=node-memory-swap.md
+      # Covers all three platform-health.node-reboots rules (KuredRebootPending,
+      # KuredNodeCoverageGap, KuredNotReporting) from platform/kured/alerts.yaml.
+      - node-reboot-automation=node-reboot-automation.md
       # Linked from OAuth2ProxyGateUnobserved, and the diagnostic procedure for any gated app
       # that "is up but does not work" — the gate answers an unauthenticated XHR with a 302,
       # not a 401, so the failure arrives as an HTTP 200 carrying the wrong body.

runbooks/node-reboot-automation.md

@@ -0,0 +1,254 @@
+# Node reboot automation (kured)
+
+**Alerts:** `KuredRebootPending`, `KuredNodeCoverageGap`, `KuredNotReporting`
+**File:** `platform/kured/alerts.yaml`
+**Expressions:** `max by (node) (kured_reboot_required) == 1` / `count(kured_reboot_required) < count(kube_node_info)` / `absent(kured_reboot_required)`
+
+## What this means here
+
+These three alerts cover the OS reboot half of the automated upgrade process. The design is two
+pieces that only work together:
+
+- **`roles/unattended_upgrades`** (Ansible, on the host) installs Debian **security** updates
+  nightly. It never upgrades the kernel, never restarts k3s, and never reboots. When something it
+  installed needs a reboot, it leaves `/var/run/reboot-required` behind.
+- **kured** (`platform/kured`, a DaemonSet) watches that file. When it exists, kured takes a
+  **cluster-wide lock**, cordons the node, drains it, reboots it, and uncordons it — then holds the
+  lock for a further 30 minutes before any other node may go.
+
+**kured does not decide whether to reboot.** On these nodes the sentinel is written almost
+exclusively by a deliberate operator kernel upgrade
+(`playbooks/node-baseline.yml --tags kernel`), because the kernel is blacklisted from unattended
+upgrades. So a pending reboot normally means *a human already asked for this*.
+
+Which alert you have tells you which half is wrong:
+
+| alert | meaning |
+|---|---|
+| `KuredRebootPending` | The mechanism is alive and a reboot was offered, but it did not happen for >36h. **Something is blocking it.** |
+| `KuredNodeCoverageGap` | kured is not running on every node. The uncovered node will never be rebooted, silently. |
+| `KuredNotReporting` | kured is publishing nothing at all. Reboots are not happening, **and the other two alerts are inert.** Treat this as the most serious of the three. |
+
+**All three nodes are control-plane + etcd, so quorum is 2 of 3.** One node down is degraded but
+up; two is a lost cluster. Everything below is written to protect that. `concurrency: 1` plus
+`lockTtl: 0` means a node that hangs on reboot keeps the lock **forever** and no second node is ever
+rebooted behind it. That is deliberate: the cycle stops and pages a human rather than stepping over
+a corpse.
+
+**A reboot is not seamless.** Most workloads here are single-replica, so each reboot cycle takes a
+brief outage of whatever runs on that node — including `coredns` (1 replica), which means a
+cluster-wide DNS gap while it reschedules. See "Expected outage per cycle" below.
+
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
+picks a current-context per invocation, and that default has already changed once. Do not run two
+`dev.sh` invocations at once; they share one container and interfere.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `uncordon`, `rollout`) and `flux reconcile` are
+operator actions, not agent actions. Where this runbook gives a mutating command it is for an
+operator, and it says so.
+
+## First commands
+
+Start here, in order. The first question is always *which half is broken*.
+
+    # 1. Is kured running at all, and on how many nodes?
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds,pod -o wide'
+
+    # 2. Which node is asking for a reboot, and who holds the lock?
+    #    The lock is an annotation on kured's OWN DaemonSet. An empty/absent value means no holder.
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds kured \
+      -o jsonpath="{.metadata.annotations}"; echo'
+
+    # 3. Is a node cordoned, and is a reboot in progress?
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes \
+      -o custom-columns=NAME:.metadata.name,SCHEDULABLE:.spec.unschedulable,TAINTS:.spec.taints'
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes \
+      -o jsonpath="{range .items[*]}{.metadata.name}{\" \"}{.metadata.annotations.weave\.works/kured-reboot-in-progress}{\"\n\"}{end}"'
+
+    # 4. What is kured itself saying? It logs its decision every period (1h).
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured logs ds/kured --tail=80'
+
+    # 5. Is the gate blocking it? These are the ONLY alerts kured blocks on.
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n longhorn-system \
+      get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness' \
+      | grep -v healthy
+
+    # 6. Does the node actually want a reboot? (read-only; SSH as k3s, no sudo needed)
+    ssh -o BatchMode=yes 192.168.2.10 'ls -la /var/run/reboot-required; cat /var/run/reboot-required.pkgs 2>/dev/null'
+
+## Likely causes, most likely first
+
+### 1. The Longhorn gate is holding it (the common `KuredRebootPending` cause)
+
+kured queries Prometheus before taking the lock and refuses to proceed while
+`LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `KubeNodeNotReady` or `NodeExporterDown` is
+firing. This is working as designed and it is the **most likely** cause.
+
+Why it lingers: there are 3 nodes, every volume has 3 replicas, and
+`replica-soft-anti-affinity: false` means replicas may not share a node. So cordoning one node
+leaves nowhere to put a replacement and **every volume goes degraded 2/3** for the duration.
+Recovery is slow on purpose: `replica-replenishment-wait-interval` is 600s before a rebuild even
+starts, and `concurrent-replica-rebuild-per-node-limit` is 1, so ~25 volumes rebuild one at a time.
+
+Check progress, then wait:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n longhorn-system \
+      get volumes.longhorn.io -o json' | python3 -c 'import json,sys,collections; \
+      v=json.load(sys.stdin)["items"]; print(collections.Counter(x["status"]["robustness"] for x in v))'
+
+If volumes are rebuilding, **do nothing**. If a volume is stuck `degraded` or `faulted` with no
+rebuild progressing, that is a storage problem to fix first — see `/runbooks/longhorn-degraded`.
+Rebooting another node while a volume is at 1/3 is how you lose data.
+
+### 2. The drain timed out against a PodDisruptionBudget
+
+Six PDBs currently allow **zero** voluntary disruptions, and two kinds matter:
+
+- **`longhorn-system/instance-manager-*`** (one per node). `minAvailable: 1` against a selector
+  pinned to a single node's single pod, so `disruptionsAllowed` is structurally **always 0**.
+  Longhorn's `longhorn-manager` *deletes* the PDB when its `node-drain-policy`
+  (`block-if-contains-last-replica`) judges the drain safe. The drain is **supposed** to wait here.
+- **CNPG `<cluster>-primary`** (`keycloak-db`, `nextcloud-db`, `semaphore-db`). Also structurally 0
+  until the CNPG operator switches the primary to another instance. If all three primaries are on
+  the node being drained, three switchovers must happen.
+
+kured's `drainTimeout` is 15m: a genuinely stuck drain **aborts**, releases the lock, and is retried
+next period rather than wedging forever. Repeated timeouts show in kured's logs.
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pdb -A -o wide' | awk '$NF==0 || NR==1'
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
+      -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PRIMARY:.status.currentPrimary'
+
+If CNPG will not switch over, an **operator** can move the primary deliberately before the next
+window rather than fighting the drain.
+
+### 3. A node hung on reboot and still holds the lock
+
+`k3s-prod-2` has done this before (2026-09-03: powered, off the LAN ~38 minutes, physical power
+button required). `lockTtl: 0` means the lock does **not** expire, so the cycle stops here —
+correctly, since the cluster is at 2/3 and must not go to 1/3.
+
+Symptoms: one node `NotReady` or absent, `KuredNodeCoverageGap` firing alongside
+`KuredRebootPending`, and the lock annotation naming the missing node.
+
+**Operator recovery:** attend to the node physically. Once it is back and `Ready`, kured clears its
+own lock on the next period. If the node is gone for good and the lock must be cleared by hand,
+that is an operator action:
+
+    # OPERATOR ONLY. Only when you have confirmed the named node is NOT mid-reboot.
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured annotate ds kured \
+      weave.works/kured-node-lock- --overwrite'
+
+If the node was pruned mid-reboot and is left **cordoned** with no daemon to uncordon it:
+
+    # OPERATOR ONLY.
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod uncordon <node>'
+
+### 4. Nothing can write the sentinel (so nothing ever reboots)
+
+If `KuredRebootPending` never fires and reboots never happen even after a kernel upgrade, suspect
+the sentinel writer. On Debian, `/var/run/reboot-required` is created by a hook shipped in
+**`update-notifier-common`** — and it was absent on all three nodes before this automation landed.
+Without it kured is permanently, invisibly idle.
+
+    ssh -o BatchMode=yes 192.168.2.10 'dpkg -l update-notifier-common needrestart unattended-upgrades | tail -4'
+    ssh -o BatchMode=yes 192.168.2.10 'ls /etc/kernel/postinst.d/ /etc/apt/apt.conf.d/ | sort'
+
+Fix by re-running the role, which asserts this explicitly:
+
+    cd infra/ansible && ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify
+
+### 5. kured is not being scraped (`KuredNotReporting`)
+
+The chart defaults `service.create` and `metrics.create` to **false**, and the `kured` namespace
+ships a default-deny NetworkPolicy. If either enablement regressed, or the Prometheus ingress allow
+was dropped, targets are *discovered* and every scrape times out — the TSDB stays empty and all
+three rules go quiet. Longhorn hit exactly this failure on this cluster.
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get svc,servicemonitor,networkpolicy,ciliumnetworkpolicy'
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n monitoring exec \
+      sts/prometheus-kube-prometheus-stack-prometheus -c prometheus -- \
+      wget -qO- "http://localhost:9090/api/v1/targets?state=active" ' | grep -o '"job":"kured[^,]*' | head
+
+### 6. A node is uncovered by the DaemonSet (`KuredNodeCoverageGap`)
+
+kured declares **no tolerations**, because no prod node is tainted. A newly provisioned or
+re-provisioned node carrying a taint gets no kured pod and will never be rebooted.
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes \
+      -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints'
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds kured \
+      -o custom-columns=DESIRED:.status.desiredNumberScheduled,READY:.status.numberReady'
+
+The fix is a PR adding the needed toleration to `platform/kured/helmrelease.yaml` — not a `kubectl
+patch`, which Flux reverts.
+
+## Expected outage per reboot cycle
+
+Stated plainly, because this process is **not** seamless. Every workload at 1 replica on the
+draining node is down until it reschedules and becomes Ready elsewhere. Measured 2026-10-17: **45
+Deployments and 3 StatefulSets run a single replica.** The ones worth knowing about:
+
+- **`kube-system/coredns` — 1 replica.** A cluster-wide in-cluster DNS gap while it moves. This is
+  the single most impactful item and it is **not** fixed by this PR; raising coredns to 2–3 replicas
+  is recommended follow-up work.
+- **`monitoring/prometheus-…-0`, `monitoring/loki-0`, `monitoring/alertmanager-…-0`** — all
+  single-replica StatefulSets on Longhorn RWO volumes, so they additionally wait for volume
+  detach/reattach. **You lose monitoring and alerting for part of the cycle**, including these very
+  alerts.
+- **All three CNPG Postgres primaries** blip during switchover, which means Keycloak, Nextcloud and
+  Semaphore briefly lose their database.
+- **`gitea`, `nextcloud`, `plex`, `jackett`, `qbittorrent`, `radarr`, `sonarr`, `seerr`, `crafty`,
+  `homepage`, `headlamp`, `semaphore`** — each down for its reschedule.
+- **`unifi` (a KubeVirt VM)** has `evictionStrategy: LiveMigrate` but
+  `migrationMethod: BlockMigration`, so evicting it copies the whole root disk over the network.
+  Expect the drain of its node to be slow.
+- **`kube-vip`** fails the control-plane VIP over on every node reboot.
+
+Multi-replica and therefore fine: `cilium-operator`, `oauth2-proxy`, `kyverno-admission-controller`,
+the Longhorn CSI sidecars, `keycloak`, `pihole`, `runbooks`.
+
+## When to escalate or stop
+
+**STOP and do not intervene** if:
+
+- Longhorn volumes are `degraded` **and rebuilding**. This is the gate doing its job. Wait.
+- A single node is mid-reboot and under 15 minutes have passed.
+
+**STOP and get a human to the hardware** if:
+
+- Any node is `NotReady` while another reboot is pending. **Never clear the kured lock in this
+  state** — that is the one action that can take etcd from 2/3 to 1/3 and lose the cluster.
+- Two nodes are down. The cluster has no quorum; this runbook cannot help. Recover hardware first.
+
+**Escalate to a deliberate maintenance decision** if:
+
+- `KuredRebootPending` has been firing for over a week. The reboot is not going to happen on its
+  own; find the blocker above, or take the reboot manually during a watched window.
+- A volume is `faulted`. Fix storage before any further reboot.
+
+**Turn the automation off** if it is misbehaving and you need the cluster stable. The honest way is
+a PR, but for an emergency an operator may suspend the tier — this stops future reboots and leaves
+running nodes untouched:
+
+    # OPERATOR ONLY. Stops kured being reconciled; does NOT uncordon a node mid-cycle.
+    infra/devcontainer/dev.sh run bash -c 'flux --context k3s-prod suspend kustomization kured'
+
+Check afterwards whether a node was left cordoned (First commands, step 3) and uncordon it if so.
+Resume with `flux resume kustomization kured`. Nothing in the graph depends on `kured`, so
+suspending it stalls nothing else.
+
+---
+
+Source of truth: `runbooks/node-reboot-automation.md`. Index: `/runbooks/`
+
+Records: `decisions/2026-10-17-automated-upgrades.md` (design + what was not verified),
+`platform/kured/helmrelease.yaml` (every tuned value and why),
+`infra/ansible/roles/unattended_upgrades/README.md` (the package half, and the kernel exclusion).

runbooks/pm-agent.md

@@ -67,7 +67,9 @@ Work through these in order; the first three are by far the most likely.
 | Symptom in the log | Cause | Fix |
 |---|---|---|
 | `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
-| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) |
+| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) — but read the row below first |
+| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
+| `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours |
 | `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` |
 | `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |
 | `cycle: 2 worker(s) active` and nothing new | at `PM_MAX_ACTIVE_WORKERS` | wait, or raise it (RAM is the binding constraint: each worker's limit is 2Gi) |

runbooks/vpn-remote-access.md

@@ -21,7 +21,7 @@ behaves like a machine on the LAN: full reach to `192.168.2.0/24`, the Gateway's
 DNS, and internet egress out the house WAN.
 
 Authorisation is one thing only: the device's **public key is in
-`platform/vpn-ingress/peers.yaml`**. An unlisted key gets no reply at all.
+`platform/vpn-ingress/peers.conf`**. An unlisted key gets no reply at all.
 
 > **An enrolled device is as privileged as a machine sitting in the house** — including the three
 > k3s nodes (`192.168.2.10/.11/.12`: apiserver 6443, kubelet 10250, SSH 22). That is what "reach
@@ -34,10 +34,26 @@ deliberately **not** granted `CAP_SYS_MODULE`, so it cannot autoload the module.
 `wg-quick` fails outright and the pod CrashLoops.
 
 ```bash
-# operator action (Ansible is not an agent action)
-cd infra/ansible && ./site.sh --tags system --limit k3s_prod
+# operator action (Ansible is not an agent action). Ansible lives ONLY in the pinned toolbox,
+# and ansible.cfg is only picked up from infra/ansible, so cd there inside the container.
+./infra/devcontainer/dev.sh run bash -c \
+  'cd /work/infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags system'
 ```
 
+Three details in that command are load-bearing, not style:
+
+- **`-e k3s_hosts=k3s_prod`.** `node-baseline.yml` targets `k3s_servers` by default, which is the
+  **dev** group — and dev was decommissioned, so an unqualified run does nothing useful. This is
+  the targeting pattern `infra/ansible/README.md` "Targeting prod" documents for every playbook.
+- **`--tags system`, and never an untagged run.** The two kernel-module tasks carry the `system`
+  tag. An untagged run would also execute the `disks` tasks, which assert on block-device variables
+  prod has deliberately left as placeholders (`infra/PROD-PROVISIONING.md` §2) — the playbook's own
+  header warns about this. `--tags system` is broader than the modules alone (it also re-asserts
+  sysctls, iscsid, qemu-guest-agent and the swap masking) but every one of those is idempotent and
+  already in its desired state on a live node.
+- **It is a playbook, not a wrapper script.** There is no `site.sh` in this repo; `playbooks/site.yml`
+  is the full bring-up chain and is the wrong tool here.
+
 Verify on each node — the module is in-tree since kernel 5.6, so nothing is installed, only loaded:
 
 ```bash
@@ -46,46 +62,109 @@ lsmod | grep '^wireguard'
 grep -w wireguard /etc/modules-load.d/k3s-cilium-longhorn.conf
 ```
 
-## Step 2 — create the server key (once)
+## Step 2 — fill in the server key (once)
 
-This is the **only** secret in the component, and the one thing that must never be generated in an
-agent session or pasted into a transcript. Generate it on your own machine:
+The key is **SOPS-encrypted in this tree** (R-SEC-01, spec docs/07), not created by hand:
+`platform/vpn-ingress/wg-server-key.sops.yaml`. It is already committed — encrypted, with a
+`REPLACE-ME` placeholder — and it is already listed in the component's `resources:` with a
+`decryption:` patch in both overlays. **Nothing needs wiring. You are only replacing one value.**
+
+Why in git rather than `kubectl create secret`: an out-of-band Secret puts the one credential every
+enrolled device depends on outside the GitOps tree, where a cluster rebuild silently loses it — and
+losing it means re-enrolling every device, because each client profile pins this key's public half.
+
+Everything below runs **inside the dev container** — `wg`, `sops` and `qrencode` are all baked into
+the image (CLAUDE.md §5: the toolchain lives only in the pinned container; don't install it on the
+host). One shell, one toolchain.
+
+**1. Generate the key straight into the encrypted file.** One pipeline: `wg genkey` makes it,
+`sops set` writes it into the already-encrypted document in place. The plaintext never lands on
+disk and never reaches a terminal.
 
 ```bash
-mkdir -p ~/.config/wireguard-home && chmod 700 ~/.config/wireguard-home
-umask 077
-wg genkey > ~/.config/wireguard-home/server.key          # keep this; see "Back it up" below
+./infra/devcontainer/dev.sh run bash -c '
+  cd /work
+  umask 077
+  sops set platform/vpn-ingress/wg-server-key.sops.yaml \
+    "[\"stringData\"][\"privatekey\"]" "\"$(wg genkey)\""
+'
 ```
 
-Load it into the cluster. **Do not echo it, do not pass it on an interactive command line that
-lands in shell history, and never print the resulting Secret's `.data`** (CLAUDE.md §6):
+**2. Record it in your PASSWORD MANAGER.** Derive both halves from the file you just wrote:
 
 ```bash
-./infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n vpn-ingress create secret generic wg-server-key \
-     --from-file=privatekey=/work/.cache/server.key'
+# the PUBLIC half — safe to display, and what every client profile carries
+./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
+  platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey'
+
+# the PRIVATE half — paste into the password manager, then clear your scrollback
+./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
+  platform/vpn-ingress/wg-server-key.sops.yaml'
 ```
 
-…where `/work/.cache/server.key` is a path you copied the key to **inside the repo's gitignored
-cache** (or mount it however you prefer — the requirement is only that the bytes never pass through
-a log). Delete that copy afterwards: `shred -u .cache/server.key`.
+The second command is the one place this key is deliberately shown. The encrypted file is in git and
+Velero captures this namespace to R2, so the password-manager copy is belt-and-braces — but it is
+the only copy that survives losing the cluster *and* the backups, and the key cannot be regenerated
+without invalidating every client profile.
 
-Confirm it exists **by key name only**:
+**3. Confirm the value is a real key**, without printing it — if `wg pubkey` prints a key, the
+value is valid; if it errors, it is still wrong:
 
 ```bash
-./infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n vpn-ingress get secret wg-server-key -o jsonpath="{.data}" | tr "," "\n" | cut -d: -f1'
-# expected: {"privatekey   -- a NAME. If you see base64 payload, you printed the key. Rotate it.
+./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
+  platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey >/dev/null && echo "valid WireGuard key"'
 ```
 
-**Why out of band and not SOPS.** An agent must not mint a long-lived private key, and a key that
-passed through a session transcript would have to be treated as compromised. The trade-off is that
-this Secret is **not reconstructible from git** — so:
+That is the same check the pod's init container runs, so a pass here predicts a pass there.
 
-**Back it up.** Keep `server.key` in your password manager. Velero's `daily` schedule excludes only
-`monitoring` and `unifi` (`platform/velero/helmrelease.yaml`), so this namespace *is* captured
-off-cluster to R2 as well. Restoring this one key is what makes every already-enrolled device work
-again after a cluster rebuild — **losing it means re-enrolling every device.**
+**4. PR it, and merge.** Flux rolls the DaemonSet and the pods leave CrashLoop.
+
+If you would rather use an editor than `sops set`:
+`./infra/devcontainer/dev.sh run sops platform/vpn-ingress/wg-server-key.sops.yaml` (replace
+`REPLACE-ME`, save, exit — it re-encrypts on write).
+
+### What the pod tells you if this went wrong
+
+The init container validates the value with `wg pubkey` before writing `wg0.conf`, and names the
+specific failure in its log — so you never get a server that silently cannot complete a handshake:
+
+| pod log says | meaning |
+|---|---|
+| `still the committed REPLACE-ME PLACEHOLDER` | step 3 not done (the expected fresh-install state) |
+| `It is EMPTY` | the value was blanked |
+| `still SOPS CIPHERTEXT (starts with ENC[)` | the Secret reached the cluster undecrypted — this environment's overlay is missing its `vpn-ingress` decryption patch. **This is the dangerous one:** the Secret applies cleanly and every Flux condition stays green |
+| `wg pubkey rejected it` with none of the above | a mangled value, or a *public* key pasted by mistake |
+| `net.ipv4.ip_forward is 0 on this node` | the node_baseline role has not run here (step 1). The pod **cannot** set this itself — under `hostNetwork` the runtime mounts `/proc/sys` read-only, because it is the host's global setting |
+| `0 peer(s)` after you merged a peer | the DaemonSet did not roll. Check its `generation` (above). Pre-2026-10-06 this was guaranteed, because the register was a fixed-name ConfigMap; if it recurs, something has reverted `peers.conf` to a plain resource instead of a `configMapGenerator` |
+
+### What a healthy start looks like
+
+Measured on prod 2026-10-06. The init container logs one line, naming the shape and never the key:
+
+```
+wg0.conf assembled: 0 peer(s), server 10.90.0.1/24, udp/51820.
+```
+
+then the `wireguard` container brings the tunnel up:
+
+```
+**** Found WG conf /config/wg_confs/wg0.conf, adding to list ****
+**** Activating tunnel /config/wg_confs/wg0.conf ****
+[#] ip link add dev wg0 type wireguard
+[#] wg addconf wg0 /dev/fd/63
+[#] ip -4 address add 10.90.0.1/24 dev wg0
+[#] ip link set mtu 1420 up dev wg0
+```
+
+**Ignore the `chown: ... Operation not permitted` run before that**, and the image's own
+"Permissions could not be set ... we will not provide support for it". Capabilities are dropped to
+`NET_ADMIN` only, so its s6 init has no `CAP_CHOWN`; `/config` is a writable tmpfs, everything runs
+as root, and `wg-quick` proceeds. It is wrong about the cause and it does not matter.
+
+**A pod stuck `0/1 Running` with `Startup probe failed: Unable to access interface: No such device`
+means `wg-quick` tore the tunnel down** — read the `wireguard` container log for the `[#]` line it
+died on. That is the shape of a real failure; the probe is doing its job of refusing to call a
+server without a `wg0` "ready".
 
 ## Step 3 — forward udp/51820 on the router (once)
 
@@ -126,17 +205,37 @@ unauthenticated traffic either way. If the pod is Ready, the interface exists.
 ## Step 5 — enrol a device
 
 ```bash
-./infra/vpn-ingress/new-peer.sh pixel-8 2              # full tunnel  (LAN + WAN egress)
-./infra/vpn-ingress/new-peer.sh thinkpad 3 --split     # split tunnel (LAN only)
+# in the dev container, like everything else
+./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh pixel-8 2'            # full tunnel
+./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh thinkpad 3 --split'   # LAN only
 ```
 
-It generates the keypair **on your machine**, writes a `0600` client profile outside the checkout,
-prints a QR code if `qrencode` is installed, and prints the `[Peer]` stanza to add. Then:
+It derives the server's public key from `wg-server-key.sops.yaml` itself, so there is no local key
+copy to keep in step with anything. Profiles land in `.cache/wg-peers/` — gitignored, and persistent
+because the repo is the container's only read-write mount.
 
-1. Paste the stanza into `platform/vpn-ingress/peers.yaml` under `data.peers.conf`, keeping the
+It generates the device's keypair, writes a `0600` client profile, prints a QR code, and prints the
+`[Peer]` stanza to add. It refuses to write a profile anywhere git does not ignore.
+
+Then:
+
+1. Append the stanza to `platform/vpn-ingress/peers.conf`, keeping the
    `# <name> — <owner>, enrolled <date>` comment. `wg show` can only ever print the key, so that
    comment is the only record of which stanza belongs to which device.
-2. PR → merge. Flux rolls the DaemonSet (~1 min).
+2. PR → merge. Flux rolls the DaemonSet (~1 min), because `peers.conf` is a
+   **content-hashed** `configMapGenerator`: editing it changes the ConfigMap's generated name,
+   which changes the pod template, which forces the roll. (It did **not** work this way at first —
+   a fixed-name ConfigMap updates in place, the pod template never changes, and the init container
+   keeps the old peer list forever. Measured on prod 2026-10-06.) Confirm it actually rolled:
+
+   ```bash
+   ./infra/devcontainer/dev.sh run bash -c \
+     'kubectl --context k3s-prod -n vpn-ingress get ds wireguard \
+        -o jsonpath="{.metadata.generation}{\"/\"}{.status.observedGeneration}{\"\n\"}"'
+   ```
+
+   The generation must have **increased** since before the merge. If it is still `1`, the roll did
+   not happen and `wg show` will list no peer no matter how long you wait.
 3. Get the profile onto the device (QR scan for phones; "Import tunnel from file" on
    Windows/macOS; `/etc/wireguard/<name>.conf` + `wg-quick@<name>` on Linux).
 4. **Delete the profile from your machine**: `shred -u <path>`. It is the only copy of that
@@ -176,7 +275,7 @@ if the router is not delivering the packets, and a `handshake` with no completed
 
 ## Revoking a device
 
-Delete its `[Peer]` stanza from `peers.yaml`, PR, merge. Nothing else — the key stops working on the
+Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the
 next reconcile. **Never re-use that address octet**: an old profile would silently start working
 again as a different device.