merge: 35 runbooks, served, with a guard that they resolve

All 46 runbook_url annotations pointed at https://<apex>/runbooks/<name>
and every one returned 404 — there was no runbooks/ directory and no route.
Every alert that reached a human handed them a dead link at the moment they
needed it.

Runbooks are now served from their own non-gating tier at the apex prefix,
PUBLIC on purpose: gating them behind Keycloak would make keycloak-down
unreadable exactly when Keycloak is down, which is a loop with no exit. The
new L0 guard checks both directions — a runbook_url with no file, and a file
not wired into the generator — and it caught a live one on contact with
main: CNPGPrimariesCoLocated had shipped hours earlier pointing at a runbook
that did not exist.

.gitea/workflows/scripts/install-tools.sh

@@ -39,6 +39,21 @@ install_chainsaw() {
   curl -fsSL "$url" | tar -xz -C "$bindir" chainsaw
 }
 
+# promtool — the alert-rule UNIT-TEST engine the L0 check
+# conformance/tests/l0-static/alert-rules-unit-tested runs. Installed for the l0 stage because
+# without it that check exits 3 (SKIPPED, never green) on the runner and the alert unit tests
+# would be enforced nowhere — the exact "coverage that looks present" failure it exists to end.
+# Pinned from ci/versions.env to the Prometheus version prod runs (promtool IS the rule engine);
+# the check asserts `promtool --version` equals that pin, so a drifted install FAILS, not skips.
+install_promtool() {
+  have promtool && return 0
+  log "promtool ${PROMTOOL_VERSION} (from the prometheus release tarball)"
+  local tarball="prometheus-${PROMTOOL_VERSION}.${os}-${arch}"
+  curl -fsSL "https://github.com/prometheus/prometheus/releases/download/v${PROMTOOL_VERSION}/${tarball}.tar.gz" \
+    | tar -xz -C "$bindir" --strip-components=1 "${tarball}/promtool"
+  chmod +x "${bindir}/promtool"
+}
+
 install_kubectl() {
   have kubectl && return 0
   log "kubectl ${KUBECTL_VERSION}"
@@ -50,10 +65,10 @@ have() { command -v "$1" >/dev/null 2>&1; }
 
 case "$stage" in
   lint)        install_yamllint ;;
-  l0)          install_yamllint; install_check_jsonschema; install_kyverno ;;  # schema (check-jsonschema) + Kyverno L0
+  l0)          install_yamllint; install_check_jsonschema; install_kyverno; install_promtool ;;  # schema (check-jsonschema) + Kyverno L0 + alert unit tests
   l1)          install_chainsaw ;;              # L1 render via conformance runner (Chainsaw)
   conformance) install_chainsaw; install_kubectl ;;   # trigger the in-cluster Job
-  all)         install_yamllint; install_check_jsonschema; install_kyverno; install_chainsaw; install_kubectl ;;
+  all)         install_yamllint; install_check_jsonschema; install_kyverno; install_promtool; install_chainsaw; install_kubectl ;;
   *) echo "unknown stage: $stage" >&2; exit 2 ;;
 esac
 log "stage '${stage}' tools ready"

DECISIONS-LOG.md

@@ -2688,3 +2688,184 @@ removing the readiness gate, or shortening `for:` each makes a test *fail*, so t
 - **No `runbooks/` tree exists** anywhere in the repo, yet 20+ alerts carry `runbook_url`.
   `cnpg-primary-spread` follows that existing (aspirational) convention; the rule's `description`
   carries the full actionable procedure so the missing page is not load-bearing.
+
+## Remote Path Mappings: the arrs can finally find what qBittorrent finished — 2026-09-12
+
+The previous session gave qBittorrent a writable download destination (`/downloads` =
+`192.168.2.102:/mnt/Storage/plex/downloads`, `subPath: downloads` of the export sonarr/radarr mount at
+`/media`). That closed "nowhere to write". It did **not** close "the two sides disagree about the name
+of the same inode": qBittorrent calls a finished release `/downloads/<name>`, sonarr/radarr call the
+same bytes `/media/downloads/<name>`. Completed Download Handling looks up the path the client
+reports, so the download finished and then nothing imported.
+
+### What was actually broken was bigger than the path
+
+Before touching path mapping I checked the prerequisite the brief asked for — and the arrs had **no
+download client at all**:
+
+```
+sonarr  GET /api/v3/downloadclient -> []
+radarr  GET /api/v3/downloadclient -> []
+```
+
+So "Completed Download Handling fails" was, strictly, "Completed Download Handling was never wired
+up". Reachability itself was fine, and the `dependencies: []` worry was unfounded in the direction
+that matters: the catalog's `qbittorrent` entry is default-deny **egress**, but sonarr and radarr each
+declare `- to: qbittorrent / port 8080`, and the renderer emits the reciprocal ingress allows
+(`qbittorrent-allow-sonarr-8080`, `qbittorrent-allow-radarr-8080` in `apps/wave-0/qbittorrent.yaml`).
+Measured from inside each pod:
+
+| from | DNS | HTTP |
+|---|---|---|
+| sonarr | `qbittorrent.qbittorrent.svc.cluster.local` -> 10.43.121.152 | `403 Forbidden`, connect 0.0012 s |
+| radarr | same | `403 Forbidden`, connect 0.0036 s |
+
+`403` is the *good* answer here: TCP connected and qBittorrent's own WebUI answered, demanding auth.
+Also note `Service/qbittorrent` port 8080 -> `targetPort 8080`, i.e. straight to the app container;
+only 4180 goes through oauth2-proxy. The arrs reach the API, not the SSO proxy.
+
+### What was configured (in each arr's own database, via its HTTP API)
+
+Both, identically, built from each app's own `/api/v3/downloadclient/schema` rather than a hand-typed
+body:
+
+- **Download client** `qBittorrent` (`implementation: QBittorrent`), host
+  `qbittorrent.qbittorrent.svc.cluster.local`, port 8080, no TLS, user `admin`, category
+  `tv-sonarr` / `radarr`, `enable: true`, `removeCompletedDownloads: true`.
+- **Remote Path Mapping** host `qbittorrent.qbittorrent.svc.cluster.local`,
+  remote `/downloads/` -> local `/media/downloads/`.
+
+**Idempotent by construction, and proven so by re-running.** The script reads the existing collections
+first and matches on identity, not on count: a client is a duplicate if
+`implementation == "QBittorrent"` *and* its `host` field equals the target host; a mapping is a
+duplicate if `host` matches *and* `remotePath` matches after trailing-slash normalisation. First run
+created (`HTTP 201` x4). Second run printed `IDEMPOTENT SKIP` for all four and created nothing.
+
+One honest wrinkle in the second run: `POST /downloadclient/test` returned `400` with
+`{"propertyName":"Name","errorMessage":"Should be unique"}` — that is name-collision validation
+against the client already saved, **not** a connectivity failure. The right way to test a persisted
+client is `POST /api/v3/downloadclient/testall`, which tests by id:
+
+```
+sonarr  -> [{"id":1,"isValid":true,"validationFailures":[]}]
+radarr  -> [{"id":1,"isValid":true,"validationFailures":[]}]
+```
+
+### Proof of the chain, with a negative control
+
+An 8 MiB payload was written **at qBittorrent's own path** as
+`/downloads/Test.Show.S01E01.1080p.WEB-DL.x264-CLAUDETEST/...mkv` (+ a movie-shaped twin), owned
+`1000:3000`. Same inode from both sides, which is the whole point:
+
+```
+qbittorrent /downloads/.../Test.Show...mkv        dev=2097261 inode=36144 size=8388608
+sonarr      /media/downloads/.../Test.Show...mkv  dev=2097261 inode=36144 size=8388608
+radarr      /media/downloads/.../Test.Movie...mkv dev=2097361 inode=34783 size=8388608
+```
+
+(The differing `dev` for radarr is a separate NFS mount instance on a different node; the inode is the
+identity that matters.) And neither arr can see the literal remote path — the defect, stated:
+`ls -d /downloads/` -> `No such file or directory` in both.
+
+A **completed torrent was then made real**, because a mapping is only exercised when the client
+*reports* a path. A minimal trackerless, `private: 1` .torrent describing that exact 8 MiB file was
+generated and added with `savepath=/downloads`, `category=tv-sonarr`, `skip_checking=true`,
+`stopped=true` — private + no trackers means qBittorrent performed **no announce, no DHT, no PEX**, so
+nothing left the cluster. qBittorrent then reported it as genuinely finished:
+
+```
+state=stoppedUP progress=1 completed=8388608
+content_path=/downloads/Test.Show.S01E01.1080p.WEB-DL.x264-CLAUDETEST/...mkv
+```
+
+Then `RefreshMonitoredDownloads`, with and without the mapping. **This is the load-bearing evidence:**
+
+| Sonarr's own `GET /api/v3/queue` | `outputPath` |
+|---|---|
+| mapping **deleted** | `/downloads/Test.Show...CLAUDETEST/...mkv` — a path that does not exist for Sonarr |
+| mapping **present** | `/media/downloads/Test.Show...CLAUDETEST/...mkv` — the real file, inode 36144 |
+
+Same torrent, same client, one row of app state between "unresolvable" and "resolved". A
+`GET /api/v3/manualimport?folder=/media/downloads/<release>` on each arr independently parsed the file
+through the local side of the mapping — `quality: WEBDL-1080p`, `releaseGroup: CLAUDETEST` — so the
+scan resolves where it previously could not.
+
+**What was NOT proven: an actual import.** Both manual-import scans end in a permanent rejection —
+`Unknown Series` (sonarr) / `Unknown Movie` (radarr) — and the queue row says `Series title mismatch`.
+That is not a path failure; it is an **empty library**: `GET /api/v3/series` -> `0`,
+`GET /api/v3/movie` -> `0`. No file was hardlinked into `/media/tv` as part of this work, and nothing
+should be until the library is populated. Hardlinkability across `/media/downloads -> /media/tv` was
+already proven live last session (same device, link count 2, no EXDEV).
+
+All test state removed and verified: torrent deleted from qBittorrent with `deleteFiles=false` then
+the payloads `rm -rf`'d; `ls | grep -c CLAUDETEST` -> `0` from qBittorrent, sonarr **and** radarr;
+`torrents/info` -> `[]`; sonarr queue `totalRecords: 0`. The two download clients and two mappings
+survive, re-verified after cleanup.
+
+### THE WART: this is app state, not git
+
+**A rebuild from git alone does NOT restore any of it.** The download clients and the Remote Path
+Mappings live in `sonarr.db` / `radarr.db` on each app's Longhorn config PVC. `catalog/*.yaml` and
+`apps/wave-1/*.yaml` describe the *pods, volumes and network policy* — nothing in this repo describes
+"sonarr has a download client" or "here is how to translate its paths". Restore the cluster from git
+with empty PVCs and the import chain is silently back to broken, with no gate failing. The same is
+true of the symmetric option: qBittorrent's `SavePath` lives in
+`/config/qBittorrent/qBittorrent.conf` on **its** config PVC, which is why the disagreement exists at
+all — two apps, two private databases, two sources of truth, and git is neither.
+
+Two ways out, and they are not equal:
+
+1. **Path agreement** — mount the `downloads` subPath at `/media/downloads` in the qbittorrent pod
+   instead of `/downloads`, and repoint `Session\DefaultSavePath` / `Downloads\SavePath` to match.
+   Then every component names the same inode the same way and **the mappings become unnecessary**.
+   Half of it is declarative: `mountPath` is a line in `catalog/qbittorrent.yaml`, i.e. in git and
+   gate-covered. The other half is still a write to qBittorrent's config PVC — but a *one-time* write
+   whose result git then enforces the shape of, rather than two rows per arr that git never mentions.
+2. **A reconciler** — a Job/CronJob that PUTs the desired clients + mappings into each arr's API from
+   a committed manifest (the `*arr-config`/Recyclarr pattern). Honest and general, restores on
+   rebuild, works for indexers and root folders too — but it is new machinery, needs each arr's API
+   key as a secret, and adds a component to own.
+
+**Recommendation: (1) path agreement, and I have not done it.** It deletes the failure mode instead of
+compensating for it, and it moves the load-bearing part into git where this repo's gate can see it.
+But it changes a running download client's save path — the exact class of change the deferral in
+`catalog/qbittorrent.yaml` shows is easy to get wrong — and it is a design decision, not a bug fix, so
+it is **put to the operator, not taken unilaterally**. The mappings asked for are in place and working
+today either way; if (1) is approved they simply get deleted afterwards.
+
+### Blocker that outlives this change: qBittorrent has no durable WebUI credential
+
+`qBittorrent.conf` contains no `WebUI\Username` and no `WebUI\Password_PBKDF2`, so v5.1.2 falls back
+to user `admin` plus a **temporary password regenerated on every start**, announced only on stdout
+(`The WebUI administrator password was not set. A temporary password is provided for this session:`).
+That is what the passing connection tests above authenticate with — read out of the pod's log, piped
+through stdin, never written to disk or printed.
+
+**So the download clients will break the next time the qbittorrent pod restarts.** Nothing about the
+mappings is affected; only the stored password goes stale, and both arrs will report the client
+unavailable until it is updated. Fixing it properly means either a real password (SOPS-encrypted,
+materialised into `qBittorrent.conf` — the app takes no password env var) or
+`WebUI\AuthSubnetWhitelistEnabled=true` + `AuthSubnetWhitelist=10.42.0.0/16`, which the Cilium
+default-deny already narrows to exactly sonarr and radarr. Both are writes to qBittorrent's config
+PVC and both change the operator's own login story, so **neither was done**. Flagged for the operator.
+
+### Layout consistency checked while in there (reported, not restructured)
+
+- **Sonarr root folder** `/media/tv`, `accessible: true`, 7.6 TB free — same mount as
+  `/media/downloads`, so hardlinked imports stay possible. But **0 series in the database** against
+  ~100 unmapped folders on disk; the library exists on the export and Sonarr does not know about it.
+- **Radarr has NO root folder at all** (`GET /api/v3/rootfolder` -> `[]`) and 0 movies, while
+  `/media/movies` exists on the export. Radarr cannot import anything until a root folder is added.
+  Not added here — where the movie library lives is the operator's call.
+- **qBittorrent categories** were empty; `tv-sonarr` and `radarr` now exist (the arrs create their own
+  category on test/grab), both with `savePath: ""`, so both inherit the global `save_path=/downloads`
+  and the single `/downloads/ -> /media/downloads/` mapping covers both. `auto_tmm_enabled=false` and
+  `use_category_paths_in_manual_mode=false`, so categories will not start inventing subdirectories. If
+  a per-category save path is ever set, the mapping set must be revisited.
+- **`temp_path=/downloads/incomplete` with `temp_path_enabled=false`** — incomplete data lands in
+  `/downloads` alongside finished data. `incomplete/` does exist on the export.
+- **Pre-existing content in the downloads area, untouched:** a `complete/` and `incomplete/` tree, a
+  **dangling symlink `downloads -> /shared/downloads`** (a legacy non-k3s path that resolves nowhere
+  in these pods), and three ~1 GB files named `...S18E05....zipx` / `...S06E01....exe` /
+  `...S29E01....exe`. Media files with `.exe`/`.zipx` extensions are worth an operator's eye; nothing
+  here read or moved them.

catalog/qbittorrent.yaml

@@ -100,16 +100,32 @@ storage:
   # — k3s or legacy — gains or loses any access. Recorded because it is a write to state shared with
   # non-cluster consumers, which should never be a surprise.
   #
-  # PATH AGREEMENT WITH THE ARRS IS *NOT* CLOSED BY THIS CHANGE — see the report/DECISIONS-LOG. This
-  # pod calls the finished file /downloads/<name>; sonarr/radarr call the same bytes
+  # PATH AGREEMENT WITH THE ARRS IS STILL *NOT* CLOSED — see DECISIONS-LOG 2026-09-12 "Remote Path
+  # Mappings". This pod calls the finished file /downloads/<name>; sonarr/radarr call the same bytes
   # /media/downloads/<name>. Same inode, different absolute path, so Completed Download Handling needs
-  # ONE Remote Path Mapping per arr (remote host `qbittorrent`, remote path /downloads/, local path
-  # /media/downloads/). That lives in the arrs' own databases (app state on their config PVCs), not in
-  # git, so it is an operator step and is called out rather than silently assumed. The alternative —
+  # ONE Remote Path Mapping per arr (remote host `qbittorrent.qbittorrent.svc.cluster.local`, remote
+  # path /downloads/, local path /media/downloads/).
+  #
+  # THOSE MAPPINGS ARE NOW CONFIGURED AND VERIFIED ON PROD (2026-09-12), together with the qBittorrent
+  # download client itself — which did not exist in EITHER arr until then (both
+  # `GET /api/v3/downloadclient` returned `[]`, so completed-download handling had never been wired up
+  # at all). `downloadclient/testall` reports isValid=true in both, and Sonarr's queue was observed
+  # reporting outputPath=/downloads/... with the mapping deleted and /media/downloads/... with it
+  # present — same torrent, same client.
+  #
+  # BUT THEY LIVE IN THE ARRS' OWN DATABASES (app state on their config PVCs), NOT IN GIT: a rebuild
+  # from git alone does NOT restore them and no gate here notices their absence. The alternative —
   # repointing qBittorrent's SavePath at /media/downloads and mounting this subPath there instead —
   # buys path agreement at the cost of editing THIS app's config PVC, which is also app state; it was
   # not chosen because it changes a running download client's save path, which the deferral above
-  # already shows is easy to get wrong.
+  # already shows is easy to get wrong. It is nonetheless the RECOMMENDED end state (it makes the
+  # mappings unnecessary and puts the mountPath half of the fix in git) and is awaiting an operator
+  # decision, not a technical blocker.
+  #
+  # RELATED OPEN BLOCKER: this app has no durable WebUI credential (no WebUI\Username /
+  # WebUI\Password_PBKDF2 in qBittorrent.conf), so v5 issues a NEW temporary `admin` password on every
+  # start. The arrs' saved download-client password therefore goes stale on every pod restart. See the
+  # same DECISIONS-LOG entry for the two fixes, neither of which was taken unilaterally.
   - name: downloads
     class: bulk
     export: /mnt/Storage/plex   # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER} — the SAME export

ci/Makefile

@@ -12,7 +12,7 @@ CI_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
 
 .DEFAULT_GOAL := help
 
-.PHONY: help spec-immutability lint schema l0 render-golden l1 prom-rules conformance gate all touched versions tools test
+.PHONY: help spec-immutability lint schema l0 render-golden l1 conformance gate all touched versions tools test
 
 help: ## Show this help
 	@awk 'BEGIN{FS=":.*##"} /^[a-zA-Z0-9_-]+:.*##/{printf "  \033[36m%-14s\033[0m %s\n",$$1,$$2}' $(MAKEFILE_LIST)
@@ -32,9 +32,6 @@ render-golden: ## render==golden: canonical render reproduces committed apps/ (n
 l1: ## L1 rendering checks (no cluster; live now)
 	@"$(CI_DIR)l1.sh"
 
-prom-rules: ## promtool rule-unit tests for platform/monitoring alerts (no cluster; live now)
-	@"$(CI_DIR)prom-rules.sh"
-
 conformance: ## L2/L3 in-cluster Chainsaw Job (SKIPS w/o CLUSTER_AVAILABLE=true)
 	@"$(CI_DIR)conformance.sh"
 

ci/README.md

@@ -22,7 +22,6 @@ Individual stages:
 make -C ci lint        # yaml/lint                          (no cluster; live now)
 make -C ci l0          # schema + Kyverno + L0 static conformance (no cluster; live now)
 make -C ci l1          # L1 rendering checks                 (no cluster; live now)
-make -C ci prom-rules  # promtool rule-unit tests on alerts  (no cluster; live now)
 make -C ci conformance # L2/L3 in-cluster Chainsaw Job       (SKIPS unless CLUSTER_AVAILABLE=true)
 make -C ci test        # unit tests for these scripts themselves (ci/tests/*.test.sh)
 ```
@@ -103,14 +102,13 @@ in that case — nothing goes untested — and says plainly that the *scoped* ch
 
 ```
 ci/
-  Makefile                 targets: lint · l0 · l1 · prom-rules · conformance · gate · touched · test · tools · versions
+  Makefile                 targets: lint · l0 · l1 · conformance · gate · touched · test · tools · versions
   versions.env             PINNED tool versions (R-TEST-02) — single source of truth
   lib.sh                   shared helpers: logging, tool checks, cluster detection, artifact paths
   yamllint.yaml            lint config for the yaml/lint stage
   lint.sh                  Stage: yaml/lint
   l0.sh                    Stage: schema validation + Kyverno L0 gate + L0 static conformance
   l1.sh                    Stage: L1 rendering checks (Chainsaw L1 via the conformance runner)
-  prom-rules.sh            Stage: `promtool test rules` over platform/monitoring's PrometheusRules
                            (harness: platform/monitoring/tests/). Catches the two alerting bugs no
                            other stage can see — permanently-firing and structurally INERT rules.
   conformance.sh           Stage: L2/L3 in-cluster Chainsaw Job (trigger + collect)

ci/gate.sh

@@ -42,7 +42,6 @@ run_stage lint          "${CI_DIR}/lint.sh"
 run_stage l0            "${CI_DIR}/l0.sh"
 run_stage render-golden "${CI_DIR}/render-golden.sh"
 run_stage l1            "${CI_DIR}/l1.sh"
-run_stage prom-rules    "${CI_DIR}/prom-rules.sh"
 run_stage conformance   "${CI_DIR}/conformance.sh"
 
 echo >&2
@@ -87,7 +86,7 @@ esac
 # Phase 0 that is tolerated (placeholders), but surfaced. render-golden is a
 # no-cluster reproducibility gate (canonical render == committed apps/ goldens):
 # a FAIL means a golden drifted from its ServiceDefinition source — always blocks.
-for s in lint l0 render-golden l1 prom-rules; do
+for s in lint l0 render-golden l1; do
   case "${OUTCOME[$s]}" in
     fail) err  "${s}: FAILED -> blocks merge"; BLOCK=1 ;;
     skip) warn "${s}: skipped (unwired seam / missing tool) — not green" ;;

ci/prom-rules.sh

@@ -1,44 +0,0 @@
-#!/usr/bin/env bash
-# ci/prom-rules.sh — Stage: Prometheus RULE-UNIT tests (no cluster; live now).
-#
-# Thin wrapper over platform/monitoring/tests/run.sh, which runs `promtool test rules` against the
-# COMMITTED PrometheusRule manifests. It is a gate stage because the two alerting defects this repo
-# has actually shipped — a permanently-firing rule and a structurally INERT one — are both invisible
-# to every other stage: yamllint parses them, Kyverno admits them, and a healthy cluster cannot tell
-# a correct alert from one that can never fire. Only the bad state, constructed on purpose, can.
-#
-# Exit codes per ci/lib.sh: 0 pass / 1 fail / 3 skipped (never green).
-set -euo pipefail
-CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-# shellcheck disable=SC1091
-source "${CI_DIR}/lib.sh"
-
-log "Stage: Prometheus rule-unit tests (promtool test rules; no cluster)"
-
-RUNNER="${REPO_ROOT}/platform/monitoring/tests/run.sh"
-if [ ! -x "$RUNNER" ]; then
-  skip "rule-unit harness not present at ${RUNNER} — UNWIRED, not green"
-  summary "- prom-rules: **SKIPPED** (harness missing)"
-  exit 3
-fi
-
-set +e
-"$RUNNER"
-rc=$?
-set -e
-
-case "$rc" in
-  0)
-    ok "Prometheus rule-unit tests passed"
-    summary "- prom-rules: **passed**"
-    ;;
-  3)
-    skip "promtool unavailable (no promtool on PATH, no podman/docker) — NOT green"
-    summary "- prom-rules: **SKIPPED** (promtool toolchain unavailable)"
-    ;;
-  *)
-    err "Prometheus rule-unit tests FAILED"
-    summary "- prom-rules: **FAILED**"
-    ;;
-esac
-exit "$rc"

ci/versions.env

@@ -21,6 +21,15 @@ YAMLLINT_VERSION="1.35.1"
 KUBECTL_VERSION="1.36.4"
 YQ_VERSION="4.44.6"
 
+# Alert-rule unit tests (promtool test rules) — the L0 check
+# conformance/tests/l0-static/alert-rules-unit-tested runs every
+# platform/**/*.promtool-test.yaml through it. MUST stay in lockstep with
+# infra/devcontainer/versions.env (which bakes it into the image) AND with the Prometheus
+# version running on prod, because promtool IS the rule engine: verified 2026-09-12 as
+# v3.14.0-distroless via `kubectl get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'`.
+# Shipped inside the prometheus release tarball, not as a separate release.
+PROMTOOL_VERSION="3.14.0"
+
 # ServiceDefinition JSON-Schema validation (schemas/validate.sh). Pip-installed;
 # pinned here so CI and a laptop validate against the exact same validator.
 CHECK_JSONSCHEMA_VERSION="0.29.4"

clusters/base/kustomization.yaml

@@ -50,6 +50,13 @@ resources:
   # reverse-proxy sidecar (Option B). POST-platform tier (dependsOn platform-config, wait:false); same
   # non-gating leaf-UI rationale as headlamp/homepage/oauth2-proxy (a UI must never gate the platform).
   - crafty.yaml          # Flux Kustomization -> ./platform/crafty (dependsOn platform-config)
+  # Alert runbooks — POST-platform tier (dependsOn platform-config, wait:false). Serves the 34
+  # runbooks the alert rules' `runbook_url` annotations point at, at <apex>/runbooks/<name>; before
+  # this tier existed that path was a 404 and every alert handed the operator a dead link. Same
+  # non-gating leaf rationale as headlamp/homepage/crafty. Carries NO postBuild substitution on
+  # purpose — it renders documentation containing `${...}` shell examples that envsubst would empty;
+  # see clusters/base/runbooks.yaml.
+  - runbooks.yaml        # Flux Kustomization -> ./runbooks (dependsOn platform-config)
   # Velero backup controller — POST-platform tier (dependsOn platform-config, wait:false). A backup
   # controller must NEVER gate the platform chain; same non-gating rationale as headlamp/oauth2-proxy.
   # Carries velero-cloud.sops.yaml, so clusters/dev adds a SOPS decryption patch for it (by name).

clusters/base/runbooks.yaml

@@ -0,0 +1,57 @@
+---
+# Flux Kustomization: alert runbooks — reconciles ./runbooks.
+#
+# WHAT THIS SERVES. Every alert rule in this repo carries
+# `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations over 34 distinct names.
+# Until this tier existed that path returned 404 on the live cluster, so every alert that reached a
+# human handed them a dead link at the moment they needed help. This Kustomization renders the
+# markdown under ./runbooks into a ConfigMap and a two-replica nginx that serves it at that exact
+# prefix. See runbooks/README.md and runbooks/serving/httproute.yaml.
+#
+# POST-PLATFORM TIER (dependsOn platform-config), NON-GATING (wait:false) — the same placement and
+# the same reasoning as homepage/headlamp/oauth2-proxy. It needs the wildcard TLS cert, the shared
+# Gateway (including the `https-apex` listener) and split-horizon DNS, all of which arrive with
+# `platform` + `platform-config`; and a leaf web server must never be able to stall the platform
+# chain, which is the whole lesson of DECISIONS-LOG's "Headlamp gating stall". wait:false guarantees
+# its health can never gate anything even if something is later mistakenly pointed dependsOn it.
+#
+# NO postBuild.substituteFrom — THE ONE DELIBERATE DIVERGENCE FROM EVERY OTHER TIER, AND THE MOST
+# IMPORTANT LINE IN THIS FILE.
+# This tier renders the runbook PROSE into a ConfigMap, and runbooks are full of shell: `$POD`,
+# `$(kubectl ...)`, and — unavoidably, since they document these very annotations — the literal text
+# `${BASE_DOMAIN}`. Flux's postBuild substitution runs envsubst over the ENTIRE built stream, and it
+# FAILS OPEN: an undefined `${X}` becomes the empty string while the Kustomization still reports
+# Ready (DECISIONS-LOG 2026-09-01 — that is how `k8sServiceHost` was emptied and Cilium broke on 2
+# of 3 dev nodes). Enabled here it would silently mutilate the documentation: every braced expansion
+# in every command example blanked, in a file nobody re-reads after writing. Worse, the L0 check
+# `conformance/tests/l0-static/flux-postbuild-substitution` scans only files reachable through
+# kustomization `resources:`, not `configMapGenerator` inputs, so the damage would be invisible to
+# the gate as well.
+# The fix is to have nothing to substitute. The only knob this tier would have wanted is the apex
+# hostname, and runbooks/serving/httproute.yaml avoids needing it by declaring NO `hostnames:` and
+# inheriting the `https-apex` listener's own hostname instead. So this tier is
+# environment-independent by construction and R-ENV-02 ("authored once, promoted dev -> prod") holds
+# without substitution rather than because of it.
+# IF A FUTURE KNOB EVER GENUINELY NEEDS SUBSTITUTION HERE: do not add substituteFrom to this
+# Kustomization. Split the serving manifests into their own substituted tier and leave the content
+# ConfigMap in an unsubstituted one, or the runbooks lose their shell examples.
+#
+# NO SOPS decryption stanza: this tier carries no encrypted Secret (markdown plus five plain
+# manifests), so — like homepage — neither clusters/dev nor clusters/prod patches it.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: runbooks
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: false
+  dependsOn:
+    - name: platform-config   # wildcard TLS + Gateway (incl. the https-apex listener) first
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./runbooks

conformance/tests/l0-static/alert-rules-unit-tested/chainsaw-test.yaml

@@ -0,0 +1,95 @@
+# =============================================================================
+# L0 STATIC — the alert rules are UNIT-TESTED, by a pinned engine, and the gaps are declared
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# R-OBS-04 ("the platform alerts on its own health") is implemented by PrometheusRule CRs, and
+# this repo has shipped both ways they fail SILENTLY:
+#   * PERMANENTLY FIRING — the CNPG primaries alert (fixed 2026-09-12), VPNTunnelDown before it;
+#   * STRUCTURALLY INERT — VeleroNoRecentBackup, whose `time() - <absent series>` yields no
+#     series, so a schedule that had NEVER backed up could not raise it. Prod ran with zero
+#     volume backups and nothing alerted, masked by a resources-only schedule completing
+#     cleanly every night (2026-09-12).
+# An inert rule and a correct rule are indistinguishable while the platform is healthy. Only
+# `promtool test rules` separates them, by constructing the bad state on purpose, offline.
+#
+# The unit tests were then written — and RUN BY NOTHING. promtool was not in the pinned
+# toolchain, no gate stage evaluated the files, and the two harnesses that grew up independently
+# each fetched an engine off the network mid-run if promtool was missing (a GitHub release
+# download in one, a prometheus container image in the other). Coverage that looks present and
+# is enforced nowhere is the failure mode this test ends.
+#
+# GIVEN the repository as authored,
+# WHEN  the ONE rule-unit harness (platform/monitoring/tests/run.sh, which flattens the
+#       COMMITTED PrometheusRule CRs into the rules file promtool wants — never a copy) is run
+#       over every test file in the repo with the PINNED promtool,
+# THEN  promtool is pinned in BOTH versions.env files in lockstep at the Prometheus version prod
+#       actually runs, baked into the image and installed on the CI runner; the promtool in use
+#       reports exactly that version; no harness acquires an engine at run time; there is
+#       exactly ONE harness; every rule-unit test file PASSES; the harness is proven able to
+#       FAIL by a mutated synthetic test on every run; and every alert rule with NO unit test is
+#       ENUMERATED in a shrink-only register, so the gap is reported rather than hidden.
+#
+# MISSING TESTS: RATCHET, NOT SKIP, NOT BLANKET-FAIL. 48 of 55 alert rules have no unit test
+# today. Failing on all of them would make the gate red until ~48 tests exist (so it would be
+# disabled, and a disabled gate protects nothing); merely reporting is precisely how the CNPG
+# tests came to be enforced by nothing. So the gap is DECLARED in check.sh, printed with counts
+# every run, and CANNOT GROW: a new PrometheusRule, or a new alert in an already-tested file,
+# FAILS without a test, and a stale register entry FAILS too. A missing test is never an exit-3
+# skip — "SKIPPED is not green" is preserved for the one thing that is genuinely an environment
+# gap: promtool being absent.
+#
+# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (standard library + the harness
+# it delegates to); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
+# Chainsaw Test wraps the SAME script.
+#
+# NOT proven (named in check.sh's header): that an alert reaches a human (Alertmanager routing
+# and SMTP are L3 / R-OBS-02), that a threshold is the right one, that a rule file is actually
+# reconciled into a cluster, or that an author's assertions are meaningful — coverage of one
+# alert is detected by its name appearing in a covering test file, and only the mutation
+# self-test proves the harness CAN fail. R-OBS-04 is tagged for the static slice only: the rules
+# the repo ships behave, under the engine prod runs, as their own unit tests say.
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: alert-rules-unit-tested
+  labels:
+    conformance.home.k3s/R-OBS-04: "true"
+    conformance.home.k3s/R-TEST-02: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-OBS-04, R-TEST-02"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE — this invariant quantifies over every PrometheusRule in the tree
+    # (apps/wave-0..2, platform/cnpg, platform/kubevirt, platform/monitoring), and the register
+    # it maintains is a property of the WHOLE repo, not of any one path.
+    #
+    # WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh's coarse path map keys
+    # apps/* to NOTHING and platform/monitoring/* to R-OBS-01/02 — not R-OBS-04. So adding a new
+    # untested PrometheusRule under apps/, or editing one under platform/kubevirt/, attributes
+    # no requirement this test carries, and a requirement-scoped selection would filter the
+    # check out of exactly the change that adds the untested alert. The same blind spot that let
+    # `prod-baremetal-not-dev-surface` be filtered out of the run that gated its own breakage
+    # (DECISIONS-LOG 2026-09-12).
+    #
+    # Declaring the scope also makes an honest SKIP loud: lib/run_l1_offline.py turns a rc-3
+    # skip of a scope-declaring check into the runner's rc 3, and ci/l0.sh into the gate's rc 3.
+    # So a gate run on a toolchain WITHOUT promtool reports UNWIRED instead of green — which is
+    # the whole reason this check exists rather than another hand-written ci/l0.sh stage.
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      Every committed alert rule that has unit tests passes them under the pinned promtool that
+      matches prod's Prometheus, exactly one harness runs them, nothing fetches a rule engine at
+      run time, the harness is proven able to fail, and every alert rule without a unit test is
+      enumerated in a register that can only shrink.
+spec:
+  steps:
+    - name: run-every-alert-rule-unit-test-and-report-the-gaps
+      description: >-
+        Assert the promtool pin is in lockstep and baked, run the single rule-unit harness over
+        every test file in the repo, prove it can fail via a mutated synthetic test, and
+        enumerate every PrometheusRule with no unit tests against the shrink-only register.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/alert-rules-unit-tested/check.sh

@@ -0,0 +1,550 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 static check (offline, no cluster) — alert-rules-unit-tested
+# Serves: R-OBS-04 ("Platform alerts on its own health (nodes, storage, GitOps, certs,
+#                    VPN)." — spec docs/10-observability-and-alerting.md)
+#         R-TEST-02 ("Tests are reproducible: deterministic, hermetic, single-command."
+#                    — spec docs/14-testing-and-conformance.md)
+#
+# WHY THIS EXISTS — AN UNTESTED ALERT IS NOT A MONITORING CONTROL
+# --------------------------------------------------------------
+# This repo has shipped BOTH halves of the alerting failure space to prod:
+#   * a PERMANENTLY-FIRING `critical` (the CNPG alert fixed 2026-09-12; VPNTunnelDown before
+#     it), and
+#   * a STRUCTURALLY INERT rule — VeleroNoRecentBackup, whose `time() - <absent series>`
+#     yields no series, so a schedule that had NEVER backed up could not raise it. Prod ran
+#     with no volume backups at all and nothing alerted (2026-09-12).
+# Neither is visible from the expression, from `kubectl apply`, or from a healthy cluster: an
+# inert rule and a correct rule are indistinguishable while everything is fine. `promtool test
+# rules` is the only tool that separates them, because the BAD state can be constructed on
+# purpose. Unit tests for those rules were then WRITTEN — and evaluated by nothing: promtool
+# was not in the pinned toolchain, no gate stage ran the files, and each of the two harnesses
+# that existed acquired an engine off the network mid-run if one was missing. Coverage that
+# looks present and is never enforced is the failure mode this check ends.
+#
+# GIVEN the repository as authored,
+# WHEN  the ONE rule-unit harness (platform/monitoring/tests/run.sh, which flattens the
+#       COMMITTED PrometheusRule CRs — never a copy) is asked which manifests are under test
+#       and then run over every test file in the repo,
+# THEN  (1) promtool is a PINNED member of the toolchain: PROMTOOL_VERSION is declared in BOTH
+#           infra/devcontainer/versions.env and ci/versions.env and the two AGREE (the repo's
+#           lockstep rule), the Containerfile takes it as a build ARG, installs it from the
+#           pinned prometheus release and records it for provenance, dev.sh passes it, and the
+#           CI runner installs it for the l0 stage;
+#       (2) the promtool ACTUALLY IN USE reports exactly that pinned version — a unit test is
+#           evidence only if the engine matches the one prod evaluates rules with;
+#       (3) NO harness acquires a rule engine at RUN TIME — neither `curl | tar -xz` of a
+#           GitHub release nor a `podman/docker run` of a prometheus image. Both existed on
+#           2026-09-12 and both are removed; this is the regression guard;
+#       (4) there is exactly ONE harness. Two were written independently within an hour of
+#           each other; the one nothing runs is the one that rots;
+#       (5) every rule-unit test file in the repo PASSES, run through that harness (which
+#           `promtool check rules` first, so a non-parsing rules file cannot make unit tests
+#           pass vacuously by finding no rules at all);
+#       (6) the harness is NOT VACUOUS: a synthetic PrometheusRule + companion test is run
+#           through the SAME harness twice — intact (must PASS) and with one expectation
+#           mutated (must FAIL) — so a harness that has silently stopped evaluating anything
+#           is caught on every run, not only when someone thinks to try it;
+#   AND (7) the alert rules with NO unit tests are ENUMERATED and RATCHETED: each is declared
+#           below, printed with counts on every run, and the register can only SHRINK.
+#
+# WHY A MISSING TEST IS A RATCHET AND NOT AN OUTRIGHT FAIL (the decision, and its reasons)
+# ---------------------------------------------------------------------------------------
+# 48 of the repo's 55 alert rules have no unit test today. Three options were on the table:
+#   * FAIL on any missing test — correct in principle, but it makes the merge gate red for
+#     every change until ~48 unit tests exist, so in practice the gate gets disabled, and a
+#     disabled gate protects nothing.
+#   * merely REPORT — the gap prints and nothing holds it. That is exactly how the CNPG unit
+#     tests came to exist and be enforced by nothing; "it is in the output somewhere" is not
+#     enforcement.
+#   * RATCHET (chosen) — the gap is DECLARED IN THE REPO as a reviewable register, printed with
+#     counts on every gate run, and it CANNOT GROW: a new PrometheusRule, or a new alert in a
+#     file that already has tests, FAILS unless it ships a unit test. Removing an entry requires
+#     writing the test, and a STALE entry (file gone, or now covered) also FAILS, so the
+#     register cannot rot into fiction.
+# "SKIPPED is not green" is honoured: a missing test is NEVER an exit-3 skip here. This check
+# exits 0 only with the gap fully enumerated, and exits 1 the moment the debt grows, the
+# register drifts, or a test that exists stops passing.
+# The one exit-3 path is promtool itself being ABSENT — an environment gap, not a code property
+# (the same convention as ci/l0.sh's kyverno handling), and this test DECLARES
+# `conformance.home.k3s/scope: whole-repo`, which makes lib/run_l1_offline.py turn that skip
+# into the runner's rc 3 and ci/l0.sh into the gate's rc 3: UNWIRED, never green. promtool
+# PRESENT AT THE WRONG VERSION is a FAILURE, not a skip: that case still prints SUCCESS while
+# evaluating the rules with an engine prod does not run.
+#
+# WHAT THIS DOES *NOT* PROVE (honest gaps):
+#   - It does not prove an alert reaches a human. promtool evaluates rules against synthetic
+#     series in-process; Alertmanager routing, SMTP delivery and inhibition are L3
+#     (tests/l3-fault-injection/alerting) and R-OBS-02, and are not claimed here.
+#   - It does not prove the THRESHOLDS are right, only that a rule behaves as its own unit
+#     test says.
+#   - It does not prove a rule file is RECONCILED into a cluster. An alert file no
+#     kustomization includes is a separate "looks applied and isn't" failure, not audited here.
+#   - Coverage of an individual alert is detected by its NAME appearing in a test file that
+#     covers its manifest. A test file that merely mentions a name without asserting anything
+#     counts as covered; only the mutation self-test (6) proves the harness CAN fail, not that
+#     every author wrote a meaningful assertion. Review still owns test quality.
+#   - It does not read the live cluster, so "the pin matches prod's Prometheus" is asserted
+#     against the repo's pin, which was established by a live read recorded in
+#     infra/devcontainer/versions.env (2026-09-12: v3.14.0-distroless).
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+HARNESS="${ALERT_RULE_HARNESS:-${ROOT}/platform/monitoring/tests/run.sh}"
+
+# Honest SKIP (never green) if the thing under test is not present at all. Because this test
+# declares scope: whole-repo, the runner turns this into rc 3 for the whole L0 stage — a
+# PrometheusRule-less repo is a vanished tree, not a pass.
+if ! grep -rqls --include='*.yaml' -e '^[[:space:]]*kind:[[:space:]]*PrometheusRule[[:space:]]*$' \
+       "${ROOT}/platform" "${ROOT}/apps" 2>/dev/null; then
+  echo "check SKIPPED: no PrometheusRule under platform/ or apps/ — nothing to test" >&2
+  exit 3
+fi
+
+# promtool resolution. PROMTOOL= overrides (the same override the harness honours); otherwise
+# PATH. Absent -> every OFFLINE assertion still runs and the check exits 3, so a toolchain gap
+# is reported as a gap and never as a pass.
+PROMTOOL_BIN="${PROMTOOL:-$(command -v promtool || true)}"
+if [ -n "${PROMTOOL_BIN}" ] && [ -x "${PROMTOOL_BIN}" ]; then
+  PROMTOOL_BIN="$(cd "$(dirname "${PROMTOOL_BIN}")" && pwd)/$(basename "${PROMTOOL_BIN}")"
+else
+  PROMTOOL_BIN=""
+fi
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" "${HARNESS}" "${PROMTOOL_BIN}" <<'PY'
+"""Assert the alert-rule unit tests are pinned, enforced, non-vacuous, and that the alert
+rules with no tests are enumerated and ratcheted. Standard library + the harness itself."""
+from __future__ import annotations
+
+import os
+import re
+import shutil
+import subprocess
+import sys
+import tempfile
+from collections import defaultdict
+from pathlib import Path
+
+ROOT = Path(sys.argv[1])
+HARNESS = Path(sys.argv[2])
+PROMTOOL = sys.argv[3]
+sys.dont_write_bytecode = True          # keep __pycache__ out of the tracked tree
+
+FAILURES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+# =============================================================================
+# THE DECLARED REGISTER (assertion 7). Shrink-only; see the header for why this is a ratchet.
+#
+# UNTESTED_FILES — PrometheusRule files no test file covers AT ALL. The twelve apps/* entries
+# are single `<app>-down` liveness rules generated from one ServiceDefinition template;
+# platform/kubevirt/alerts.yaml (4 rules) is the substantial one and is where the next tests
+# belong. A file listed here that later gains coverage, or stops carrying a PrometheusRule,
+# FAILS as stale.
+# =============================================================================
+UNTESTED_FILES = {
+    "apps/wave-0/gitea.yaml",
+    "apps/wave-0/jackett.yaml",
+    "apps/wave-0/pihole.yaml",
+    "apps/wave-0/plex.yaml",
+    "apps/wave-0/qbittorrent.yaml",
+    "apps/wave-0/vm-demo.yaml",
+    "apps/wave-1/keycloak.yaml",
+    "apps/wave-1/nextcloud.yaml",
+    "apps/wave-1/radarr.yaml",
+    "apps/wave-1/semaphore.yaml",
+    "apps/wave-1/sonarr.yaml",
+    "apps/wave-2/seerr.yaml",
+    "platform/kubevirt/alerts.yaml",
+}
+
+# UNTESTED_ALERTS — individual alerts inside a file that IS covered by a test file, but whose
+# own rule nothing asserts on. This is the half that stops a new alert being slipped into an
+# already-"tested" file untested: anything not listed here must appear in a covering test.
+# The monitoring block is large because backups.test.yaml (the repo's first suite) deliberately
+# covers the Velero/backup rules only; pvc-capacity-alerts.yaml is flattened into the same
+# rules file by the suite convention but has no assertions of its own yet.
+UNTESTED_ALERTS = {
+    ("platform/cnpg/alerts.yaml", "CNPGWALArchiveBacklog"),
+    ("platform/cnpg/alerts.yaml", "CNPGNoRecoverabilityPoint"),
+    ("platform/cnpg/alerts.yaml", "CNPGNoRecentBaseBackup"),
+    ("platform/cnpg/alerts.yaml", "CNPGWALVolumeNotRecycling"),
+    ("platform/cnpg/alerts.yaml", "CNPGMetricsCollectorErroring"),
+    ("platform/monitoring/alerts.yaml", "NodeExporterDown"),
+    ("platform/monitoring/alerts.yaml", "NodeMemoryPressureHigh"),
+    ("platform/monitoring/alerts.yaml", "NodeDiskFillingUp"),
+    ("platform/monitoring/alerts.yaml", "NodeCPUPressureHigh"),
+    ("platform/monitoring/alerts.yaml", "NodeCPULoadHigh"),
+    ("platform/monitoring/alerts.yaml", "NodeIOPressureHigh"),
+    ("platform/monitoring/alerts.yaml", "NodeIOPressureCritical"),
+    ("platform/monitoring/alerts.yaml", "LonghornVolumeDegraded"),
+    ("platform/monitoring/alerts.yaml", "LonghornVolumeFaulted"),
+    ("platform/monitoring/alerts.yaml", "LonghornVolumeCapacityHigh"),
+    ("platform/monitoring/alerts.yaml", "LonghornNodeStorageHigh"),
+    ("platform/monitoring/alerts.yaml", "FluxReconciliationFailure"),
+    ("platform/monitoring/alerts.yaml", "FluxResourceSuspended"),
+    ("platform/monitoring/alerts.yaml", "CertificateExpiringSoon"),
+    ("platform/monitoring/alerts.yaml", "CertificateExpiryImminent"),
+    ("platform/monitoring/alerts.yaml", "CertificateNotReady"),
+    ("platform/monitoring/alerts.yaml", "VPNTunnelDown"),
+    ("platform/monitoring/alerts.yaml", "HubbleDeniedFlowSpike"),
+    ("platform/monitoring/alerts.yaml", "KeycloakDown"),
+    ("platform/monitoring/alerts.yaml", "KeycloakDatabaseDown"),
+    ("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerFailing"),
+    ("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerStale"),
+    ("platform/monitoring/alerts.yaml", "MonitoringVolumeFillingUp"),
+    ("platform/monitoring/alerts.yaml", "PrometheusTargetsMissing"),
+    ("platform/monitoring/alerts.yaml", "ImageUpdateAutomationFailing"),
+    ("platform/monitoring/alerts.yaml", "ImagePolicyStalled"),
+    ("platform/monitoring/alerts.yaml", "UpdateHealthGateFailed"),
+    ("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeFillingUp"),
+    ("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeAlmostFull"),
+    ("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeFillingUpPredicted"),
+}
+
+# --------------------------------------------------------------- (1) the pins, in lockstep
+def env_pin(rel: str, key: str) -> str | None:
+    f = ROOT / rel
+    if not f.is_file():
+        return None
+    m = re.search(rf'^{re.escape(key)}=["\']?([^"\'\s#]+)', f.read_text(), re.M)
+    return m.group(1) if m else None
+
+
+DEV_PIN = env_pin("infra/devcontainer/versions.env", "PROMTOOL_VERSION")
+CI_PIN = env_pin("ci/versions.env", "PROMTOOL_VERSION")
+check(DEV_PIN is not None,
+      "infra/devcontainer/versions.env declares no PROMTOOL_VERSION — promtool is not part of the "
+      "baked toolchain, so the alert unit tests are evaluated by whatever the host happens to "
+      "have, or by nothing at all (which is where this repo was on 2026-09-12)")
+check(CI_PIN is not None,
+      "ci/versions.env declares no PROMTOOL_VERSION — the gate's own version file does not pin the "
+      "rule engine, so CI and the devcontainer can silently diverge")
+check(DEV_PIN is None or DEV_PIN == CI_PIN,
+      f"PROMTOOL_VERSION is NOT in lockstep: infra/devcontainer/versions.env={DEV_PIN!r} vs "
+      f"ci/versions.env={CI_PIN!r}. The repo's rule is that the conformance-tool versions in the "
+      f"two files stay identical, so a laptop and CI run the same engine")
+
+CF = ROOT / "infra/devcontainer/Containerfile"
+cf_text = CF.read_text() if CF.is_file() else ""
+check(re.search(r"^ARG PROMTOOL_VERSION\s*$", cf_text, re.M),
+      "infra/devcontainer/Containerfile has no `ARG PROMTOOL_VERSION` — the pin is declared but "
+      "the image build does not consume it")
+check("prometheus/prometheus/releases/download" in cf_text and "promtool" in cf_text,
+      "infra/devcontainer/Containerfile does not install promtool from the pinned prometheus "
+      "release tarball (promtool ships inside it)")
+check('echo "promtool=${PROMTOOL_VERSION}"' in cf_text,
+      "infra/devcontainer/Containerfile does not record promtool in /etc/dev-toolchain-versions — "
+      "that provenance line is how an operator tells a stale image from a current one")
+
+DEVSH = ROOT / "infra/devcontainer/dev.sh"
+check(DEVSH.is_file() and "--build-arg PROMTOOL_VERSION=" in DEVSH.read_text(),
+      "infra/devcontainer/dev.sh does not pass --build-arg PROMTOOL_VERSION — the ARG would be "
+      "empty and the build would fetch a bogus URL")
+
+INSTALL = ROOT / ".gitea/workflows/scripts/install-tools.sh"
+if INSTALL.is_file():
+    itext = INSTALL.read_text()
+    check("PROMTOOL_VERSION" in itext
+          and re.search(r"^\s*l0\)[^\n]*install_promtool", itext, re.M),
+          ".gitea/workflows/scripts/install-tools.sh does not install promtool for the l0 stage — "
+          "on the CI runner this check would exit 3 (UNWIRED) forever and the alert unit tests "
+          "would be enforced nowhere, which is the exact gap it exists to close")
+
+# --------------------------------------------- (3)(4) one harness, and it fetches no engine
+check(HARNESS.is_file() and os.access(HARNESS, os.X_OK),
+      f"the rule-unit harness is missing or not executable at {HARNESS.relative_to(ROOT)} — "
+      f"nothing would run the alert unit tests")
+
+# Any script that invokes `promtool test rules` is a rule-unit harness. There must be exactly
+# one: two harnesses were written independently on 2026-09-12 (a cnpg-only script and this one),
+# and the one nothing runs is the one that rots out of sync with the rules.
+harnesses: list[str] = []
+fetchers: list[str] = []
+FETCH_RE = re.compile(
+    r"(curl|wget)[^\n]*prometheus/prometheus/releases"          # GitHub release download
+    r"|(podman|docker)\s+run[^\n]*prometheus",                  # container-image pull
+    re.I)
+# Two exclusions, both the repo's existing "reference, not declaration" rule (see
+# ci/touched-requirements.sh changed_diff() rules 2-3):
+#   * THIS CHECK'S OWN DIRECTORY. It names every forbidden pattern in order to forbid it, so
+#     scanning itself would make both guards permanently red — the self-referential trap.
+#   * THE PINNED TOOLCHAIN INSTALLERS. Downloading the pinned promtool is their JOB, at a
+#     reviewed seam (an image build, or the runner's tool-install step), from a version in
+#     versions.env. What is forbidden is a RULE-UNIT RUN acquiring its own engine: then the
+#     version, the network and the machine are decided silently at test time, and a missing
+#     toolchain becomes a green run instead of an exit 3.
+SELF_DIR = "conformance/tests/l0-static/alert-rules-unit-tested/"
+INSTALLERS = (".gitea/workflows/scripts/install-tools.sh", "infra/devcontainer/")
+for p in sorted(ROOT.rglob("*.sh")):
+    rel = str(p.relative_to(ROOT))
+    if rel.startswith((".git/", "conformance/runs/", SELF_DIR)):
+        continue
+    try:
+        text = p.read_text()
+    except (OSError, UnicodeDecodeError):
+        continue
+    body = "\n".join(l for l in text.splitlines() if not l.lstrip().startswith("#"))
+    if "promtool" in body and re.search(r"test\s+rules", body):
+        harnesses.append(rel)
+    if FETCH_RE.search(body) and not rel.startswith(INSTALLERS):
+        fetchers.append(rel)
+
+check(harnesses == [str(HARNESS.relative_to(ROOT))],
+      f"expected exactly ONE rule-unit harness ({HARNESS.relative_to(ROOT)}) but found "
+      f"{harnesses}. A second harness means two behaviours and two things to keep in step; the "
+      f"one the gate does not run is the one that rots. Fold it in instead")
+check(not fetchers,
+      f"{fetchers} acquire a rule ENGINE at run time (a prometheus release download or a "
+      f"prometheus container image). promtool must come from the pinned toolchain "
+      f"(PROMTOOL_VERSION): a run-time fetch decides the version, the network and the machine "
+      f"silently at test time, makes the tests non-hermetic (R-TEST-02), and quietly turns 'the "
+      f"toolchain has no promtool' into a green run — which is how it stayed missing")
+
+# ------------------------------------------------------------------ discover the alert files
+# Alert files: tracked YAML carrying a PrometheusRule. conformance/ is excluded deliberately —
+# the PrometheusRules under tests/l1-rendering/*/golden/ are EXPECTED-RENDER fixtures for the
+# render checks, not rules any cluster evaluates. Everything a cluster actually gets lives under
+# apps/, platform/, platform-config/ or clusters/.
+def tracked_yaml() -> list[Path]:
+    out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z", "*.yaml", "*.yml"],
+                         capture_output=True, text=True)
+    if out.returncode == 0 and out.stdout:
+        return [ROOT / p for p in out.stdout.split("\0") if p]
+    # No usable index (the gate container mounts a worktree whose .git points outside the
+    # mount). Fall back to a filesystem walk rather than auditing nothing.
+    return sorted(p for p in ROOT.rglob("*.y*ml") if ".git/" not in str(p))
+
+
+KIND_RE = re.compile(r"^\s*kind:\s*PrometheusRule\s*$", re.M)
+ALERT_RE = re.compile(r"^\s*-\s*alert:\s*(\S+)", re.M)
+
+alert_files: dict[str, list[str]] = {}
+for p in tracked_yaml():
+    rel = str(p.relative_to(ROOT))
+    if rel.startswith("conformance/") or not p.is_file():
+        continue
+    try:
+        text = p.read_text()
+    except (OSError, UnicodeDecodeError):
+        continue
+    if KIND_RE.search(text):
+        alert_files[rel] = ALERT_RE.findall(text)
+
+check(len(alert_files) > 0,
+      "no PrometheusRule found outside conformance/ — discovery or the repo layout changed, and "
+      "this check would silently audit nothing")
+
+# ------------------- ask the HARNESS which manifests are under test (never re-derive it here)
+# Two implementations of "which rules does this test cover" would eventually disagree, and the
+# disagreement shows up as a file the gate believes is tested and the harness never touches.
+listing = subprocess.run(["bash", str(HARNESS), "--list"], capture_output=True, text=True,
+                         cwd=str(ROOT))
+check(listing.returncode == 0,
+      f"`{HARNESS.relative_to(ROOT)} --list` failed (rc={listing.returncode}): "
+      f"{(listing.stderr or listing.stdout).strip()[-600:]}")
+covering: dict[str, list[str]] = defaultdict(list)   # manifest -> [test files]
+for line in listing.stdout.splitlines():
+    if "\t" in line:
+        tfile, manifest = line.split("\t", 1)
+        covering[manifest.strip()].append(tfile.strip())
+
+for manifest in sorted(covering):
+    check(manifest in alert_files,
+          f"the harness reports {manifest} as under test, but it is not one of the "
+          f"PrometheusRule files found in the repo — the mapping and the tree disagree")
+
+# --------------------------------------------------------- (2) the engine is the pinned one
+version_line = ""
+if PROMTOOL:
+    vp = subprocess.run([PROMTOOL, "--version"], capture_output=True, text=True)
+    out = (vp.stdout + vp.stderr).strip()
+    version_line = out.splitlines()[0] if out else ""
+    m = re.search(r"version\s+(\d+\.\d+\.\d+)", version_line)
+    found = m.group(1) if m else None
+    check(found is not None,
+          f"could not parse a version from `promtool --version`: {version_line!r}")
+    check(found is None or DEV_PIN is None or found == DEV_PIN,
+          f"promtool reports {found}, but the toolchain pins PROMTOOL_VERSION={DEV_PIN} (the "
+          f"Prometheus version prod runs). promtool IS the rule engine: a unit test green under "
+          f"one version and red under another is worse than no test. Run the gate in the pinned "
+          f"devcontainer (rebuild it: ./infra/devcontainer/dev.sh build), or set PROMTOOL= to a "
+          f"v{DEV_PIN} binary. A FAILURE, not a skip, because the wrong engine still prints "
+          f"SUCCESS")
+
+# ------------------------------------------------- (5) every rule-unit test file must pass
+if PROMTOOL:
+    env = dict(os.environ, PROMTOOL=PROMTOOL)
+    run = subprocess.run(["bash", str(HARNESS)], capture_output=True, text=True,
+                         cwd=str(ROOT), env=env)
+    tailed = "\n        ".join((run.stdout + run.stderr).strip().splitlines()[-40:])
+    check(run.returncode == 0,
+          f"`{HARNESS.relative_to(ROOT)}` FAILED (rc={run.returncode}) — the committed alert "
+          f"rules do not behave as their own unit tests say:\n        {tailed}")
+
+    # ------------------------------------------- (6) the harness is proven non-vacuous
+    # A synthetic PrometheusRule + companion test through the SAME harness, outside the repo so
+    # discovery never picks it up. Intact must PASS; a mutated expectation must FAIL. Without
+    # this, a harness that stopped evaluating anything (an empty extraction, a rule file written
+    # under the wrong name) would report green forever — which is the disease being treated.
+    work = tempfile.mkdtemp(prefix="alert-rules-selftest.")
+    try:
+        (Path(work) / "synth.yaml").write_text("""apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: selftest
+spec:
+  groups:
+    - name: selftest
+      rules:
+        - alert: SelfTestCanary
+          expr: selftest_metric > 0
+          for: 5m
+          annotations:
+            runbook_url: https://${BASE_DOMAIN}/runbooks/selftest
+""")
+        test_yaml = """rule_files:
+  - generated.rules.yaml
+evaluation_interval: 1m
+tests:
+  - interval: 1m
+    name: canary fires only after `for` elapses
+    input_series:
+      - series: 'selftest_metric'
+        values: '1x10'
+    promql_expr_test:
+      - expr: 'ALERTS{alertname="SelfTestCanary",alertstate="firing"}'
+        eval_time: EVAL_TIME
+        exp_samples:
+          - labels: 'ALERTS{alertname="SelfTestCanary",alertstate="firing"}'
+            value: 1
+"""
+        synth_test = Path(work) / "synth.promtool-test.yaml"
+        synth_test.write_text(test_yaml.replace("EVAL_TIME", "6m"))
+        good = subprocess.run(["bash", str(HARNESS), str(synth_test)],
+                              capture_output=True, text=True, cwd=str(ROOT), env=env)
+        check(good.returncode == 0,
+              "self-test: the INTACT synthetic unit test did not pass, so the harness itself is "
+              "broken and every green result above is meaningless:\n        "
+              + "\n        ".join((good.stdout + good.stderr).strip().splitlines()[-20:]))
+        # Mutation: assert firing at 2m, which `for: 5m` forbids.
+        synth_test.write_text(test_yaml.replace("EVAL_TIME", "2m"))
+        bad = subprocess.run(["bash", str(HARNESS), str(synth_test)],
+                             capture_output=True, text=True, cwd=str(ROOT), env=env)
+        check(bad.returncode != 0,
+              "self-test: a MUTATED expectation (firing before `for:` elapsed) still PASSED — "
+              "the harness is not actually evaluating anything, so every green result above is "
+              "vacuous")
+    finally:
+        shutil.rmtree(work, ignore_errors=True)
+
+# --------------------------------------------------------- (7) the gap inventory + ratchet
+INVENTORY: list[str] = []
+GAP_FILES: list[str] = []
+alerts_total = covered_total = 0
+
+for rel in sorted(alert_files):
+    names = alert_files[rel]
+    alerts_total += len(names)
+    check(len(names) > 0,
+          f"{rel} declares a PrometheusRule but no `alert:` rule was parsed out of it — the "
+          f"parser or the file shape changed, and any unit test over it would pass vacuously")
+    tests = covering.get(rel, [])
+    if not tests:
+        GAP_FILES.append(rel)
+        INVENTORY.append(f"  {rel:<46} alerts={len(names):<3} tested=0/{len(names):<3} NO TEST FILE")
+        check(rel in UNTESTED_FILES,
+              f"{rel} carries {len(names)} alert rule(s) and NO test file covers it. A new "
+              f"PrometheusRule must ship promtool unit tests: this repo has shipped both a "
+              f"permanently-firing critical and a structurally inert alert, and neither was "
+              f"visible without them. Add `{rel[:-5]}.promtool-test.yaml` (companion convention) "
+              f"or a <component>/tests/*.test.yaml suite. If this is deliberate debt it must be "
+              f"DECLARED in UNTESTED_FILES in this check — the register is what keeps the gap "
+              f"visible instead of silent")
+        continue
+
+    check(rel not in UNTESTED_FILES,
+          f"{rel} is listed in UNTESTED_FILES but is now covered by {tests} — the register is "
+          f"shrink-only: delete the line so the next missing test is caught")
+    text = "".join((ROOT / t).read_text() for t in tests if (ROOT / t).is_file())
+    covered = [n for n in names if re.search(rf"\b{re.escape(n)}\b", text)]
+    covered_total += len(covered)
+    for n in names:
+        if n in covered:
+            check((rel, n) not in UNTESTED_ALERTS,
+                  f"{rel}: alert {n} is listed in UNTESTED_ALERTS but IS asserted on now — "
+                  f"remove the stale entry (the register is shrink-only)")
+        else:
+            check((rel, n) in UNTESTED_ALERTS,
+                  f"{rel}: alert {n} has NO unit test in {tests}. This file is already covered, "
+                  f"so a new alert in it must be tested too — otherwise a 'tested' alert file "
+                  f"quietly grows untested rules")
+    INVENTORY.append(f"  {rel:<46} alerts={len(names):<3} tested={len(covered)}/{len(names):<3} "
+                     f"{','.join(tests)}")
+
+# stale register entries
+for rel in sorted(UNTESTED_FILES - set(alert_files)):
+    check(False, f"UNTESTED_FILES lists {rel}, which carries no PrometheusRule in this repo any "
+                 f"more — a stale exemption hides the next real gap; delete the line")
+for rel in sorted({f for f, _ in UNTESTED_ALERTS} - set(alert_files)):
+    check(False, f"UNTESTED_ALERTS lists alerts in {rel}, which carries no PrometheusRule in this "
+                 f"repo any more — delete those lines")
+
+# --------------------------------------------------------------------------------- report
+print("Alert-rule unit-test inventory (PrometheusRule -> covering test file(s)):")
+for line in INVENTORY:
+    print(line)
+print(f"\nharness: {HARNESS.relative_to(ROOT)}   promtool: {version_line or 'NOT FOUND'}")
+print(f"pinned:  PROMTOOL_VERSION={DEV_PIN} (devcontainer) / {CI_PIN} (ci) — prod runs Prometheus "
+      f"v{DEV_PIN}")
+print(f"alert files: {len(alert_files)}   covered by a test file: "
+      f"{len(alert_files) - len(GAP_FILES)}   with NO test file: {len(GAP_FILES)}")
+print(f"alert rules: {alerts_total}   unit-tested: {covered_total}   "
+      f"NOT unit-tested: {alerts_total - covered_total}")
+
+if GAP_FILES:
+    print("\nDECLARED GAPS — alert files nothing tests at all (shrink-only register; each is a")
+    print("rule nothing has ever evaluated, NOT a rule known to work):")
+    for g in GAP_FILES:
+        print(f"  - {g}  ({len(alert_files[g])} alert(s))")
+if UNTESTED_ALERTS:
+    print("\nDECLARED GAPS — individual alerts inside a covered file that nothing asserts on:")
+    for f, n in sorted(UNTESTED_ALERTS):
+        print(f"  - {f}  {n}")
+
+print()
+if not PROMTOOL and not FAILURES:
+    print("check SKIPPED (exit 3, NOT a pass): promtool is not on PATH, so NO alert rule was "
+          "parsed and NO unit test was evaluated.", file=sys.stderr)
+    print(f"  promtool is pinned at v{DEV_PIN} in infra/devcontainer/versions.env + "
+          f"ci/versions.env.", file=sys.stderr)
+    print("  Run the gate in the pinned toolbox (./infra/devcontainer/dev.sh build, then "
+          "make -C ci gate), or set PROMTOOL=/path/to/promtool.", file=sys.stderr)
+    print(f"  The {CHECKS} offline assertions above (pins, lockstep, one harness, no run-time "
+          f"engine fetch, the gap register) DID run and passed.", file=sys.stderr)
+    print("  This test declares scope: whole-repo, so the runner reports rc 3 — UNWIRED, never "
+          "green.", file=sys.stderr)
+    raise SystemExit(3)
+
+if not FAILURES:
+    print(f"check OK: alert-rules-unit-tested — {CHECKS} assertions "
+          f"({covered_total}/{alerts_total} alert rules unit-tested, "
+          f"{len(GAP_FILES)} files with no test file, all declared) (R-OBS-04, R-TEST-02)")
+    raise SystemExit(0)
+print(f"CHECK FAIL (alert-rules-unit-tested): {len(FAILURES)} failed, "
+      f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
+for f in FAILURES:
+    print(f"  FAIL {f}", file=sys.stderr)
+raise SystemExit(1)
+PY

conformance/tests/l0-static/alert-runbooks-resolvable/chainsaw-test.yaml

@@ -0,0 +1,89 @@
+# =============================================================================
+# L0 STATIC — every alert's runbook_url resolves to a runbook that is actually served
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# Every platform-health alert rule in this repo carries
+# `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations across four files, over 34
+# distinct names. For the whole of phases 4-6 there was NO runbooks/ directory and NO HTTPRoute serving
+# that path: an in-cluster probe of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404.
+# So an alert reaching a human handed them a dead link at the one moment documentation is read, while the
+# annotation made the problem look solved — the same green-surface-over-absent-mechanism class as the
+# import Job that reported success while storing placeholder secrets, and the guard whose glob silently
+# stopped matching.
+#
+# The link is served now (runbooks/, clusters/base/runbooks.yaml). This test is the guard that keeps it
+# true, and it asserts BOTH halves of the claim because the regression is silent in both directions: an
+# alert with a new runbook_url and no runbook links to a 404, AND a runbook committed without its
+# configMapGenerator entry is in git, reviewed, merged — and still 404s in the cluster. "The file exists"
+# and "the URL resolves" are different claims; only the second one helps an operator.
+#
+# GIVEN every `runbook_url` annotation on every PrometheusRule in the repo,
+# WHEN  each is parsed out of git and resolved against runbooks/,
+# THEN  (1) each URL has EXACTLY the shape the server resolves — https://${BASE_DOMAIN}/runbooks/<name>,
+#           the ${BASE_DOMAIN} token (never a literal hostname), no `.md`, no trailing slash, no query;
+#       (2) every referenced <name> has a runbooks/<name>.md in git;
+#       (3) each runbook clears a QUALITY FLOOR — the four operator sections, a real body, and the
+#           dev-container kubectl invocation with --context k3s-prod — so a stub cannot satisfy (2) and
+#           make the link look fixed while still helping nobody;
+#       (4) every runbook is PROJECTED INTO THE SERVER by runbooks/kustomization.yaml's
+#           configMapGenerator under the extensionless key the URL asks for;
+#       (5) route, listener and server agree with the annotations (PathPrefix == the annotations' prefix,
+#           a sectionName that platform/gateway/gateway.yaml actually defines, the namespace opted in to
+#           the Gateway, and an nginx location for that same prefix with a text/plain default_type);
+#       (6) the Flux tier is wired into clusters/base AND declares NO postBuild.substituteFrom —
+#           substitution would run envsubst over the runbook prose and silently empty every `${...}`
+#           shell example in it (it fails open: DECISIONS-LOG 2026-09-01, the emptied Cilium
+#           k8sServiceHost), and the flux-postbuild-substitution check cannot see it because that check
+#           scans files reachable through `resources:`, not configMapGenerator inputs;
+#   AND (7) the check is LOAD-BEARING: this Test DECLARES
+#           `conformance.home.k3s/scope: whole-repo`, so ci/l0.sh Stage 3b's
+#           `--include-scope whole-repo` unions it into EVERY gate invocation rather than only the runs
+#           whose touched requirements happen to include R-OBS-04; and ci/gate.sh sets the merge block
+#           when the l0 stage fails. A SKIP of a scope-declaring check is UNWIRED, never green.
+#
+# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library only);
+# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test wraps the SAME
+# script.
+#
+# NOT proven (named in check.sh's header): this does not FETCH the URL. That it returns HTTP 200 with
+# real content was proven by an in-cluster probe recorded in the commit message; a live probe needs a
+# cluster, which L0 by definition does not have. What is asserted here is every static precondition for
+# that 200. It also does not judge whether a runbook is correct or current — (3) is a floor, not a
+# review — and it does not prove Alertmanager delivers the annotation to a human (that is the L3
+# `alerting` test). R-OBS-04 is tagged for the narrow slice only: the platform's own health alerts carry
+# runbook links that resolve to real, served procedure rather than to a 404.
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: alert-runbooks-resolvable
+  labels:
+    conformance.home.k3s/R-OBS-04: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-OBS-04"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE — this is what makes the check unconditional. Stage 3b of ci/l0.sh runs
+    # `conformance/run.sh --layer l0 --include-scope whole-repo`, which UNIONS every test carrying this
+    # annotation into the selection regardless of which requirements the change touched. That union is
+    # load-bearing here: ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02" —
+    # NOT R-OBS-04 — and maps platform/kubevirt/* and runbooks/* to nothing at all, so the single most
+    # likely breaking change (adding an alert with a runbook_url, or adding a runbook without its
+    # configMapGenerator entry) attributes a requirement set that would otherwise FILTER THIS CHECK OUT
+    # of exactly the change that introduces the bug. Declaring the scope replaces the hand-written
+    # per-test ci/l0.sh stage that this repo used to need (and that nobody remembered to add).
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      Every runbook_url on every alert rule resolves to a runbook that exists in git, clears a quality
+      floor, and is actually projected into the server that answers that path — so an alert can never
+      hand an operator a dead link.
+spec:
+  steps:
+    - name: audit-every-alert-runbook-link
+      description: >-
+        Parse every runbook_url out of every PrometheusRule and assert the runbook exists, is real, is
+        served at that exact path, and that the serving tier is wired and unsubstituted.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/alert-runbooks-resolvable/check.sh

@@ -0,0 +1,410 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 static check (offline, no cluster) — alert-runbooks-resolvable
+# Serves: R-OBS-04 ("The platform alerts on its own health (nodes, storage, GitOps, certs,
+#         VPN)." — spec docs/10-observability-and-alerting.md)
+#
+# WHY THIS EXISTS — A DEAD LINK IN AN ALERT IS WORSE THAN NO LINK
+# --------------------------------------------------------------
+# Every platform-health alert rule in this repo carries
+#   runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"
+# 42 such annotations across four files, over 34 distinct names. For the whole of phases 4-6
+# there was NO runbooks/ directory and NO HTTPRoute serving that path: a live in-cluster probe
+# of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404. So an alert reaching a
+# human at 3am handed them a link that fails, at the one moment the documentation is read. The
+# annotation LOOKED like the problem was solved, which is the specific failure this repo keeps
+# finding (an import Job that reports success while storing placeholder secrets; a guard whose
+# glob silently stopped matching) — a green-looking surface over an absent mechanism.
+#
+# The link is now served (runbooks/, clusters/base/runbooks.yaml). This check is the guard that
+# keeps it true, because the regression is SILENT IN BOTH DIRECTIONS and nothing else catches it:
+#   * add an alert with a new runbook_url and forget the runbook  -> that alert links to a 404,
+#     and no test, lint, render or policy notices;
+#   * write runbooks/<name>.md but forget to add it to the ConfigMap generator -> the file is in
+#     git, reviewable, apparently done, and the URL still 404s in the cluster. "The file exists"
+#     and "the URL resolves" are different claims and only the second one helps an operator.
+# Both halves are asserted below, which is why this check reaches past the markdown into the
+# kustomization, the HTTPRoute, the nginx config and the Flux tier.
+#
+# GIVEN the repository as authored,
+# WHEN  every `runbook_url` annotation on every PrometheusRule is parsed out of git,
+# THEN  (1) each URL has EXACTLY the shape the server resolves —
+#           https://${BASE_DOMAIN}/runbooks/<name>, `${BASE_DOMAIN}` as the token (never a literal
+#           hostname: a literal is how prod was authored to serve dev's hostnames, see the
+#           ServiceDefinition schema's note), no `.md` suffix, no trailing slash, no query;
+#       (2) every referenced <name> has a runbooks/<name>.md in git;
+#       (3) each runbook clears a QUALITY FLOOR — the four operator sections, a real body, and the
+#           dev-container kubectl invocation — so a one-line stub cannot satisfy (2) and make the
+#           link look fixed while still helping nobody (the failure mode this check is most likely
+#           to be defeated by is a placeholder file, not a missing one);
+#       (4) every runbook markdown file is actually PROJECTED INTO THE SERVER by
+#           runbooks/kustomization.yaml's configMapGenerator, under the EXTENSIONLESS key the URL
+#           asks for — the "exists in git but 404s in the cluster" half;
+#       (5) the route and the server agree with the annotations: the HTTPRoute's PathPrefix is the
+#           prefix the URLs use, it attaches to a Gateway listener that actually exists, its
+#           namespace is opted in to that Gateway, and the nginx config answers that same prefix;
+#       (6) the Flux tier that reconciles all of it is wired into clusters/base AND declares NO
+#           postBuild.substituteFrom — substitution here would run envsubst over the runbook prose
+#           and silently empty every `${...}` shell example in it (it fails open: DECISIONS-LOG
+#           2026-09-01, the emptied Cilium k8sServiceHost);
+#   AND (7) this check is LOAD-BEARING: ci/l0.sh runs it on every gate run and ci/gate.sh sets the
+#           merge block when the l0 stage fails.
+#
+# WHY (7) IS AN UNCONDITIONAL ci/l0.sh STAGE and not left to the requirement-scoped run: Stage 3b
+# scopes `conformance/run.sh --layer l0` to the TOUCHED requirements (R-TEST-03), and
+# ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02" — NOT R-OBS-04 — and
+# maps platform/kubevirt/* and runbooks/* to nothing at all. So the single most likely breaking
+# change (adding an alert with a runbook_url to platform/monitoring/alerts.yaml, or adding a
+# runbook and forgetting the generator entry) attributes a requirement set that would FILTER THIS
+# CHECK OUT. Declaring `conformance.home.k3s/scope: whole-repo` makes ci/l0.sh Stage 3b union it into
+# EVERY gate invocation instead; a SKIP of a scope-declaring check is UNWIRED, never green.
+#
+# WHAT THIS DOES *NOT* PROVE (honest gaps — this is a STATIC check):
+#   - It does not fetch the URL. That it returns HTTP 200 with real content was proven by an
+#     in-cluster probe (recorded in the commit message and DECISIONS-LOG); a live probe needs a
+#     cluster, which L0 by definition does not have. What is asserted here is every static
+#     precondition for that 200: the file, its projection, the key name, the route, the prefix.
+#   - It does not judge whether a runbook is CORRECT or current. (3) is a floor, not a review: it
+#     can tell a stub from a runbook, not a good runbook from a stale one. Nothing automatable can.
+#   - It does not prove Alertmanager actually delivers the annotation to a human. That is the L3
+#     `alerting` test.
+#   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06) and is not consulted.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+# Honest SKIP (never green) if there are no alert rules at all — nothing to audit yet.
+if ! grep -rl 'kind: PrometheusRule' "${ROOT}/platform" >/dev/null 2>&1; then
+  echo "check SKIPPED: no PrometheusRule found under platform/ — no alert rules to audit" >&2
+  exit 3
+fi
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+"""Static audit: every alert runbook_url resolves to a runbook that is actually served.
+
+Pure standard library, no PyYAML — the same constraint and the same style as the neighbouring
+L0 checks (lib/coverage.py, lib/status.py). Every parse contract is documented at its parser;
+all of them are line-constrained shapes in hand-authored manifests.
+"""
+from __future__ import annotations
+
+import re
+import sys
+from pathlib import Path
+
+ROOT = Path(sys.argv[1])
+sys.dont_write_bytecode = True          # keep __pycache__ out of the tracked tree
+
+FAILURES: list[str] = []
+ADVISORIES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+def read(p: Path) -> str:
+    try:
+        return p.read_text()
+    except (OSError, UnicodeDecodeError):
+        return ""
+
+
+# =============================================================== (1) parse the annotations
+# Parse contract: `runbook_url:` is an annotation on a Prometheus alerting rule, authored one per
+# line as `runbook_url: "<url>"` (single or double quoted, or bare). Only files that declare
+# `kind: PrometheusRule` are scanned, so a runbook_url quoted inside this check or inside prose
+# elsewhere in the repo is not mistaken for an alert annotation. Commented-out lines are skipped:
+# platform/monitoring/alerts.yaml carries a `# runbook_url points at a docs base` header comment.
+SCOPE_DECL_RE = r"^\s*conformance\.home\.k3s/scope:\s*[\"\']?whole-repo"
+RUNBOOK_URL_RE = re.compile(r"""^\s*runbook_url:\s*["']?([^"'\s#]+)["']?\s*$""")
+
+rule_files = sorted(
+    p for p in ROOT.rglob("*.yaml")
+    if ".git" not in p.parts and "kind: PrometheusRule" in read(p)
+)
+check(len(rule_files) > 0,
+      "no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "
+      "this check would silently audit nothing")
+
+# url -> [ "<relpath>:<lineno>", ... ]
+refs: dict[str, list[str]] = {}
+for f in rule_files:
+    rel = f.relative_to(ROOT)
+    for lineno, line in enumerate(read(f).splitlines(), 1):
+        if line.lstrip().startswith("#"):
+            continue
+        m = RUNBOOK_URL_RE.match(line)
+        if m:
+            refs.setdefault(m.group(1), []).append(f"{rel}:{lineno}")
+
+n_annotations = sum(len(v) for v in refs.values())
+check(n_annotations > 0,
+      "no runbook_url annotation parsed out of any PrometheusRule — the parser contract broke; "
+      "this check would pass vacuously while every alert link rots")
+
+# ---- URL SHAPE. The server resolves exactly one shape. Anything else is a link that 404s even
+# though a file with a plausible name exists, so shape is a failure, not a style note.
+URL_RE = re.compile(r"^https://\$\{BASE_DOMAIN\}/(runbooks)/([a-z0-9][a-z0-9-]*)$")
+names: dict[str, list[str]] = {}        # runbook name -> annotation sites
+prefixes: set[str] = set()
+for url, sites in sorted(refs.items()):
+    m = URL_RE.match(url)
+    check(m is not None,
+          f"runbook_url '{url}' ({', '.join(sites)}) is not the shape this platform serves. "
+          f"Required: https://${{BASE_DOMAIN}}/runbooks/<name> — the ${{BASE_DOMAIN}} TOKEN (never a "
+          f"literal hostname: a literal is how prod got authored to serve dev's public names), a "
+          f"lowercase-hyphen <name>, and NO '.md' suffix, trailing slash or query. The runbook "
+          f"server maps the URL path straight onto an extensionless ConfigMap key, so any other "
+          f"shape is a 404 no matter which files exist")
+    if m:
+        prefixes.add(m.group(1))
+        names.setdefault(m.group(2), []).extend(sites)
+
+check(len(prefixes) <= 1,
+      f"alert annotations use more than one runbook path prefix ({sorted(prefixes)}); the single "
+      f"HTTPRoute + nginx location can only serve one, so the others are dead links")
+PREFIX = (sorted(prefixes) or ["runbooks"])[0]
+
+# =================================================== (2)(3) the runbooks exist and are real
+RUNBOOK_DIR = ROOT / PREFIX
+check(RUNBOOK_DIR.is_dir(),
+      f"{PREFIX}/ is not a directory in this repo, so every one of the {n_annotations} "
+      f"runbook_url annotations is a dead link")
+
+# The quality floor. (2) alone is satisfiable by `touch runbooks/node-down.md`, which makes the
+# link resolve and still abandons the operator — a worse outcome than the 404, because the 404 at
+# least told the truth. These four sections are the operator's path through an incident: what the
+# alert means HERE, what to type first, what is probably wrong, and when to stop. `dev.sh` +
+# `--context k3s-prod` are required because kubectl does not work on the workstation at all
+# (the toolchain lives only in the pinned dev container) and because dev.sh defaults its
+# current-context to k3s-DEV — a runbook that omits the context teaches the reader to answer
+# questions about the wrong cluster during an incident on this one.
+REQUIRED_SECTIONS = (
+    "## What this means here",
+    "## First commands",
+    "## Likely causes, most likely first",
+    "## When to escalate or stop",
+)
+MIN_BYTES = 1500
+
+for name, sites in sorted(names.items()):
+    md = RUNBOOK_DIR / f"{name}.md"
+    if not md.is_file():
+        check(False,
+              f"runbook '{name}' does not exist: {md.relative_to(ROOT)} is missing, but "
+              f"{len(sites)} alert annotation(s) link to it ({', '.join(sites[:3])}"
+              f"{', ...' if len(sites) > 3 else ''}). An alert firing now hands the operator a 404. "
+              f"Write {PREFIX}/{name}.md (see {PREFIX}/README.md for the required sections)")
+        continue
+    body = read(md)
+    check(len(body) >= MIN_BYTES,
+          f"{md.relative_to(ROOT)} is {len(body)} bytes, under the {MIN_BYTES}-byte floor — a stub "
+          f"that makes the link resolve while still helping nobody is worse than the 404 it "
+          f"replaced, because it looks fixed")
+    for sec in REQUIRED_SECTIONS:
+        check(sec in body,
+              f"{md.relative_to(ROOT)} has no '{sec}' section — the four operator sections are what "
+              f"make this a runbook rather than a restatement of the alert name")
+    check("dev.sh" in body,
+          f"{md.relative_to(ROOT)} never mentions dev.sh — kubectl does not exist on the "
+          f"workstation; every command in a runbook here must be runnable as "
+          f"`infra/devcontainer/dev.sh run bash -c '...'` and the runbook must say so")
+    check("--context k3s-prod" in body,
+          f"{md.relative_to(ROOT)} never passes --context k3s-prod — dev.sh merges both clusters "
+          f"and picks a current-context PER INVOCATION — a default that has already changed once "
+          f"(k3s-dev until dev was decommissioned on 2026-09-13, k3s-prod now). A context-less command "
+          f"silently answers about whichever cluster the default happens to be, in the middle of an "
+          f"incident")
+
+# ADVISORY, not a failure: a runbook nothing links to yet. Writing the runbook BEFORE the alert is
+# the right order, and the repo's precedent for "authored but unreferenced" is advisory (see the
+# dead-knob advisory in flux-postbuild-substitution/check.sh).
+served_extra = {"README"}
+if RUNBOOK_DIR.is_dir():
+    for md in sorted(RUNBOOK_DIR.glob("*.md")):
+        stem = md.stem
+        if stem in served_extra or stem in names:
+            continue
+        ADVISORIES.append(f"{md.relative_to(ROOT)} is a runbook no alert links to (written ahead of "
+                          f"its alert, or the alert was removed — advisory, not a failure)")
+
+# ================================= (4) every runbook is PROJECTED INTO THE SERVER, not just in git
+# THIS IS THE HALF A FILE-EXISTENCE CHECK WOULD MISS. The pod serves a ConfigMap, not the git tree.
+# Parse contract: runbooks/kustomization.yaml declares one configMapGenerator whose `files:` entries
+# are `- <key>=<file>.md`, one per line. The key is EXTENSIONLESS on purpose (the URL carries no
+# `.md`, and the ConfigMap volume mount turns each key into a file the nginx root resolves directly).
+KUST = RUNBOOK_DIR / "kustomization.yaml"
+check(KUST.is_file(), f"{PREFIX}/kustomization.yaml is missing — the Flux tier renders nothing and "
+                      f"every runbook 404s no matter what is in git")
+GEN_FILE_RE = re.compile(r"^\s*-\s*([A-Za-z0-9._-]+)=([A-Za-z0-9._/-]+\.md)\s*(?:#.*)?$")
+generated: dict[str, str] = {}
+kust_text = read(KUST)
+if kust_text:
+    check("configMapGenerator:" in kust_text,
+          f"{PREFIX}/kustomization.yaml has no configMapGenerator — the runbook markdown is not "
+          f"rendered into the ConfigMap the server mounts")
+    for line in kust_text.splitlines():
+        if line.lstrip().startswith("#"):
+            continue
+        m = GEN_FILE_RE.match(line)
+        if m:
+            generated[m.group(1)] = m.group(2)
+    check(len(generated) > 0,
+          f"{PREFIX}/kustomization.yaml declares no `<key>=<file>.md` generator entries — the "
+          f"parser contract broke, or the markdown is being added with implicit keys (which would "
+          f"carry the '.md' suffix the alert URLs do NOT have, so every link would 404)")
+
+for name in sorted(names):
+    check(generated.get(name) == f"{name}.md",
+          f"runbook '{name}' is not projected into the server: {PREFIX}/kustomization.yaml has no "
+          f"configMapGenerator entry `- {name}={name}.md`. The file can be in git, reviewed and "
+          f"merged and the URL still returns 404 — 'the file exists' and 'the URL resolves' are "
+          f"different claims")
+
+# And the reverse, so a generator entry cannot point at a file that is not there.
+for key, fname in sorted(generated.items()):
+    check((RUNBOOK_DIR / fname).is_file(),
+          f"{PREFIX}/kustomization.yaml generates key '{key}' from {PREFIX}/{fname}, which does not "
+          f"exist — `kustomize build` fails, so the WHOLE tier stops reconciling and every runbook "
+          f"goes stale at once")
+
+# ============================== (5) the route, the listener and the server agree with the URLs
+ROUTE = RUNBOOK_DIR / "serving" / "httproute.yaml"
+NGINX = RUNBOOK_DIR / "serving" / "nginx-config.yaml"
+NS = RUNBOOK_DIR / "serving" / "namespace.yaml"
+GATEWAY = ROOT / "platform" / "gateway" / "gateway.yaml"
+
+route_text = read(ROUTE)
+check(bool(route_text), f"{PREFIX}/serving/httproute.yaml is missing — nothing routes the "
+                        f"/{PREFIX} prefix to the server")
+# Parse contract: a single HTTPRoute with one PathPrefix match and one parentRefs entry naming a
+# listener via sectionName; hand-authored, one key per line.
+m_path = re.search(r"^\s*type:\s*PathPrefix\s*$\n\s*value:\s*(\S+)\s*$", route_text, re.M)
+check(m_path is not None and m_path.group(1) == f"/{PREFIX}",
+      f"{PREFIX}/serving/httproute.yaml does not match PathPrefix /{PREFIX} "
+      f"(found {m_path.group(1) if m_path else 'no PathPrefix match'}). The alert annotations point "
+      f"at /{PREFIX}/<name>; if the route's prefix drifts from the annotations the links 404 while "
+      f"both files look individually correct")
+m_sec = re.search(r"^\s*sectionName:\s*(\S+)", route_text, re.M)
+check(m_sec is not None,
+      f"{PREFIX}/serving/httproute.yaml declares no parentRefs sectionName. The apex host needs the "
+      f"dedicated apex listener: the wildcard `*.${{BASE_DOMAIN}}` listener does NOT match the bare "
+      f"apex, so a route without sectionName binds nothing and serves nothing")
+if m_sec:
+    listener = m_sec.group(1)
+    gw = read(GATEWAY)
+    check(re.search(rf"^\s*-\s*name:\s*{re.escape(listener)}\s*$", gw, re.M) is not None,
+          f"{PREFIX}/serving/httproute.yaml attaches to Gateway listener '{listener}', which "
+          f"platform/gateway/gateway.yaml does not define — the route would report NotAllowed and "
+          f"never serve")
+# An HTTPRoute with no `hostnames:` inherits the listener's hostname, which is how this tier avoids
+# needing Flux substitution (see (6)). Either form is routable; a LITERAL hostname is not acceptable,
+# for the same reason the schema forbids it in a ServiceDefinition.
+m_host = re.search(r"^\s*hostnames:\s*$\n\s*-\s*(\S+)", route_text, re.M)
+check(m_host is None or "${BASE_DOMAIN}" in m_host.group(1),
+      f"{PREFIX}/serving/httproute.yaml hardcodes hostname '{m_host.group(1) if m_host else ''}' — "
+      f"use the ${{BASE_DOMAIN}} token, or omit `hostnames:` entirely and inherit the listener's")
+
+ns_text = read(NS)
+check('gateway-access: "shared"' in ns_text or "gateway-access: shared" in ns_text,
+      f"{PREFIX}/serving/namespace.yaml does not carry gateway-access=shared — the shared Gateway "
+      f"selects namespaces by that label, so without it the HTTPRoute is refused attachment")
+
+nginx_text = read(NGINX)
+check(f"location /{PREFIX}/" in nginx_text,
+      f"{PREFIX}/serving/nginx-config.yaml has no `location /{PREFIX}/` block — the route would "
+      f"deliver the request and the server would answer 404 for every runbook")
+check("default_type" in nginx_text and "text/plain" in nginx_text,
+      f"{PREFIX}/serving/nginx-config.yaml does not force a text/plain default_type. The ConfigMap "
+      f"keys are EXTENSIONLESS, so with the stock MIME table nginx serves them as "
+      f"application/octet-stream and a browser DOWNLOADS the runbook instead of showing it")
+
+# ====================================== (6) the Flux tier is wired, and is NOT substituted
+FLUX = ROOT / "clusters" / "base" / f"{PREFIX}.yaml"
+BASE_KUST = ROOT / "clusters" / "base" / "kustomization.yaml"
+flux_text = read(FLUX)
+check(bool(flux_text),
+      f"clusters/base/{PREFIX}.yaml is missing — nothing reconciles {PREFIX}/ into the cluster, so "
+      f"the runbooks exist in git and the URL still 404s")
+check(re.search(rf"^\s*path:\s*\./{re.escape(PREFIX)}\s*$", flux_text, re.M) is not None,
+      f"clusters/base/{PREFIX}.yaml does not set `path: ./{PREFIX}`")
+check(re.search(r"^\s*wait:\s*false\s*$", flux_text, re.M) is not None,
+      f"clusters/base/{PREFIX}.yaml is not wait:false — a leaf web server that can be legitimately "
+      f"unhealthy must never gate the reconcile chain (DECISIONS-LOG: 'Headlamp gating stall')")
+# Strip comments before testing: this file's own comment block EXPLAINS at length why
+# substituteFrom must never be added here, and a naive substring test would fire on the warning.
+flux_code = "\n".join(l for l in flux_text.splitlines() if not l.lstrip().startswith("#"))
+check("substituteFrom" not in flux_code,
+      f"clusters/base/{PREFIX}.yaml declares postBuild.substituteFrom. It MUST NOT: this tier "
+      f"renders documentation into a ConfigMap, and Flux's envsubst would run over the prose and "
+      f"silently EMPTY every `${{...}}` shell example in every runbook — it fails open, so the "
+      f"Kustomization would still report Ready (DECISIONS-LOG 2026-09-01, the emptied Cilium "
+      f"k8sServiceHost). The flux-postbuild-substitution check cannot see this: it scans files "
+      f"reachable through `resources:`, not configMapGenerator inputs")
+check(re.search(rf"^\s*-\s*{re.escape(PREFIX)}\.yaml\b", read(BASE_KUST), re.M) is not None,
+      f"clusters/base/kustomization.yaml does not list {PREFIX}.yaml — the Flux Kustomization is "
+      f"authored and never applied, which is the 'looks done, is absent' class this repo has been "
+      f"bitten by repeatedly")
+
+# ============================================ (7) this check is load-bearing in the merge gate
+L0 = read(ROOT / "ci" / "l0.sh")
+GATE = read(ROOT / "ci" / "gate.sh")
+# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism, in two halves.
+# It is not enough that check.sh exists: the merge gate must SELECT it on every run. The repo used to
+# get that with a hand-written "UNCONDITIONAL" stage per whole-repo check in ci/l0.sh, hardcoding the
+# path; those stages are gone, because the next whole-repo invariant only got one if somebody
+# REMEMBERED to write it. The property is now DECLARED by the test and HONOURED by the gate, so both
+# halves are asserted — neither alone is sufficient. The mechanism itself (that the union really
+# selects, and that a scope-declared SKIP exits 3 rather than passing green) is proven end-to-end and
+# falsifiably by conformance/tests/l0-static/whole-repo-invariants-wired.
+SELF_YAML = (ROOT / "conformance" / "tests" / "l0-static" / "alert-runbooks-resolvable"
+             / "chainsaw-test.yaml")
+SELF = read(SELF_YAML)
+check(re.search(SCOPE_DECL_RE, SELF, re.M) is not None,
+      "alert-runbooks-resolvable/chainsaw-test.yaml does not declare "
+      "conformance.home.k3s/scope: whole-repo, so the merge gate selects this check only when a "
+      "change happens to touch a requirement it is labelled with. ci/touched-requirements.sh maps "
+      "platform/monitoring/* to 'R-OBS-01 R-OBS-02' (NOT R-OBS-04) and maps platform/kubevirt/* and "
+      "runbooks/* to nothing at all — so adding an alert with a new runbook_url, or adding a runbook "
+      "without its configMapGenerator entry, would be filtered out of exactly the change that "
+      "introduces the dead link")
+check("--include-scope" in L0,
+      "ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is a "
+      "purely requirement-scoped run again and the scope annotation above is inert decoration — this "
+      "check would then run only by coincidence")
+check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
+      "ci/gate.sh does not set the merge block when the l0 stage fails")
+
+# ======================================================================== report
+print(f"Alert runbook inventory: {n_annotations} runbook_url annotation(s) across "
+      f"{len(rule_files)} PrometheusRule file(s) -> {len(names)} distinct runbook(s), "
+      f"prefix /{PREFIX}")
+missing = sorted(n for n in names if not (RUNBOOK_DIR / f"{n}.md").is_file())
+print(f"  runbooks present: {len(names) - len(missing)}/{len(names)}   "
+      f"projected into the ConfigMap: {sum(1 for n in names if generated.get(n) == f'{n}.md')}/{len(names)}")
+if missing:
+    print(f"  MISSING: {', '.join(missing)}", file=sys.stderr)
+
+if ADVISORIES:
+    print("\nADVISORIES (not failures):")
+    for a in ADVISORIES:
+        print(f"  - {a}")
+
+print()
+if not FAILURES:
+    print(f"check OK: alert-runbooks-resolvable — {CHECKS} assertions (R-OBS-04)")
+    raise SystemExit(0)
+print(f"CHECK FAIL (alert-runbooks-resolvable): {len(FAILURES)} failed, "
+      f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
+for f in FAILURES:
+    print(f"  FAIL {f}", file=sys.stderr)
+raise SystemExit(1)
+PY

infra/devcontainer/Containerfile

@@ -1,7 +1,7 @@
 # Reproducible dev toolbox for the home-network-k3s platform (Phase 1+).
 # Bakes the NON-Python prereqs (OpenTofu, age, sops, flux) plus ansible + kube tooling
 # AND the pinned conformance-harness toolchain (kyverno CLI, kyverno-json, chainsaw,
-# yamllint, check-jsonschema, yq) into a single pinned baseline image, so `make -C ci gate`
+# yamllint, check-jsonschema, yq, promtool) into a single pinned baseline image, so `make -C ci gate`
 # runs identically here and in CI. This is the Proxmox (bpg/proxmox) toolbox + conformance
 # harness: OpenTofu drives the Proxmox-hosted k3s cluster via the bpg provider (dev.sh opens
 # an SSH tunnel to the kube-API), NOT a local libvirt/KVM host. The libvirt-client/qemu-img
@@ -23,6 +23,10 @@ ARG CHAINSAW_VERSION
 ARG YAMLLINT_VERSION
 ARG YQ_VERSION
 ARG CHECK_JSONSCHEMA_VERSION
+# promtool: the engine that evaluates the alert-rule UNIT TESTS (platform/**/*.promtool-test.yaml).
+# Pinned to the Prometheus version prod actually runs — see versions.env for why that lockstep is
+# the point, and for the live-cluster command that establishes it.
+ARG PROMTOOL_VERSION
 ARG TARGETARCH=amd64
 
 # Distro packages: libvirt client libs (harmless disk-image utilities), ansible, ssh,
@@ -59,7 +63,11 @@ RUN set -eux; \
     KYVERNO_CLI_ARCH="$([ "${TARGETARCH}" = amd64 ] && echo x86_64 || echo "${TARGETARCH}")"; \
     curl -fsSL "https://github.com/kyverno/kyverno/releases/download/v${KYVERNO_VERSION}/kyverno-cli_v${KYVERNO_VERSION}_linux_${KYVERNO_CLI_ARCH}.tar.gz" | tar -xz -C /usr/local/bin kyverno; \
     curl -fsSL "https://github.com/kyverno/kyverno-json/releases/download/v${KYVERNO_JSON_VERSION}/kyverno-json_linux_${TARGETARCH}.tar.gz" | tar -xz -C /usr/local/bin kyverno-json; \
-    chmod +x /usr/local/bin/yq /usr/local/bin/chainsaw /usr/local/bin/kyverno /usr/local/bin/kyverno-json; \
+    PROM_TARBALL="prometheus-${PROMTOOL_VERSION}.linux-${TARGETARCH}"; \
+    curl -fsSL "https://github.com/prometheus/prometheus/releases/download/v${PROMTOOL_VERSION}/${PROM_TARBALL}.tar.gz" \
+      | tar -xz -C /usr/local/bin --strip-components=1 "${PROM_TARBALL}/promtool"; \
+    chmod +x /usr/local/bin/yq /usr/local/bin/chainsaw /usr/local/bin/kyverno /usr/local/bin/kyverno-json \
+             /usr/local/bin/promtool; \
     rm -rf /tmp/*
 
 # Record the baked toolchain for provenance.
@@ -68,6 +76,7 @@ RUN { echo "opentofu=${OPENTOFU_VERSION}"; echo "age=${AGE_VERSION}"; echo "sops
       echo "kyverno=${KYVERNO_VERSION}"; echo "kyverno-json=${KYVERNO_JSON_VERSION}"; \
       echo "chainsaw=${CHAINSAW_VERSION}"; echo "yamllint=${YAMLLINT_VERSION}"; \
       echo "yq=${YQ_VERSION}"; echo "check-jsonschema=${CHECK_JSONSCHEMA_VERSION}"; \
+      echo "promtool=${PROMTOOL_VERSION}"; \
     } > /etc/dev-toolchain-versions
 
 WORKDIR /work

infra/devcontainer/dev.sh

@@ -86,6 +86,7 @@ build() {
     --build-arg YAMLLINT_VERSION="$YAMLLINT_VERSION" \
     --build-arg YQ_VERSION="$YQ_VERSION" \
     --build-arg CHECK_JSONSCHEMA_VERSION="$CHECK_JSONSCHEMA_VERSION" \
+    --build-arg PROMTOOL_VERSION="$PROMTOOL_VERSION" \
     -t "$IMAGE" -f "${HERE}/Containerfile" "${HERE}"
 }
 

infra/devcontainer/versions.env

@@ -2,7 +2,7 @@
 # The non-Python prereqs (OpenTofu, age, sops, flux) are Go binaries, so — per the
 # operator's rule — they are baked into a container image instead of a venv. The
 # conformance test toolchain (kyverno CLI, kyverno-json, chainsaw, yamllint,
-# check-jsonschema, yq) is likewise baked in so `make -C ci gate` runs here == in CI.
+# check-jsonschema, yq, promtool) is likewise baked in so `make -C ci gate` runs here == in CI.
 #
 # NOTE: these are best-effort pins; bump to current stable if a download 404s at build.
 # The conformance-tool versions below MUST stay in lockstep with ci/versions.env.
@@ -28,4 +28,20 @@ CHAINSAW_VERSION=0.2.12
 YAMLLINT_VERSION=1.35.1
 YQ_VERSION=4.44.6
 CHECK_JSONSCHEMA_VERSION=0.29.4
+# promtool — the Prometheus rule linter + UNIT-TEST runner (`promtool test rules`), which is
+# what evaluates platform/**/*.promtool-test.yaml in the L0 gate
+# (conformance/tests/l0-static/alert-rules-unit-tested).
+#
+# THIS PIN IS NOT FREE-FLOATING: it MUST equal the Prometheus version actually running on prod,
+# because promtool is the rule ENGINE and a different major/minor can parse or evaluate PromQL
+# differently — a unit test green under one engine and red under the other is worse than no test.
+# Verified against the live cluster on 2026-09-12:
+#   kubectl --context k3s-prod get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'
+#     -> v3.14.0-distroless   (image quay.io/prometheus/prometheus:v3.14.0-distroless)
+# When kube-prometheus-stack ships a new Prometheus to prod, bump this (and ci/versions.env) with
+# it; the L0 check asserts the promtool on PATH reports exactly this version, so a drifted
+# toolchain FAILS the gate instead of quietly testing with the wrong engine.
+# promtool ships INSIDE the prometheus release tarball (prometheus-<v>.linux-<arch>.tar.gz), so
+# the version string here is a Prometheus release, not a separate promtool release.
+PROMTOOL_VERSION=3.14.0
 # Proxmox provider (bpg/proxmox) is pinned in the tofu module (infra/tofu/versions.tf), not here.

platform/cnpg/alerts.promtool-test.yaml

@@ -9,11 +9,25 @@
 # not evidence that it can fire, and it is certainly not evidence that it can stop. These tests pin
 # BOTH directions plus the two ways the rule is meant to stay quiet.
 #
-# HOW TO RUN. `./run-alert-tests.sh` in this directory — it extracts `.spec` from the PrometheusRule
-# (promtool wants a bare Prometheus rules file, not a Kubernetes CR), substitutes ${BASE_DOMAIN},
-# and runs promtool against both files in a temp dir. promtool is NOT in the baked devcontainer
-# toolchain (infra/devcontainer/versions.env); the script downloads the version that matches the
-# Prometheus actually running on prod (v3.14.0) if it is not already on PATH.
+# HOW TO RUN. `platform/monitoring/tests/run.sh` — the ONE repo-wide rule-unit harness. It finds
+# this file by the companion convention (`<manifest>.promtool-test.yaml` sits beside
+# `<manifest>.yaml`), flattens the PrometheusRule CR's `.spec.groups` into the bare Prometheus
+# rules file promtool wants (a CR is not a rules file), writes it under the name `rule_files:`
+# below declares, `promtool check rules` it first so a non-parsing file cannot make these tests
+# pass vacuously, and runs them — all in a temp dir, because a generated file sitting next to its
+# source is how a stale copy becomes the thing under test. Run just this file with
+# `platform/monitoring/tests/run.sh platform/cnpg/alerts.promtool-test.yaml`.
+# (There used to be a cnpg-only `run-alert-tests.sh` here; it was folded into that harness on
+# 2026-09-12 — two harnesses means two behaviours, and the one nothing runs rots.)
+#
+# promtool IS in the baked devcontainer toolchain (PROMTOOL_VERSION in
+# infra/devcontainer/versions.env, mirrored in ci/versions.env), pinned to the Prometheus version
+# prod actually runs (v3.14.0) because promtool is the rule ENGINE. A missing promtool is exit 3
+# (SKIPPED, never green), not a silent download.
+#
+# WHO RUNS IT IN CI. conformance/tests/l0-static/alert-rules-unit-tested — the L0 static check
+# that runs the harness over every *.promtool-test.yaml in the repo and reports which alert files
+# still have none. Before that check existed these tests were never evaluated by anything.
 #
 # WHY THE ASSERTIONS USE `ALERTS` AND NOT `exp_alerts`. promtool's alert_rule_test compares
 # annotations VERBATIM, which would mean pasting this rule's ~1200-character runbook description in

platform/cnpg/alerts.yaml

@@ -349,7 +349,9 @@ spec:
         #     on k3s-prod-1, keycloak-db-1 on k3s-prod-2, nextcloud-db-1 on k3s-prod-3. Quiet because
         #     the condition is false, NOT because the query went blind.
         #   * UNIT-TESTED, AND THE TESTS CAN FAIL. alerts.promtool-test.yaml (run it with
-        #     ./run-alert-tests.sh) covers four cases under promtool 3.14.0, the exact Prometheus
+        #     platform/monitoring/tests/run.sh — the one repo-wide harness; the merge gate runs the
+        #     same path via conformance/tests/l0-static/alert-rules-unit-tested) covers four cases
+        #     under promtool 3.14.0, the exact Prometheus
         #     version running here: co-located fires after 30m, one-per-node is silent, a NotReady
         #     node suppresses it and then re-arms when the node returns, and a non-CNPG `*-rw`
         #     Service is ignored. Mutation-checked — deleting the cnpg_collector_up join, the

platform/cnpg/run-alert-tests.sh

@@ -1,60 +0,0 @@
-#!/usr/bin/env bash
-###############################################################################
-# platform/cnpg/run-alert-tests.sh — run the promtool unit tests for alerts.yaml.
-#
-# WHY A SCRIPT AND NOT JUST `promtool test rules`. Two impedance mismatches:
-#   1. alerts.yaml is a Kubernetes PrometheusRule CR; promtool wants a bare Prometheus
-#      rules file (`groups:` at the top level). So `.spec` is extracted first.
-#   2. runbook_url carries `${BASE_DOMAIN}`, a Flux postBuild substitution that is not
-#      resolved in git. promtool does not care what the string says, but leaving a raw
-#      `${...}` in a rendered rules file is exactly the kind of thing that gets copied
-#      somewhere it DOES matter, so it is substituted with a placeholder here.
-# Both derived files go to a temp dir — nothing generated is left in the worktree, because
-# a generated file sitting next to its source is how a stale copy becomes the thing under
-# test.
-#
-# promtool is NOT in the baked devcontainer toolchain (infra/devcontainer/versions.env).
-# If it is not on PATH this fetches the version that matches the Prometheus actually
-# running on prod, so the unit tests are evaluated by the same engine that will evaluate
-# the rules for real. Override with PROMTOOL=/path/to/promtool.
-#
-#   ./run-alert-tests.sh
-###############################################################################
-set -euo pipefail
-
-# Match the Prometheus running on prod: quay.io/prometheus/prometheus:v3.14.0-distroless
-# (`kubectl get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'`).
-PROMETHEUS_VERSION="${PROMETHEUS_VERSION:-3.14.0}"
-
-here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-tmp="$(mktemp -d)"
-trap 'rm -rf "$tmp"' EXIT
-
-promtool="${PROMTOOL:-$(command -v promtool || true)}"
-if [[ -z "$promtool" ]]; then
-  echo "==> promtool not on PATH; fetching v${PROMETHEUS_VERSION} (matches prod Prometheus)"
-  tarball="prometheus-${PROMETHEUS_VERSION}.linux-amd64"
-  curl -fsSL "https://github.com/prometheus/prometheus/releases/download/v${PROMETHEUS_VERSION}/${tarball}.tar.gz" \
-    | tar -xz -C "$tmp" --strip-components=1 "${tarball}/promtool"
-  promtool="$tmp/promtool"
-fi
-# Absolutised because step 3 runs promtool from inside $tmp — a relative PROMTOOL= would
-# break there, and it did during development.
-promtool="$(cd "$(dirname "$promtool")" && pwd)/$(basename "$promtool")"
-echo "==> $("$promtool" --version 2>&1 | head -1)"
-
-# 1. PrometheusRule CR -> bare Prometheus rules file, with ${BASE_DOMAIN} resolved.
-sed 's/\${BASE_DOMAIN}/base-domain.invalid/g' "$here/alerts.yaml" \
-  | python3 -c 'import sys,yaml; yaml.safe_dump(yaml.safe_load(sys.stdin)["spec"], sys.stdout, sort_keys=False, width=10**6)' \
-  > "$tmp/alerts.rules.yaml"
-cp "$here/alerts.promtool-test.yaml" "$tmp/"
-
-# 2. Syntax gate first — a rules file that does not parse would make every unit test
-#    "pass" vacuously by finding no rules at all.
-echo "==> promtool check rules"
-"$promtool" check rules "$tmp/alerts.rules.yaml"
-
-# 3. The unit tests.
-echo "==> promtool test rules"
-cd "$tmp"
-"$promtool" test rules alerts.promtool-test.yaml

platform/monitoring/README.md

@@ -18,7 +18,7 @@
 | `loki.yaml` | `HelmRelease` — Loki single-binary, filesystem, compactor-enforced retention |
 | `alloy.yaml` | `HelmRelease` — Grafana Alloy DaemonSet, tails pod logs → Loki |
 | `alerts.yaml` | Platform-health `PrometheusRule`s (R-OBS-04) |
-| `tests/` | `promtool test rules` unit tests over the committed `PrometheusRule`s — each rule pinned quiet on the good state and firing on the bad one (`tests/run.sh`, or `make -C ci prom-rules`) |
+| `tests/` | `promtool test rules` unit tests over the committed `PrometheusRule`s — each rule pinned quiet on the good state and firing on the bad one (`tests/run.sh` — the repo-wide harness, it also runs other components' companion `*.promtool-test.yaml`; enforced by the L0 check `alert-rules-unit-tested`) |
 | `kustomization.yaml` | builds all of the above |
 
 ## Requirement mapping

platform/monitoring/tests/backups.test.yaml

@@ -1,5 +1,6 @@
 # promtool rule-unit tests for the platform-health.backups group (platform/monitoring/alerts.yaml).
-# Run with: platform/monitoring/tests/run.sh   (or `make -C ci prom-rules`)
+# Run with: platform/monitoring/tests/run.sh   (the merge gate runs the same harness via
+#           conformance/tests/l0-static/alert-rules-unit-tested)
 #
 # Every rule is pinned in BOTH directions, because this repo has shipped both failure modes to
 # prod: a permanently-firing alert (CNPG, 2026-09-12) and a structurally inert one

platform/monitoring/tests/extract_rules.py

@@ -12,10 +12,19 @@ would let the fixture stay green while the deployed rule rots -- which is the cl
 this harness exists to catch (a permanently-firing CNPG alert and a structurally inert
 Velero alert both reached prod in 2026-09).
 
-Flux `postBuild` placeholders (`${BASE_DOMAIN}`) are left verbatim: no expression in
-platform/monitoring uses one, and leaving them alone keeps annotation text identical to
-the file under test. If a placeholder ever appears inside an `expr`, that is refused
-loudly rather than silently producing an untestable rule file.
+Flux `postBuild` placeholders (`${BASE_DOMAIN}`) are left verbatim: no expression in any
+tested manifest uses one, and leaving them alone keeps annotation text identical to the
+file under test. If a placeholder ever appears inside an `expr`, that is refused loudly
+rather than silently producing an untestable rule file.
+
+The cnpg-only script this replaced (platform/cnpg/run-alert-tests.sh, folded in 2026-09-12)
+SUBSTITUTED those placeholders instead, on the reasoning that "leaving a raw ${...} in a
+rendered rules file is exactly the kind of thing that gets copied somewhere it DOES
+matter". That hazard is real but it is addressed by WHERE the file goes, not by rewriting
+it: run.sh writes every generated rule file into a temp dir it deletes, so no rendered copy
+ever exists next to its source to be picked up. Verbatim is then the better default —
+one transformation fewer between the committed rule and the rule under test, and an
+`exp_alerts` assertion on annotation text still matches the file as shipped.
 
 Group names are made unique per source file so two files may legally reuse a name.
 

platform/monitoring/tests/run.sh

@@ -1,5 +1,5 @@
 #!/usr/bin/env bash
-# platform/monitoring/tests/run.sh — Prometheus RULE-UNIT-TEST harness (no cluster).
+# platform/monitoring/tests/run.sh — the repo's Prometheus RULE-UNIT-TEST harness (no cluster).
 #
 # WHY THIS EXISTS
 # This repo has now shipped both halves of the alerting failure space to prod:
@@ -17,76 +17,228 @@
 # PrometheusRule CRs into a rule file) — never a copy, so a fixture cannot drift green while
 # the deployed rule rots.
 #
-# Usage:  ./run.sh            # run every *.test.yaml here
-#         ./run.sh foo.test.yaml
+# ONE HARNESS, REPO-WIDE (consolidated 2026-09-12). This started as the monitoring-only
+# runner and there was a second, cnpg-only script (`platform/cnpg/run-alert-tests.sh`) doing
+# the same three steps by hand. Two harnesses means two behaviours, and the one nothing ran
+# rots — so the cnpg script is gone and this one covers EVERY PrometheusRule in the repo that
+# has unit tests. It still lives under platform/monitoring/ because observability owns alert
+# testing; the tests themselves live next to the rules they test.
+#
+# TWO TEST-FILE CONVENTIONS, both discovered automatically:
+#   1. COMPANION — `<manifest>.promtool-test.yaml` beside `<manifest>.yaml`. The manifest under
+#      test is that sibling. This is how a platform component ships tests with its own alerts
+#      (platform/cnpg/alerts.promtool-test.yaml).
+#   2. SUITE — `<dir>/tests/*.test.yaml`. The manifests under test are every PrometheusRule
+#      file in `<dir>` (platform/monitoring/tests/backups.test.yaml covers alerts.yaml +
+#      pvc-capacity-alerts.yaml, so one suite can assert across rule groups).
+#
+# WHAT THE ADAPTER DOES, AND WHY (inherited from the cnpg script's header, which was right):
+#   * A PrometheusRule is a Kubernetes CR; promtool wants a bare rules file with `groups:` at
+#     the top level. So `.spec.groups` is extracted (extract_rules.py), never re-typed.
+#   * The extracted file is written under whatever name the test file's own `rule_files:`
+#     declares, so a test file needs no knowledge of this harness.
+#   * `promtool check rules` runs FIRST. A rules file that does not parse would make every
+#     unit test "pass" vacuously by finding no rules at all.
+#   * Everything generated goes to a temp dir. A generated file sitting next to its source is
+#     how a stale copy becomes the thing under test — and it is also why leaving Flux's
+#     `${BASE_DOMAIN}` placeholders verbatim (see extract_rules.py) is safe here.
+#
+# Usage:  ./run.sh                       # every test file in the repo (both conventions)
+#         ./run.sh platform/cnpg/alerts.promtool-test.yaml   # one file
+#         ./run.sh --list                # print `<test-file>\t<manifest>` pairs, run nothing
+#
+# `--list` exists so the L0 check can ask THIS script which manifests are under test instead of
+# re-deriving the conventions itself. Two implementations of "which rules does this test cover"
+# would eventually disagree, and the disagreement would show up as a file the gate believes is
+# tested and the harness never touches. It needs no promtool, so it works on any host.
 #
 # Exit codes follow ci/lib.sh: 0 pass, 1 fail, 3 SKIPPED (toolchain unavailable — never green).
-# promtool is taken from $PATH if present, otherwise run from the pinned Prometheus image via
-# podman/docker. The image tag matches the version prod runs, so "passes here" means "passes
-# with the engine that will evaluate it".
+#
+# PROMTOOL COMES FROM THE PINNED TOOLCHAIN, AND NOWHERE ELSE. `PROMTOOL_VERSION` in
+# infra/devcontainer/versions.env (baked into the image) and ci/versions.env (installed on the
+# CI runner), pinned to the Prometheus version prod actually runs — promtool IS the rule
+# engine, so a test green under one version and red under another is worse than no test.
+# The two run-time acquisition paths this harness and the cnpg script used to have — a
+# `curl | tar -xz` of a GitHub release, and a `podman run` of quay.io/prometheus/prometheus —
+# are both REMOVED. Not because a pinned download is unusual here (the Containerfile downloads
+# every tool it bakes) but because those decided the version, the network and the machine
+# SILENTLY, at test time, on whatever host ran the tests instead of once at reviewed
+# image-build time; and each one quietly turned "the toolchain has no promtool" into a green
+# run, which is how the toolchain stayed without one. Missing promtool is now exit 3, never a
+# pass. Override with PROMTOOL=/path/to/promtool.
+#
+# ENFORCEMENT. This script is the developer entrypoint. The merge gate runs it through
+# conformance/tests/l0-static/alert-rules-unit-tested, which additionally reports every
+# PrometheusRule in the repo that has NO unit tests at all.
 set -euo pipefail
 
 HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-MONITORING_DIR="$(cd "${HERE}/.." && pwd)"
-PROM_IMAGE="${PROM_IMAGE:-quay.io/prometheus/prometheus:v3.14.0}"
+ROOT="$(cd "${HERE}/../../.." && pwd)"
 
-MANIFESTS=(
-  "${MONITORING_DIR}/alerts.yaml"
-  "${MONITORING_DIR}/pvc-capacity-alerts.yaml"
-)
-
-if [ "$#" -gt 0 ]; then
-  TESTS=("$@")
-else
-  mapfile -t TESTS < <(find "${HERE}" -maxdepth 1 -name '*.test.yaml' | sort)
-fi
-if [ "${#TESTS[@]}" -eq 0 ]; then
-  echo "run.sh: no *.test.yaml found in ${HERE}" >&2
-  exit 3
-fi
+LIST_ONLY=0
+if [ "${1:-}" = "--list" ]; then LIST_ONLY=1; shift; fi
 
 # ---------------------------------------------------------------- toolchain resolution
-RUNNER=""
-if command -v promtool >/dev/null 2>&1; then
-  RUNNER="native"
-elif command -v podman >/dev/null 2>&1; then
-  RUNNER="podman"
-elif command -v docker >/dev/null 2>&1; then
-  RUNNER="docker"
+# Skipped for --list: that mode runs no rules, and the L0 check must be able to resolve the
+# test->manifest mapping (and report the gaps) even on a host with no promtool.
+PROMTOOL_BIN=""
+if [ "$LIST_ONLY" = 0 ]; then
+PROMTOOL_BIN="${PROMTOOL:-$(command -v promtool || true)}"
+if [ -z "${PROMTOOL_BIN}" ]; then
+  pinned="$(sed -n 's/^PROMTOOL_VERSION="\{0,1\}\([^"]*\)"\{0,1\}.*/\1/p' \
+              "${ROOT}/ci/versions.env" 2>/dev/null | head -1)"
+  {
+    echo "run.sh: SKIP (exit 3, NOT a pass) — promtool is not on PATH, and this harness no"
+    echo "  longer fetches one (no GitHub release download, no container-image pull)."
+    echo "  promtool is pinned at v${pinned:-<see ci/versions.env>} — the Prometheus version prod runs:"
+    echo "    * infra/devcontainer/versions.env  PROMTOOL_VERSION   (baked into the image)"
+    echo "    * ci/versions.env                  PROMTOOL_VERSION   (installed on the CI runner)"
+    echo "  Run it in the pinned toolbox:"
+    echo "    ./infra/devcontainer/dev.sh build     # if the image predates the promtool pin"
+    echo "    ./infra/devcontainer/dev.sh run bash -c 'platform/monitoring/tests/run.sh'"
+    echo "  Or point at a local binary of the same version: PROMTOOL=/path/to/promtool ./run.sh"
+  } >&2
+  exit 3
+fi
+# A PROMTOOL= that does not resolve is an OPERATOR MISTAKE, not a missing toolchain: exit 2,
+# so a typo'd override cannot masquerade as the honest "no promtool here" skip above.
+if [ ! -x "${PROMTOOL_BIN}" ]; then
+  echo "run.sh: PROMTOOL='${PROMTOOL_BIN}' is not an executable file — fix the override" >&2
+  exit 2
+fi
+# Absolutised: each test runs with cwd inside a temp dir, where a relative PROMTOOL= breaks
+# (it did, during development).
+PROMTOOL_BIN="$(cd "$(dirname "${PROMTOOL_BIN}")" && pwd)/$(basename "${PROMTOOL_BIN}")"
+fi
+
+# ---------------------------------------------------------------- which test files to run
+# NOT `git ls-files`: the gate runs inside a container that mounts only the worktree, whose
+# `.git` file points outside the mount, so git cannot read the index there — and with
+# `pipefail` that failure silently emptied the discovery, i.e. a green run over ZERO tests
+# (observed 2026-09-12). `find` needs no repo metadata, so discovery cannot be disabled by
+# how the tree happens to be mounted. .git/ is pruned so a stale object copy is never picked up.
+discover() {
+  # Companion convention, anywhere in the tree.
+  find "${ROOT}" -path "${ROOT}/.git" -prune -o -name '*.promtool-test.yaml' -print 2>/dev/null
+  # Suite convention: this directory's own *.test.yaml files.
+  find "${HERE}" -maxdepth 1 -name '*.test.yaml' -print 2>/dev/null
+}
+
+TESTS=()
+if [ "$#" -gt 0 ]; then
+  for t in "$@"; do
+    # Accept absolute, repo-relative, or cwd-relative paths.
+    if [ -f "$t" ]; then TESTS+=("$(cd "$(dirname "$t")" && pwd)/$(basename "$t")")
+    elif [ -f "${ROOT}/$t" ]; then TESTS+=("${ROOT}/$t")
+    else echo "run.sh: no such test file: $t" >&2; exit 2
+    fi
+  done
 else
-  echo "run.sh: SKIP — no promtool on PATH and no podman/docker to run ${PROM_IMAGE}" >&2
+  while IFS= read -r abs; do
+    [ -n "$abs" ] && TESTS+=("$abs")
+  done < <(discover | sort -u)
+fi
+
+if [ "${#TESTS[@]}" -eq 0 ]; then
+  echo "run.sh: SKIP (exit 3) — no *.promtool-test.yaml or tests/*.test.yaml found in ${ROOT}" >&2
   exit 3
 fi
 
-WORK="$(mktemp -d)"
-trap 'rm -rf "${WORK}"' EXIT
+# ---------------------------------------------------------------- manifest resolution
+# Which committed PrometheusRule files a given test file tests. The conventions are the
+# CONTRACT (see the header); anything else is refused rather than guessed at, because a test
+# silently run against the wrong (or an empty) rule set is the vacuous pass this harness exists
+# to prevent.
+manifests_for() {
+  local test_abs="$1" base dir
+  base="$(basename "$test_abs")"
+  dir="$(dirname "$test_abs")"
+  case "$base" in
+    *.promtool-test.yaml)
+      local sib="${dir}/${base%.promtool-test.yaml}.yaml"
+      if [ ! -f "$sib" ]; then
+        echo "run.sh: ${test_abs#"${ROOT}/"} has no companion manifest (${sib#"${ROOT}/"})" >&2
+        return 1
+      fi
+      printf '%s\n' "$sib" ;;
+    *.test.yaml)
+      if [ "$(basename "$dir")" != tests ]; then
+        echo "run.sh: ${test_abs#"${ROOT}/"}: a *.test.yaml must live in a tests/ directory so its" >&2
+        echo "  manifests are the PrometheusRules of the component above it" >&2
+        return 1
+      fi
+      local comp; comp="$(dirname "$dir")"
+      local found=0 f
+      while IFS= read -r f; do
+        found=1; printf '%s\n' "$f"
+      done < <(grep -rl --include='*.yaml' -e '^[[:space:]]*kind:[[:space:]]*PrometheusRule[[:space:]]*$' \
+                 "$comp" --exclude-dir=tests 2>/dev/null | sort)
+      if [ "$found" = 0 ]; then
+        echo "run.sh: ${test_abs#"${ROOT}/"}: no PrometheusRule manifest in ${comp#"${ROOT}/"}" >&2
+        return 1
+      fi ;;
+    *)
+      echo "run.sh: ${test_abs#"${ROOT}/"}: unrecognised test-file name; use" >&2
+      echo "  <manifest>.promtool-test.yaml (companion) or tests/<name>.test.yaml (suite)" >&2
+      return 1 ;;
+  esac
+}
 
-# The test files reference the generated rule file by bare name, so everything must sit in one
-# directory (that is also the only directory the container needs mounted).
+# The rule-file name(s) a test file expects, from its own `rule_files:` block — so the
+# extracted rules land under the name the test already references and no test file needs to
+# know this harness exists.
+rule_file_names() {
+  sed -n '/^rule_files:[[:space:]]*$/,/^[^[:space:]-]/p' "$1" \
+    | sed -n 's/^[[:space:]]*-[[:space:]]*//p' | sed 's/[[:space:]]*$//' | grep . || true
+}
+
+# ------------------------------------------------------------------------------- --list
+if [ "$LIST_ONLY" = 1 ]; then
+  lrc=0
+  for test_abs in "${TESTS[@]}"; do
+    mapfile -t mans < <(manifests_for "$test_abs") || { lrc=1; continue; }
+    for m in "${mans[@]}"; do
+      printf '%s\t%s\n' "${test_abs#"${ROOT}/"}" "${m#"${ROOT}/"}"
+    done
+  done
+  exit "$lrc"
+fi
+
+# ---------------------------------------------------------------------------------- run
 rc=0
-python3 "${HERE}/extract_rules.py" --out "${WORK}/generated.rules.yaml" "${MANIFESTS[@]}" || rc=$?
-if [ "$rc" -eq 4 ]; then
-  echo "run.sh: SKIP — PyYAML unavailable, cannot flatten the PrometheusRule manifests" >&2
-  exit 3
-elif [ "$rc" -ne 0 ]; then
-  exit "$rc"
-fi
+for test_abs in "${TESTS[@]}"; do
+  rel="${test_abs#"${ROOT}/"}"
+  mapfile -t mans < <(manifests_for "$test_abs") || { rc=1; continue; }
+  mapfile -t rule_names < <(rule_file_names "$test_abs")
+  if [ "${#rule_names[@]}" -eq 0 ]; then
+    echo "run.sh: ${rel}: no \`rule_files:\` entries — promtool would test nothing" >&2
+    rc=1; continue
+  fi
 
-names=()
-for t in "${TESTS[@]}"; do
-  cp "$t" "${WORK}/"
-  names+=("$(basename "$t")")
+  work="$(mktemp -d)"
+  # The test file references its rule file by bare name, so everything sits in one directory.
+  gen="${work}/${rule_names[0]}"
+  ex_rc=0
+  python3 "${HERE}/extract_rules.py" --out "$gen" "${mans[@]}" || ex_rc=$?
+  if [ "$ex_rc" -eq 4 ]; then
+    echo "run.sh: SKIP (exit 3) — PyYAML unavailable, cannot flatten the PrometheusRule manifests" >&2
+    rm -rf "$work"; exit 3
+  elif [ "$ex_rc" -ne 0 ]; then
+    rm -rf "$work"; rc=1; continue
+  fi
+  # A test file may legally name the same rule file more than once, or several names.
+  for n in "${rule_names[@]:1}"; do cp "$gen" "${work}/${n}"; done
+  cp "$test_abs" "${work}/"
+
+  # Syntax gate first: rules that do not parse would make every unit test pass vacuously.
+  echo "run.sh: promtool check rules  ${rel}  (manifests: ${mans[*]#"${ROOT}/"})" >&2
+  ( cd "$work" && "$PROMTOOL_BIN" check rules "${rule_names[0]}" ) || rc=1
+  echo "run.sh: promtool test rules   ${rel}" >&2
+  ( cd "$work" && "$PROMTOOL_BIN" test rules "$(basename "$test_abs")" ) || rc=1
+  rm -rf "$work"
 done
-chmod -R a+rX "${WORK}"
 
-echo "run.sh: promtool test rules (${RUNNER}) ${names[*]}" >&2
-case "$RUNNER" in
-  native)
-    ( cd "${WORK}" && promtool test rules "${names[@]}" ) ;;
-  podman)
-    podman run --rm --user 0 -v "${WORK}:/rules:z" -w /rules \
-      --entrypoint /bin/promtool "${PROM_IMAGE}" test rules "${names[@]}" ;;
-  docker)
-    docker run --rm --user 0 -v "${WORK}:/rules" -w /rules \
-      --entrypoint /bin/promtool "${PROM_IMAGE}" test rules "${names[@]}" ;;
-esac
+if [ "$rc" -eq 0 ]; then
+  echo "run.sh: OK — ${#TESTS[@]} rule-unit test file(s) passed under $("$PROMTOOL_BIN" --version 2>&1 | head -1)" >&2
+fi
+exit "$rc"

runbooks/README.md

@@ -0,0 +1,142 @@
+# Alert runbooks — home-network-k3s prod
+
+This is the index page served at `https://home.keithjackson.dev/runbooks/`.
+
+Every alert rule in this platform carries an annotation of the form
+
+    runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"
+
+46 of them, over the 35 names below. Until this directory existed, that path returned **HTTP 404** on the
+live cluster: every alert that reached a human handed them a dead link at the exact moment they needed
+help. These pages are what makes the link resolve.
+
+## How to run the commands in these runbooks
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that hang
+is not the cluster being down.
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod`. `dev.sh` merges BOTH clusters into one kubeconfig and picks a
+current-context per invocation, and **that default has already changed once** — k3s-dev until dev was
+decommissioned on 2026-09-13, k3s-prod now. A command that names its context still means the same thing
+next month; one that relies on the default does not, and the failure is silent. `CLUSTER=prod
+infra/devcontainer/dev.sh run kubectl ...` selects it explicitly, and every invocation prints a banner
+naming the cluster — read it. Do not run two `dev.sh` invocations at once; they share one container.
+
+**Use fully-qualified resource names, always.** Three CRDs on this cluster register the kind `Backup` —
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1`, `velero.io/v1` — and the bare short name resolves to
+Longhorn's, which holds zero objects. `kubectl get backup -A` therefore prints **"No resources found"** on
+a cluster holding 6 Velero and 27 CNPG backups (verified on prod 2026-09-13). Acting on that false
+negative is how a CLI quirk becomes real data loss. Write `backups.velero.io`,
+`backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`,
+`volumes.longhorn.io`, `nodes.longhorn.io`.
+
+`git push origin main` **is** the deploy; there is no other apply step. Durable changes go through git.
+
+## Alerts you should not trust the silence of (measured 2026-09-13)
+
+Prometheus scrapes 19 jobs on prod. There is **no** Longhorn, cert-manager, Flux-controller or
+Hubble/Cilium job, so these rule groups currently evaluate against nothing and **cannot fire**:
+
+| Missing scrape job | Inert alerts |
+|---|---|
+| Longhorn | `LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `LonghornVolumeCapacityHigh`, `LonghornNodeStorageHigh` |
+| cert-manager | `CertificateExpiringSoon`, `CertificateExpiryImminent`, `CertificateNotReady` |
+| Flux (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` |
+| Hubble (`hubble_drop_total`) | `HubbleDeniedFlowSpike` |
+| etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all |
+
+Each affected runbook says so on its own page and gives the by-hand equivalent. See `targets-down`.
+
+## The runbooks
+
+### Nodes
+- **`node-down`** — a node's node-exporter is unscrapable; the node may be gone, or its kubelet may have
+  stopped posting status while the host stays up (the 2026-09-07 shape).
+- **`node-io`** — disk I/O pressure/starvation. The etcd WAL shares a partition with Longhorn replicas;
+  this is the alert that would have called the 2026-09-07 outage and did not exist at the time.
+- **`node-memory`** — memory pressure. Includes why `SwapTotal == 0` is not evidence swap is off.
+- **`node-disk`** — a node filesystem filling up. `/data` is the one that matters.
+- **`node-cpu`** — CPU PSI and run-queue depth, and how to tell CPU contention from I/O contention.
+
+### Storage
+- **`longhorn-degraded`** — a volume has lost redundancy. Normal after any reboot, and the rebuild is the
+  expensive part. Freeze/thaw commands here.
+- **`longhorn-faulted`** — no usable replica. Data at risk; stop before anything destructive.
+- **`longhorn-capacity`** — a volume or a node's Longhorn disk near capacity.
+- **`pvc-capacity`** — PVC filling up / almost full / predicted to fill. These rules work.
+- **`monitoring-capacity`** — Prometheus or Loki running out of disk; don't let the monitor blind itself.
+
+### GitOps and updates
+- **`flux-reconcile`** — a Flux resource stuck at Ready=False. Which tier decides the blast radius.
+- **`flux-suspended`** — reconciliation turned off and forgotten. Drift becomes silent.
+- **`update-rollback`** — a HelmRelease failed its health gate and is rolling back.
+- **`image-automation`** — the git write-back engine is failing (feature is disabled here).
+- **`image-policy-stalled`** — an ImagePolicy/ImageRepository cannot resolve a tag.
+
+### Certificates
+- **`cert-issuance`** — ACME issuance failing. The DNS-01 self-check is the usual cause.
+- **`cert-expiry`** — a certificate is expiring. Almost always a late symptom of `cert-issuance`.
+
+### Networking
+- **`vpn-down`** — the egress VPN tunnel. Includes the silent egress-gateway staleness that this alert
+  cannot see, and what the kill-switch actually does.
+- **`denied-flows`** — policy-denied flow spike. The Cilium hairpin/identity regression class.
+
+### Backups and DR
+- **`backup-failure`** — a Velero backup failed. The R2 object-tagging incompatibility is documented here.
+- **`backup-partial`** — a Velero backup completed with partial failures, and which skips are deliberate.
+- **`backup-missed`** — no successful Velero backup in 24h.
+- **`cnpg-archiving`** — WAL archiving stuck / no recoverability point. **An availability incident, not
+  just a backup one.**
+- **`cnpg-backup-missed`** — no recent CNPG base backup. Velero does not cover these databases.
+- **`cnpg-wal-growth`** — `pg_wal` not being recycled. Never delete WAL by hand.
+- **`cnpg-exporter`** — the CNPG SQL exporter erroring: a blind spot that looks exactly like health.
+- **`cnpg-primary-spread`** — two or more CNPG primaries on one node. Blast radius, not an outage,
+  and the rebalance costs ~9s of writes per database — one cluster at a time.
+
+### Identity
+- **`keycloak-down`** — the IdP is down. Blast radius, and the break-glass path that still works.
+- **`keycloak-db`** — Keycloak's CloudNativePG database is unavailable.
+- **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
+  every green signal missed.
+
+### Virtualisation
+- **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.
+- **`kubevirt-launch-failed`** — a VMI reached the Failed phase. The three k3s-specific launch traps.
+- **`kubevirt-crashloop`** — a virt-launcher pod restarting repeatedly.
+- **`cdi-import-stuck`** — a DataVolume import not progressing. Check node I/O before restarting it.
+
+## Editing a runbook
+
+Each page is `runbooks/<name>.md` in the `home-network-k3s-impl` repo. Edit it, push to `main`, and Flux
+re-renders the ConfigMap and rolls the two-replica server; kustomize's content hash on the generated
+ConfigMap name is what triggers the roll.
+
+Adding a **new** alert with a `runbook_url` requires the matching `runbooks/<name>.md` **and** its
+`<name>=<name>.md` entry in `runbooks/kustomization.yaml`. Both are enforced by
+`conformance/tests/l0-static/alert-runbooks-resolvable`, which runs unconditionally in the merge gate —
+"the file exists in git" and "the URL resolves in the cluster" are different claims, and it checks both.
+
+Every page must carry the four operator sections (*What this means here*, *First commands*, *Likely
+causes, most likely first*, *When to escalate or stop*), mention `dev.sh`, and pass `--context k3s-prod`.
+The check enforces that too, because a stub that makes the link resolve while helping nobody is worse than
+the 404 it replaced — it looks fixed.
+
+## Why this route has no login
+
+Deliberately public. The UIs that declare forward-auth (pihole, jackett, qbittorrent, radarr, sonarr)
+really are closed: enforcement is per-pod, each one's HTTPRoute targeting its oauth2-proxy sidecar on
+:4180 rather than the app port. If
+these pages did too, `keycloak-down` would be unreadable exactly when Keycloak is down,
+`keycloak-db` unreadable when its database is down, and `keycloak-client-secrets` unreadable during the
+precise failure that broke all 12 OIDC clients on 2026-09-06 behind an entirely green cluster. An
+operator who cannot log in is this route's primary reader. The content is operational procedure — metric
+names, thresholds, kubectl invocations, topology — and no secrets. See
+`runbooks/serving/httproute.yaml` for the full reasoning, including how forward-auth is actually
+enforced here and the instruction to leave this route **out** of any Gateway-level ext-authz attachment
+if one is ever introduced.
+
+The running narrative of real incidents on this cluster is `DECISIONS-LOG.md`.

runbooks/backup-failure.md

@@ -0,0 +1,173 @@
+# backup-failure — A Velero backup failed
+
+**Alert:** `VeleroBackupFailure` (critical, 5m)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.backups`
+**Expression:** `increase(velero_backup_failure_total[1h]) > 0`
+
+## What this means here
+
+A Velero backup in the last hour ended in `phase=Failed`. Off-cluster DR (R-STATE-03) is at risk. Velero
+backs up the **cluster's Kubernetes objects and its Longhorn PVC data**; it does **not** provide the DR
+path for the three CloudNativePG databases — CNPG owns those via its own Barman archive, because a
+Velero snapshot of a live Postgres is not a backup. So a Velero failure and a CNPG failure are separate
+incidents with separate runbooks (`/runbooks/cnpg-archiving`).
+
+## READ THIS BEFORE YOU CONCLUDE THE BACKUPS ARE GONE
+
+`kubectl get backup -A` prints **"No resources found"** on this cluster **even when backups exist**.
+Three CRDs register the kind `Backup` — `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and
+`velero.io/v1` — and the bare short name resolves to **Longhorn's**, which holds zero objects. Verified
+on prod 2026-09-13, where the fully-qualified queries returned **6 Velero backups** and **27 CNPG
+backups** while the short name returned nothing.
+
+This has already caused two false "the backups are missing" conclusions in one night. **A destructive
+recovery action taken on that false negative is how a monitoring quirk becomes real data loss.** Use:
+
+```
+backups.velero.io           backups.postgresql.cnpg.io           backups.longhorn.io
+schedules.velero.io         scheduledbackups.postgresql.cnpg.io  clusters.postgresql.cnpg.io
+```
+
+## Where the backups go
+
+Cloudflare **R2**, since the operator's 2026-09-05 decision (commit `32d6766`). Prod cannot reach the
+on-Proxmox MinIO at all — measured from a prod pod, *connection timed out after 15s* — and that MinIO
+ran on the same physical host as dev's VMs, so dev's "off-cluster" backup shared a failure domain.
+Live config: endpoint `https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com`, region
+**`auto`**, bucket `k3s-prod-backup`, prefix `velero`, schedule `0 3 * * *`, retention `720h`.
+`backupstoragelocation.velero.io/default` should read **Available**.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+Which backup failed, and why — fully-qualified, always:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get backups.velero.io -A \
+     -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,START:.status.startTimestamp,ERRORS:.status.errors,WARN:.status.warnings'
+```
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n velero get backups.velero.io <name> -o yaml | sed -n "/status:/,\$p"'
+```
+
+The storage location and the controller's log (the R2 error text is in the log, not the CR):
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n velero get backupstoragelocations.velero.io,schedules.velero.io; echo; \
+   kubectl --context k3s-prod -n velero logs deploy/velero --tail=150 2>&1 | grep -iE "error|NotImplemented|InvalidArgument|tagging|checksum" | tail -25'
+```
+
+Did the **volume data** upload even though the backup failed? (This distinction is the whole point —
+see cause 1.)
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n velero get podvolumebackups.velero.io \
+     -o custom-columns=NAME:.metadata.name,BACKUP:.metadata.labels.velero\\.io/backup-name,PHASE:.status.phase,BYTES:.status.progress.bytesDone'
+```
+
+The plugin pin (cause 1 depends entirely on this):
+
+```
+grep -n "velero-plugin-for-aws\|checksumAlgorithm\|region" platform/velero/helmrelease.yaml
+```
+
+## Likely causes, most likely first
+
+### 1. The R2 object-tagging incompatibility (the known, specific, repo-documented one)
+`velero-plugin-for-aws` **v1.14.2** sets PutObject's Tagging unconditionally —
+`Tagging: aws.String(o.tagging)` with `o.tagging == ""` — so the SDK emits `x-amz-tagging: ` (empty) on
+**every** PutObject. **R2 does not implement object tagging at all** and answers:
+
+```
+NotImplemented: Header 'x-amz-tagging' with value '' not implemented
+```
+
+Observed live on prod **twice**, on the write of `velero/backups/<name>/velero-backup.json`. The shape
+of the failure is the dangerous part: **kopia uploads the volume DATA successfully (its own S3 client),
+then the Backup goes `phase=Failed` with NO metadata object — an unrestorable backup.**
+**No configuration avoids it** (`config.tagging` with a real value sends a tag set R2 also rejects). The
+fix is upstream PR #309, which landed on release-1.14 *after* v1.14.2 was cut, so the image is pinned to
+**`velero/velero-plugin-for-aws:v1.14.3-rc.2`**.
+**EXIT CONDITION: move to v1.14.3 the moment it ships GA. Do NOT "clean this up" back to v1.14.2 — that
+silently breaks every prod backup's metadata.** If this alert fires and the pin has drifted, that is
+your answer.
+
+Two more mandatory R2 settings in the same file: `config.checksumAlgorithm: ""` (R2 rejects the AWS
+SDK's `x-amz-checksum-*` headers with **HTTP 400 InvalidArgument** — harmless against MinIO, fatal
+against R2) and `region` exactly **`auto`** (it was once hardcoded `us-east-1`, correct-by-accident for
+MinIO).
+
+### 2. A placeholder or wrong endpoint
+`clusters/prod` deliberately carried the **loud** placeholder
+`https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` rather than inheriting the base (dev) value,
+because *"an omitted op means prod silently inherits a dev address — far worse than a failed apply"*.
+Check `S3_BACKUP_ENDPOINT`, `S3_BACKUP_REGION`, `VELERO_BACKUP_BUCKET`, `VELERO_BACKUP_PREFIX` and
+`BACKUP_CREDENTIALS_SUFFIX` in `clusters/base/cluster-vars.yaml` and the prod overlay. A
+`backupstoragelocation` that is not `Available` points straight here.
+
+### 3. Credentials
+The R2 token lives in a SOPS-encrypted Secret (`velero-cloud-prod.sops.yaml`), and the `velero` tier
+carries its own `decryption` stanza. An expired or mis-scoped token gives 403s in the log.
+
+### 4. A volume could not be read
+Check `podvolumebackups.velero.io` for a specific volume. Note that NFS-sourced volumes are
+**deliberately skipped** by the `velero-volume-policies` ConfigMap — see `/runbooks/backup-partial`,
+which is where that belongs.
+
+### 5. The node or storage was unhealthy during the run
+Velero's daily runs at 03:00 and reads Longhorn PVCs. If a node was under I/O pressure or a volume was
+degraded, the read can fail. Cross-read `/runbooks/node-io` and `/runbooks/longhorn-degraded`.
+
+## When to escalate or stop
+
+- **Escalate promptly.** This is a `critical` because it is a DR alarm: the cluster may currently have no
+  recent restorable backup.
+- **Do not delete a Failed backup to "clean up".** A Failed Velero backup can leave **orphaned kopia
+  blobs** in the bucket — prod has two such backups from 2026-09-12 (`prod-proof-20260912`,
+  `prod-vp-probe-20260912`) whose orphaned blobs are **pending owner consent to delete**. Deleting
+  objects in R2 is an operator decision, not a tidy-up.
+- **Do not assume a Completed backup is restorable** if the plugin pin has drifted — the failure mode is
+  precisely "data uploaded, metadata missing".
+- **Be honest about restore**: `dr-restore` is **authored and has never been executed**
+  (`conformance.home.k3s/never-executed`), so the R2 restore path is unrehearsed. R2 egress is free,
+  which is the whole reason restore rehearsals are affordable — but nobody has done one yet.
+- The three CNPG databases are **not** covered by Velero. If the concern is a database, go to
+  `/runbooks/cnpg-archiving`.
+
+---
+Source of truth: `runbooks/backup-failure.md`. Index: `/runbooks/`
+Records: commit `2e259a8` and `platform/velero/helmrelease.yaml` lines ~68-95 (the R2 tagging
+incompatibility, `checksumAlgorithm`, `region: auto`); commit `32d6766` (the move to R2); commit
+`2d5171e` (Velero shipped with no ServiceMonitor, so these alerts could never fire).

runbooks/backup-missed.md

@@ -0,0 +1,161 @@
+# backup-missed — No successful Velero backup in 24 hours
+
+**Alert:** `VeleroNoRecentBackup` (critical, 1h)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.backups`
+**Expression:** `(time() - velero_backup_last_successful_timestamp{schedule!=""}) > (24 * 3600)`
+
+## What this means here
+
+A Velero **schedule** has not produced a successful backup in over a day. This is the alert that catches
+the quiet failure — not "a backup broke" (that is `/runbooks/backup-failure`) but "backups stopped
+happening and nobody noticed". The `{schedule!=""}` filter means it only watches scheduled backups, so
+ad-hoc ones do not mask a dead schedule.
+
+Live shape on prod: two schedules, both `0 3 * * *` —
+`schedule.velero.io/velero-daily` and `schedule.velero.io/velero-daily-resources-only` — retention
+`720h`, target Cloudflare R2 (bucket `k3s-prod-backup`, prefix `velero`, region `auto`).
+
+**Why this alert is worth its `critical`:** the precedent is that these rules *could not fire at all*.
+Velero shipped with three backup alerts and the chart default `metrics.serviceMonitor.enabled: false`,
+confirmed live on prod — `kubectl -n velero get servicemonitor` returned no resources. Recorded verbatim:
+*"A nightly backup could fail — or silently stop running — for weeks with nothing to say so."* The
+ServiceMonitor now exists (`velero` and `velero/node-agent` are both live scrape jobs as of 2026-09-13),
+which is the only reason this alert is real.
+## READ THIS BEFORE YOU CONCLUDE THE BACKUPS ARE GONE
+
+`kubectl get backup -A` prints **"No resources found"** on this cluster **even when backups exist**.
+Three CRDs register the kind `Backup` — `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and
+`velero.io/v1` — and the bare short name resolves to **Longhorn's**, which holds zero objects. Verified
+on prod 2026-09-13: the fully-qualified queries returned **6 Velero backups** and **27 CNPG backups**
+while the short name returned nothing. This false negative has already produced two wrong "the backups
+are missing" conclusions in one night. **A destructive recovery action taken on it is how a CLI quirk
+becomes real data loss.** Always:
+
+```
+backups.velero.io           backups.postgresql.cnpg.io           backups.longhorn.io
+schedules.velero.io         scheduledbackups.postgresql.cnpg.io  clusters.postgresql.cnpg.io
+```
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+Are the schedules enabled, and when did each last run?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n velero get schedules.velero.io \
+     -o custom-columns=NAME:.metadata.name,SCHEDULE:.spec.schedule,PAUSED:.spec.paused,LASTBACKUP:.status.lastBackup,PHASE:.status.phase'
+```
+
+What backups actually exist (fully-qualified — the short name lies):
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get backups.velero.io -A \
+     -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,SCHEDULE:.metadata.labels.velero\\.io/schedule-name,START:.status.startTimestamp --sort-by=.status.startTimestamp'
+```
+
+Is the storage location usable at all?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n velero get backupstoragelocations.velero.io \
+     -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,LASTVALIDATED:.status.lastValidationTime,DEFAULT:.spec.default'
+```
+
+Is Velero's own Flux tier reconciling, or was it suspended?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \
+     -o custom-columns=NAME:.metadata.name,SUSPENDED:.spec.suspend,READY:.status.conditions[?\(@.type==\"Ready\"\)].status | grep -Ei "velero|NAME"'
+```
+
+And the controller:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n velero get pods; \
+   kubectl --context k3s-prod -n velero logs deploy/velero --tail=120 2>&1 | tail -40'
+```
+
+## Likely causes, most likely first
+
+### 1. The Velero Flux tier was suspended and nobody resumed it
+This has happened: **Velero was found suspended live, with the suspension never recorded in git**, and
+was resumed with `flux resume`. A suspended backup controller is a silent DR gap and is precisely the
+class this alert exists for. Remediation (operator): `flux resume kustomization velero`, and see
+`/runbooks/flux-suspended` for why an undocumented live suspension is itself a problem.
+
+### 2. The schedule is paused
+`schedules.velero.io` has `spec.paused`. Note from the live listing that
+`velero-daily-resources-only` shows an empty PAUSED column while `velero-daily` shows `false` — read the
+field, not the column width. A paused schedule is a git change to unpause.
+
+### 3. Every run is failing rather than not running
+Then `backups.velero.io` will show recent `Failed` entries and this alert is a downstream symptom. Go to
+`/runbooks/backup-failure` — and in particular check the plugin pin
+(`velero/velero-plugin-for-aws:v1.14.3-rc.2`), because the R2 tagging bug produces exactly this: volume
+data uploads, the backup goes Failed, and no successful backup ever lands.
+
+### 4. The BackupStorageLocation is not Available
+A wrong endpoint, a bad credential, or the placeholder. `clusters/prod` deliberately carried the loud
+placeholder `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` rather than inheriting the base (dev)
+value, because *"an omitted op means prod silently inherits a dev address — far worse than a failed
+apply"*. Check `S3_BACKUP_ENDPOINT`, `S3_BACKUP_REGION` (`auto`), `VELERO_BACKUP_BUCKET`,
+`VELERO_BACKUP_PREFIX`, `BACKUP_CREDENTIALS_SUFFIX` (`-prod`).
+
+### 5. Velero cannot schedule a pod
+Node pressure, or the node-agent DaemonSet unhealthy. Cross-read `/runbooks/node-io`.
+
+### 6. The metric, not the backup, is missing
+If `velero_backup_last_successful_timestamp` has no series at all the rule cannot fire — which is the
+historical state. Confirm the `velero` scrape job is up (`/runbooks/targets-down`) before trusting either
+its silence or its firing.
+
+## When to escalate or stop
+
+- **Escalate: this is a DR alarm and 24h is already a real window of exposure.**
+- **Do not create an ad-hoc backup and consider the alert answered.** The expression filters
+  `{schedule!=""}`, so a manual backup will not clear it, and more importantly it does not fix the
+  schedule. (An operator's ad-hoc run for reassurance is
+  `velero backup create --from-schedule velero-daily`.)
+- **Do not conclude "no backups exist" from `kubectl get backup`.** See the trap above.
+- **Remember the databases are separate.** `keycloak-db`, `nextcloud-db` and `semaphore-db` are covered
+  by CNPG's Barman archive, not by Velero. This alert being clear says nothing about them, and vice
+  versa — check `/runbooks/cnpg-backup-missed` too.
+- **Be honest about restore:** `dr-restore` is authored and has **never been executed**, so the R2
+  restore path is unrehearsed. Free R2 egress makes a rehearsal cheap; it is still outstanding.
+
+---
+Source of truth: `runbooks/backup-missed.md`. Index: `/runbooks/`
+Records: commit `2d5171e` (Velero had no ServiceMonitor, so these alerts could never fire);
+`DECISIONS-LOG.md` (Velero found suspended with the suspension never in git; `dr-restore` never
+executed); commit `32d6766` (R2 decision and endpoint knobs).

runbooks/backup-partial.md

@@ -0,0 +1,164 @@
+# backup-partial — A Velero backup completed with partial failures
+
+**Alert:** `VeleroBackupPartialFailure` (warning, 5m)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.backups`
+**Expression:** `increase(velero_backup_partial_failure_total[1h]) > 0`
+
+## What this means here
+
+The backup finished, but some resources or volumes were skipped or errored. It is a warning rather than a
+critical because *most of it worked* — and that is exactly what makes it dangerous: a
+`PartiallyFailed` backup looks like a backup in every listing, and the thing it silently omitted may be
+the thing you need.
+
+**The first question is always: was the skip deliberate?** On this cluster a great deal of skipping is
+intentional and correct.
+## READ THIS BEFORE YOU CONCLUDE THE BACKUPS ARE GONE
+
+`kubectl get backup -A` prints **"No resources found"** on this cluster **even when backups exist**.
+Three CRDs register the kind `Backup` — `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and
+`velero.io/v1` — and the bare short name resolves to **Longhorn's**, which holds zero objects. Verified
+on prod 2026-09-13: the fully-qualified queries returned **6 Velero backups** and **27 CNPG backups**
+while the short name returned nothing. This false negative has already produced two wrong "the backups
+are missing" conclusions in one night. **A destructive recovery action taken on it is how a CLI quirk
+becomes real data loss.** Always:
+
+```
+backups.velero.io           backups.postgresql.cnpg.io           backups.longhorn.io
+schedules.velero.io         scheduledbackups.postgresql.cnpg.io  clusters.postgresql.cnpg.io
+```
+
+## The deliberate exclusions (know these before calling anything a failure)
+
+**NFS-sourced volumes are skipped on purpose, and the reason is a near-miss worth remembering.** The
+opt-out file-system-backup schedule (`defaultVolumesToFsBackup: true`, excluding only `monitoring` and
+`unifi`) would have read the NFS media library **three times** — `plex-media`, `sonarr-media` and
+`radarr-media` are all bound to the same `192.168.2.102:/mnt/Storage/plex` export, **8.32 TiB
+measured** — plus `nextcloud-legacy-archive` (270 GiB), pushing roughly **8.6 TiB to R2 nightly with
+720h retention**. Namespace exclusion could not express the fix, because those namespaces also hold real
+Longhorn PVCs that must be backed up.
+
+The fix is a **per-volume opt-out by SOURCE TYPE**: the ConfigMap `velero-volume-policies` skips any
+volume whose source is NFS, and the schedule references it. The per-pod
+`backup-volumes-excludes` annotation was **rejected for failing OPEN** (forget the annotation and you
+back up 8 TiB). Proven on prod: a Backup of namespace `nextcloud` produced exactly **one**
+PodVolumeBackup (`data`, Longhorn, 875,417,102 bytes, Completed) and the NFS volume produced none.
+**Residual gap, stated honestly: the rule matches NFS only.** A future non-NFS bulk source would not be
+covered.
+
+**The three CNPG databases are also out of scope by design.** `keycloak-db`, `nextcloud-db` and
+`semaphore-db` have their own DR path (Barman WAL archive + base backups to R2) because a Velero snapshot
+of a live Postgres is not a backup. If a `*-db-N` volume is skipped, that is correct — see
+`/runbooks/cnpg-archiving` for the path that does cover them.
+
+**`monitoring` and `unifi` are excluded** from file-system backup as well.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+Which backup, and what exactly it warned about:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get backups.velero.io -A \
+     -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,ERRORS:.status.errors,WARN:.status.warnings,START:.status.startTimestamp'
+```
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n velero get backups.velero.io <name> -o yaml | sed -n "/status:/,\$p"'
+```
+
+Per-volume results — this tells you whether a skip was the NFS policy or a real failure:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n velero get podvolumebackups.velero.io \
+     -o custom-columns=NAME:.metadata.name,BACKUP:.metadata.labels.velero\\.io/backup-name,POD:.spec.pod.name,VOL:.spec.volume,PHASE:.status.phase,BYTES:.status.progress.bytesDone'
+```
+
+The volume policy that does the deliberate skipping:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n velero get cm velero-volume-policies -o yaml'
+```
+
+The controller's own account of what it skipped:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n velero logs deploy/velero --tail=200 2>&1 | grep -iE "skip|excluded|warn|error" | tail -30'
+```
+
+## Likely causes, most likely first
+
+### 1. The deliberate NFS / CNPG / monitoring exclusions
+Most common by far, and correct. Confirm against the `velero-volume-policies` ConfigMap and the
+schedule's excluded namespaces. **No action.** If you are tempted to "fix" it, re-read the 8.6 TiB
+near-miss above.
+
+### 2. A resource Velero could not serialise
+Usually a CRD whose API was unavailable at the moment of the backup, or an admission webhook refusing a
+read. The log names the group/kind. Often transient; if it recurs for the same kind, that kind is
+genuinely not backed up.
+
+### 3. A volume was busy, degraded, or on a node under pressure
+Velero's daily runs at 03:00 against Longhorn PVCs. A degraded volume or a node with high I/O-PSI can
+fail one volume's read while the rest succeed. Cross-read `/runbooks/longhorn-degraded` and
+`/runbooks/node-io`.
+
+### 4. A partial R2 write
+Related to the tagging/checksum incompatibilities in `/runbooks/backup-failure`: those normally produce a
+hard `Failed`, but a partially-written backup is worth checking the plugin pin for
+(`velero/velero-plugin-for-aws:v1.14.3-rc.2` — do **not** revert to v1.14.2).
+
+### 5. A namespace was created or deleted mid-backup
+Benign race. Leaked test namespaces have existed on this cluster and been deleted; a namespace
+disappearing under Velero produces exactly this warning.
+
+## When to escalate or stop
+
+- **Decide, and record, whether the skipped items matter.** The honest failure mode here is shrugging at
+  a `PartiallyFailed` and discovering later that something was never covered. If the skip was not one of
+  the documented exclusions, treat it as an uncovered resource.
+- **Do not add namespaces to the exclusion list to quiet the alert.** Those namespaces hold real
+  Longhorn PVCs; namespace-level exclusion is the mistake the source-type policy exists to avoid.
+- **Do not switch to the per-pod `backup-volumes-excludes` annotation** — it was rejected for failing
+  OPEN, and the consequence of forgetting it is 8 TiB per night.
+- **Do not delete a PartiallyFailed backup** — like a Failed one it can leave orphaned kopia blobs, and
+  deleting R2 objects is an operator decision.
+- Escalate if the same resource or volume is skipped on consecutive nights: that is a coverage gap, which
+  is an R-STATE-03 matter and not a transient.
+
+---
+Source of truth: `runbooks/backup-partial.md`. Index: `/runbooks/`
+Records: commit `2e259a8` (the 8.6 TiB near-miss, the source-type volume policy, and the rejection of the
+fail-open annotation); `docs/BULK-NFS-STORAGE.md`.

runbooks/cdi-import-stuck.md

@@ -0,0 +1,172 @@
+# cdi-import-stuck — A CDI DataVolume import is not progressing
+
+**Alert:** `CDIDataVolumeImportStuck` (warning, 30m)
+**File:** `platform/kubevirt/alerts.yaml`, group `platform-health.kubevirt`
+**Expression:** `sum(kubevirt_cdi_import_pods_high_restart) > 0`
+
+## What this means here
+
+A CDI DataVolume import or clone has not completed after 30 minutes, so **the VM disk it backs cannot
+start**. `kubevirt_cdi_import_pods_high_restart` counts importer pods stuck restarting; a sustained
+positive value means the import is not making progress rather than merely being slow.
+
+The 30-minute `for` is generous on purpose: on this hardware a genuine large import is legitimately slow.
+The reference case is the **UniFi VM's 60 GiB import**, and the `apps-vms` tier is deliberately
+**non-gating** (`wait: false`, `dependsOn: platform-config`) precisely because a VM's image import plus
+in-guest install is legitimately long-not-Ready.
+
+**Two things to know before triaging:**
+
+1. **Imports are the single heaviest write this platform performs**, and the etcd WAL (`/data/k3s`) shares
+   one partition with every Longhorn replica (`/data/longhorn`). A 60 GiB import is exactly the class of
+   write load behind the 2026-09-07 `k3s-prod-1` NotReady outage. So "restart the import" is not a free
+   action — check `/runbooks/node-io` first.
+2. **There is a known false-positive precedent in this area.** A broken HTTP CDI ServiceMonitor once caused
+   a false `CDIOperatorDown` (commit `6a7e9d1`), and the sibling `KubeVirtControlPlaneDown` once used a
+   non-existent metric with the wrong job label (`b57b3e9`). The metric name in this rule is flagged in
+   the file as one to confirm against the deployed CDI. `cdi-prometheus-metrics` **is** a live scrape job
+   on prod (measured 2026-09-13), so the target is up — but verify the series exists before trusting either
+   this alert's firing or its silence.
+
+Platform versions: **CDI v1.66.0** with **KubeVirt v1.9.0**, on real nested KVM.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+Which DataVolume, and how far it got:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get datavolumes.cdi.kubevirt.io -A \
+     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PHASE:.status.phase,PROGRESS:.status.progress,RESTARTS:.status.restartCount'
+```
+
+The importer pod is where the real error is:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get pods -A | grep -E "importer|upload|clone"; echo; \
+   kubectl --context k3s-prod -n unifi logs -l app=containerized-data-importer --tail=100 2>&1 | tail -50'
+```
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n unifi describe pod -l app=containerized-data-importer 2>&1 | sed -n "/Events:/,\$p"'
+```
+
+Is there anywhere to put it — and is the scratch class configured?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get pvc -A | grep -Ei "scratch|dv|rootdisk"; echo; \
+   kubectl --context k3s-prod get sc; echo; \
+   kubectl --context k3s-prod get cdis.cdi.kubevirt.io -o yaml | grep -A5 scratchSpace'
+```
+
+Is the node being flattened by the import? **Check this before restarting anything:**
+
+```
+ssh k3s-prod-1 'cat /proc/pressure/io; df -h /data'
+```
+
+Can the importer reach the image source at all? (Default-deny is the baseline everywhere.)
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \
+     hubble observe --verdict DROPPED --last 100 -o compact | grep -i import'
+```
+
+Does the metric this alert depends on actually exist?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
+   for q in "up{job=\"cdi-prometheus-metrics\"}" "count(kubevirt_cdi_import_pods_high_restart)"; do \
+     printf "%-50s " "$q"; curl -s -G --data-urlencode "query=$q" localhost:19090/api/v1/query \
+       | python3 -c "import json,sys;print([x[\"value\"][1] for x in json.load(sys.stdin)[\"data\"][\"result\"]] or \"ABSENT\")"; done'
+```
+
+## Likely causes, most likely first
+
+### 1. Not enough schedulable storage for the target or the scratch space
+The recorded failure, and the reason a whole StorageClass exists: **the 3-replica default class failed the
+UniFi 60Gi import with insufficient storage / `ReplicaSchedulingFailure`.** The fix was a dedicated
+single-replica **`longhorn-scratch`** class (`numberOfReplicas: 1`, `dataLocality: best-effort`),
+referenced by the CDI CR's `scratchSpaceStorageClass`. If scratch is pointed at the 3-replica default,
+this is your cause. Note that a 60 GiB import needing 3x replication plus scratch space is a very
+different ask from 60 GiB.
+
+### 2. An egress or default-deny block on the image source
+Import pods are ordinary pods under the cluster's default-deny baseline (R-NET-01). The alert's own
+description names this: *"often an egress/default-deny block on the image source, or a slow mirror."*
+Remediation is an explicit allow in git; see `/runbooks/denied-flows` — and in particular the
+post-Cilium-upgrade finding that `toEntities: world` no longer covers a destination that resolves to an
+in-cluster backend.
+
+### 3. The importer cannot open the raw Block device
+`nonroot-devices: true` is set on the k3s servers because the **non-root CDI importer must open a raw
+Block device that Kubernetes never chowns for it** (commit `df5979f`). A permission error in the importer
+log against a Block-mode PVC is this. `longhorn-vm` is Block-mode only, so VM disks are always in scope.
+
+### 4. The source is slow, unreachable, or the URL is wrong
+A 404/403 or a stalled transfer appears plainly in the importer log. A slow mirror is a genuine cause — the
+30m `for` already allows for it, so check throughput rather than assuming a fault.
+
+### 5. Node I/O pressure is starving the import
+Self-reinforcing: the import is heavy, the disk is shared with etcd, and a node under I/O pressure slows
+the import which then runs longer. `/runbooks/node-io`.
+
+### 6. Longhorn is degraded
+An import onto a degraded volume competes with the rebuild. `/runbooks/longhorn-degraded`.
+
+### 7. The alert is wrong
+Given the CDI/KubeVirt false-positive precedents, if the DataVolumes all read `Succeeded` this is a
+monitoring fault. Fixing the rule is a real change, not a shrug — a permanently-firing alert is a
+recorded, expensive failure mode here.
+
+## When to escalate or stop
+
+- **Check node I/O pressure BEFORE restarting or recreating an import.** A 60 GiB re-import is hours of
+  write load on the partition the embedded etcd fsyncs to, and that combination took a node out of the
+  cluster on 2026-09-07. If I/O-PSI is already elevated, wait or freeze Longhorn rebuilds first
+  (`/runbooks/longhorn-degraded`).
+- **Do not delete the target DataVolume/PVC casually** — that guarantees a full re-import.
+- **Do not point `scratchSpaceStorageClass` at the 3-replica default** to "keep things consistent". The
+  single-replica class exists because the default one demonstrably failed.
+- **Do not widen a NetworkPolicy to `world`** to let an import out; author the narrow allow.
+- Scope it honestly: the VM tier is non-gating, so a stuck import does not affect the container platform.
+  The exception is the UniFi VM, which is the home network's controller.
+- Escalate for k3s server-flag changes (`nonroot-devices`) or CDI CR changes — Ansible and operator work.
+
+---
+Source of truth: `runbooks/cdi-import-stuck.md`. Index: `/runbooks/`
+Records: `DECISIONS-LOG.md` (the UniFi 60Gi import failing on the 3-replica default; `longhorn-scratch`);
+commits `df5979f` (`nonroot-devices`), `6a7e9d1` and `b57b3e9` (false CDI/KubeVirt alerts).

runbooks/cert-expiry.md

@@ -0,0 +1,146 @@
+# cert-expiry — A certificate is expiring soon, or imminently
+
+**Alerts:** `CertificateExpiringSoon` (warning, <14d, 1h) and `CertificateExpiryImminent` (critical, <3d, 1h)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.certificates`
+**Expression:** `(certmanager_certificate_expiration_timestamp_seconds - time()) < (14 * 24 * 3600)` / `(3 * 24 * 3600)`
+
+## What this means here
+
+cert-manager has a Certificate whose expiry is inside 14 days (warning) or 3 days (critical) and it has
+not renewed. cert-manager normally renews at two-thirds of lifetime — about 60 days into a 90-day Let's
+Encrypt certificate — so **by the time this fires, renewal has already been failing for weeks**. Treat
+it as a late symptom of `/runbooks/cert-issuance`, not as a separate problem: the fix is almost always
+"why is issuance failing", and expiry is the clock running out on that.
+
+What is at stake concretely. There is effectively **one certificate that matters**: `wildcard-home` in
+the `gateway` namespace, secret `wildcard-home-tls`, requesting **both** `*.${BASE_DOMAIN}` **and**
+`${BASE_DOMAIN}` from `${ACME_CLUSTER_ISSUER}` (= `letsencrypt-prod` on prod). Every HTTPS listener on
+the shared Gateway terminates with it — the wildcard `https` listener and the `https-apex` listener both
+reference `wildcard-home-tls`. If it expires:
+
+- every public hostname serves an untrusted certificate;
+- **SSO breaks in-cluster**, because OIDC clients validate the issuer's TLS. This is not hypothetical:
+  while the wildcard was still LE-**staging**, in-cluster clients rejected it with
+  `x509: certificate signed by unknown authority`, which is what forced the move to `letsencrypt-prod`;
+- and the page you are reading right now is served over that same certificate.
+
+**There is no history of an actual expiry on this cluster.** The recorded history is all issuance
+failures. Say so rather than inventing precedent — but note the corollary: the renewal path has never
+been exercised under stress, so do not assume it will quietly work.
+
+### Verify the alert can even fire (measured 2026-09-13)
+
+**This alert is currently INERT.** `count(certmanager_clock_time_seconds)` and
+`count(certmanager_certificate_expiration_timestamp_seconds)` both returned **0 series** on prod:
+there is no cert-manager scrape job among Prometheus's 19 jobs, so all three certificate rules evaluate
+against nothing. **Certificate expiry is unmonitored today.** Check expiry by hand with the commands
+below, and see `/runbooks/targets-down` for the full list of inert rule groups. Adding cert-manager's
+ServiceMonitor is open work.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+Every certificate and its real expiry, straight from the CRs (works with no metrics):
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get certificates.cert-manager.io -A \
+     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,NOTAFTER:.status.notAfter,RENEWAL:.status.renewalTime,ISSUER:.spec.issuerRef.name'
+```
+
+What the Gateway is actually serving (the answer that matters), read from outside the cluster with an
+explicit SNI — testing HTTPS from **inside** a pod is misleading here, and an IP+SNI probe without the
+right SNI returns curl exit 35:
+
+```
+curl -sv --resolve home.keithjackson.dev:443:192.168.2.20 https://home.keithjackson.dev/runbooks/ 2>&1 | grep -E "expire|subject|issuer|SSL certificate"
+```
+
+```
+echo | openssl s_client -connect 192.168.2.20:443 -servername grafana.home.keithjackson.dev 2>/dev/null | openssl x509 -noout -subject -issuer -dates
+```
+
+Is a renewal in flight or stuck?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get certificaterequests.cert-manager.io,orders.acme.cert-manager.io,challenges.acme.cert-manager.io -A; echo; \
+   kubectl --context k3s-prod -n cert-manager logs deploy/cert-manager --tail=80 | tail -40'
+```
+
+## Likely causes, most likely first
+
+### 1. Issuance has been failing since the last successful renewal
+The dominant cause, and the one with real history. Go to `/runbooks/cert-issuance` — it covers the
+DNS-01 self-check failure (fixed with `dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53"` and
+`dns01RecursiveNameserversOnly: true`), the Cloudflare token, and the ACME order/challenge chain. Fixing
+issuance fixes expiry; nothing else does.
+
+### 2. Two clusters racing the same `_acme-challenge` TXT record
+A specific, recorded hazard for any future two-cluster window. `infra/PROD-PROVISIONING.md` §10 records
+that after the 2026-09-04 incident, `platform-config` **would have** requested a
+`*.home.keithjackson.dev` wildcard from letsencrypt-prod for a certificate dev already held; both
+clusters would then have raced each other's `_acme-challenge` TXT records on one Cloudflare zone and
+**neither would reliably solve DNS-01**. It never fired only because the tier was suspended. If a second
+cluster exists, confirm which one owns the zone before renewing.
+
+### 3. Let's Encrypt rate limits
+One wildcard is well inside the limits, so this is unlikely — unless something has been retrying in a
+loop. `cert-manager` logs will say `too many certificates already issued`. Remediation: stop the loop,
+wait out the window; do **not** switch to letsencrypt-staging as a workaround, because in-cluster OIDC
+clients reject the staging chain (that is the bug that forced LE-prod).
+
+### 4. The issuer or its credential changed
+`${ACME_CLUSTER_ISSUER}` is a cluster-vars knob; the Cloudflare API token is a SOPS-encrypted Secret.
+Note the standing rule: **non-secret, environment-specific values must not live inside an encrypted
+Secret** — `${BASE_DOMAIN}` substitution can never reach inside one, which is exactly how prod's OIDC
+clients once ended up pointed at dev's Keycloak.
+
+### 5. The Certificate is Ready but the Secret is stale
+Check the Secret's actual notAfter (the `openssl` command above) rather than the CR's status. The Gateway
+reads the Secret.
+
+## When to escalate or stop
+
+- **`CertificateExpiryImminent` on `wildcard-home` is a cluster-wide outage in under 3 days**, including
+  SSO and including this runbook server. Get the operator immediately.
+- **Do not switch to `letsencrypt-staging`** to make issuance succeed. It will issue, and it will break
+  every in-cluster OIDC client with `x509: unknown authority`.
+- **Do not delete the `wildcard-home-tls` Secret** to force a reissue while issuance is broken — you
+  would replace a certificate that still has days left with none at all.
+- **Do not hand-edit the Secret.** cert-manager owns it; the durable change is in git.
+- Verify DNS at the **authority** (`dig @betty.ns.cloudflare.com _acme-challenge.home.keithjackson.dev TXT`),
+  never at a resolver: 1.1.1.1 has served a deleted TXT record for minutes here and produced a false
+  failure diagnosis.
+
+---
+Source of truth: `runbooks/cert-expiry.md`. Index: `/runbooks/`
+Records: `DECISIONS-LOG.md` "Fix 5 — cert-manager DNS-01 self-check via public resolvers" and the
+LE-staging → LE-prod switch (Session 7); `infra/PROD-PROVISIONING.md` §10 (the queued ACME race).

runbooks/cert-issuance.md

@@ -0,0 +1,165 @@
+# cert-issuance — A Certificate is not Ready (ACME issuance is failing)
+
+**Alert:** `CertificateNotReady` (warning, 1h)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.certificates`
+**Expression:** `certmanager_certificate_ready_status{condition="False"} == 1`
+
+## What this means here
+
+cert-manager cannot issue or renew a certificate. The one that matters is `wildcard-home` in the
+`gateway` namespace — secret `wildcard-home-tls`, requesting **both** `*.${BASE_DOMAIN}` **and**
+`${BASE_DOMAIN}` from `${ACME_CLUSTER_ISSUER}` (`letsencrypt-prod` on prod). Every HTTPS listener on the
+shared Gateway terminates with it, so issuance failing is a slow-burning cluster-wide outage: it does
+nothing today and takes down every hostname plus in-cluster SSO when the existing certificate expires.
+The 1h `for` reflects that — a failing order is worth investigating today, not paging at 3am.
+
+The solver is **DNS-01 against Cloudflare**, which matters because DNS-01 has a failure mode this
+cluster has already hit:
+
+**The self-check, not the challenge, is the usual failure.** On 2026-08-16 the wildcard stalled with the
+challenge at `Presented=true` — the Cloudflare TXT record *was* created, so the API token worked — while
+cert-manager's propagation self-check looped **"DNS record not yet propagated"**. The record was
+publicly resolvable at 1.1.1.1 and 8.8.8.8; the NAT'd cluster's own DNS could not see it, because
+split-horizon CoreDNS answers authoritatively for the zone and never forwards. The fix is live in
+`platform/cert-manager/helmrelease.yaml`:
+
+```yaml
+dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53"
+dns01RecursiveNameserversOnly: true
+```
+
+The challenge went pending → valid and the certificate Issued. If you find those two lines missing or
+changed, that is your answer.
+
+### Verify the alert can even fire (measured 2026-09-13)
+
+**This alert is currently INERT.** `count(certmanager_clock_time_seconds)` returned **0 series** on
+prod — there is no cert-manager scrape job among Prometheus's 19 jobs, so all three certificate rules
+evaluate against nothing. Certificate health must be read from the CRs, as below. See
+`/runbooks/targets-down` for the full list of inert rule groups.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+Walk the chain top-down — Certificate → CertificateRequest → Order → Challenge. The failure is almost
+always visible at one specific level:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get certificates.cert-manager.io -A \
+     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,NOTAFTER:.status.notAfter'
+```
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get certificaterequests.cert-manager.io -A; echo ==; \
+   kubectl --context k3s-prod get orders.acme.cert-manager.io -A; echo ==; \
+   kubectl --context k3s-prod get challenges.acme.cert-manager.io -A -o wide'
+```
+
+The controller log, which carries the actual ACME error text:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n cert-manager logs deploy/cert-manager --tail=150 2>&1 | grep -iE "error|propagat|challenge|order|rate" | tail -30'
+```
+
+Is the TXT record really there? **Ask the authority, never a resolver** — 1.1.1.1 has served a deleted
+TXT record here for minutes and produced a false failure diagnosis:
+
+```
+dig +short @betty.ns.cloudflare.com _acme-challenge.home.keithjackson.dev TXT
+dig +short @1.1.1.1 _acme-challenge.home.keithjackson.dev TXT
+```
+
+Is the resolver override still configured?
+
+```
+grep -n "dns01Recursive" platform/cert-manager/helmrelease.yaml
+```
+
+The issuer and its credential:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get clusterissuers.cert-manager.io \
+     -o custom-columns=NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message'
+```
+
+## Likely causes, most likely first
+
+### 1. The DNS-01 propagation self-check cannot resolve the TXT record
+The recorded failure. Signature: Challenge `Presented=true`, status looping on *"DNS record not yet
+propagated"*, and `dig @betty.ns.cloudflare.com` **does** return the TXT. Cause: cert-manager is asking
+the cluster's own resolver, which is split-horizon CoreDNS answering authoritatively for the zone and
+never forwarding. Remediation: the two `dns01Recursive*` values above, in git.
+
+### 2. The Cloudflare API token is wrong, expired, or scoped too narrowly
+Signature: Challenge never reaches `Presented=true`, and the log carries a Cloudflare 403/6003. The
+token lives in a SOPS-encrypted Secret. Note the standing rule from the OIDC incident: **non-secret
+environment-specific values must never live inside an encrypted Secret** — `${BASE_DOMAIN}`
+substitution cannot reach inside one, and that is precisely how prod's OIDC clients once ended up
+pointed at dev's Keycloak. Keep hostnames out; keep only the token in.
+
+### 3. `EXTERNALDNS_DOMAIN_FILTER` / zone confusion
+Recorded as finding F-DNS1: the filter **must be the ZONE** (`keithjackson.dev`), not the subdomain.
+A zone mismatch means the TXT record is created in the wrong place or not at all.
+
+### 4. Two clusters racing the same `_acme-challenge` record
+`infra/PROD-PROVISIONING.md` §10: after the 2026-09-04 incident `platform-config` would have requested a
+`*.home.keithjackson.dev` wildcard from letsencrypt-prod for a certificate dev already held; both
+clusters would then have raced each other's `_acme-challenge` TXT records on one Cloudflare zone and
+**neither would reliably solve DNS-01**. Only the tier being suspended prevented it. If any second
+cluster exists, establish which one owns the zone first.
+
+### 5. Let's Encrypt rate limit
+`too many certificates already issued` in the log. One wildcard is well inside the limits, so this means
+something has been retrying in a loop. Stop the loop and wait out the window.
+
+### 6. A new Certificate for a hostname that does not resolve
+A newly added service whose DNS is not yet published cannot solve DNS-01. Check ExternalDNS first.
+
+## When to escalate or stop
+
+- **Escalate if `wildcard-home` has been not-Ready for more than a day**, and check
+  `/runbooks/cert-expiry` for how much runway the existing certificate has. Under 3 days of runway is an
+  operator page.
+- **Never switch `${ACME_CLUSTER_ISSUER}` to letsencrypt-staging** to get a green certificate. Staging
+  issues fine and then breaks every in-cluster OIDC client with `x509: certificate signed by unknown
+  authority` — that is the bug that forced LE-prod in the first place.
+- **Do not delete the existing `wildcard-home-tls` Secret** while issuance is broken. A stale but valid
+  certificate is far better than none.
+- **Do not create certificates by hand.** cert-manager owns them; the durable change is in git.
+- Rotating the Cloudflare token is an operator action (SOPS, DR path).
+
+---
+Source of truth: `runbooks/cert-issuance.md`. Index: `/runbooks/`
+Records: `DECISIONS-LOG.md` "Fix 5 — cert-manager DNS-01 self-check via public resolvers"
+(commit ~`59dd3d9`) and the LE-staging → LE-prod switch; `infra/PROD-PROVISIONING.md` §10.

runbooks/cnpg-archiving.md

@@ -0,0 +1,198 @@
+# cnpg-archiving — WAL archiving is stuck, or a Postgres cluster has no recoverability point
+
+**Alerts:** `CNPGWALArchiveBacklog` (warning, 1h) and `CNPGNoRecoverabilityPoint` (warning, 6h)
+**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg`
+**Expressions:**
+```
+max by (namespace, pod) (cnpg_collector_pg_wal_archive_status{value="ready"}) > 0
+max by (namespace, job)  (cnpg_collector_first_recoverability_point) == 0
+```
+
+## What this means here — and why it is an AVAILABILITY incident, not just a backup one
+
+`pg_wal/archive_status` holds one marker per WAL segment: `.ready` = written but **not yet archived**,
+`.done` = archived and safe to recycle. A healthy archiver drains `.ready` within seconds, so a
+persistently non-zero ready count means `barman-cloud-wal-archive` is failing.
+
+**Two consequences, and the second one is the emergency:**
+
+1. There is no continuous archive, so **PITR is broken** (R-IDP-05 / R-STATE-03).
+2. **`archive_mode=on` means Postgres will NOT recycle an unarchived WAL segment.** `pg_wal` grows
+   without bound until the data PVC fills and the postmaster **PANICs**. `pg_wal` shares the **data**
+   PVC on every Cluster here — no `walStorage:` is configured on any of them — so there is no separate
+   volume to absorb it.
+
+This is measured, not theoretical. Prod ran **from bootstrap to 2026-09-07** with
+`endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`: all three Clusters at `ContinuousArchiving=False` with
+`barman-cloud-wal-archive: exit status 4` and `firstRecoverabilityPoint: null`, **for 38+ hours, with no
+alert of any kind** — found by hand. On `keycloak-db` that was 17 unarchived segments / ~268 MiB on a
+5 GiB PVC, growing **~6.8 MiB/h ≈ 29 days to disk-full** (about 579 MiB already burned by 2026-09-07).
+The rule recorded verbatim: *"Never leave a placeholder here, and treat `ContinuousArchiving=False` on
+this Cluster as an availability incident, not a backup one."*
+
+`CNPGWALArchiveBacklog` is *"the one that would have caught the 2026-09-06 incident within the hour
+instead of 38 hours later."* `> 0` with `for: 1h` — not a count threshold — because `for` requires the
+condition at **every** evaluation for the full hour, so a write burst resets the timer while a genuinely
+stuck archiver never returns to zero. A `> 5` threshold would have delayed `semaphore-db` indefinitely:
+its backlog sits at 3 because it writes almost nothing.
+
+## The two alerts are NOT the same statement
+
+`CNPGWALArchiveBacklog` is about the **mechanism** and is per-**pod** (each instance has its own `pg_wal`
+on its own PVC; a `> 0` shape is safe per-pod, and `pod` names the instance to go look at).
+
+`CNPGNoRecoverabilityPoint` is about **DR coverage** and is per-**CLUSTER**, grouped
+`by (namespace, job)`. That grouping is a fix, dated 2026-09-12: only the **primary** reads the backup
+catalogue, so a replica reports a hard `0` forever. Grouped by `pod`, this `== 0` fired **permanently on
+all six replicas** while all three Clusters were healthy. Verified live:
+old `max by (namespace, pod) (...) == 0` → 6 series (`keycloak-db-1/-3`, `nextcloud-db-1/-3`,
+`semaphore-db-1/-3`); new `max by (namespace, job) (...) == 0` → **empty** — and not vacuously so:
+forcing the metric to 0 for one primary yields exactly one series. `job` is the grouping label because
+the metric carries **no `cnpg.io/cluster` label**; the operator creates one PodMonitor per Cluster, so
+prometheus-operator derives `job="<namespace>/<cluster>"`: `keycloak/keycloak-db`,
+`nextcloud-db/nextcloud-db`, `semaphore-db/semaphore-db`. `namespace` alone happens to work today but
+would let two Clusters in one namespace mask each other.
+
+**Practical note:** if you see this alert firing on *pods* rather than per cluster, the per-cluster
+aggregation is not deployed on this cluster yet — check the deployed PrometheusRule before treating it
+as a real loss of DR.
+
+**Severity is `warning` on purpose.** A database heading for disk-full deserves weeks of lead time, and
+this repo has been burned by a permanently-firing `critical` (the `VPNTunnelDown` post-mortem). The
+`critical` tier for this failure mode is `PersistentVolumeAlmostFull`, which fires only when the outage
+is genuinely imminent.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+The three Clusters' archiving and recoverability status — the single most informative command:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
+     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,LASTBACKUP:.status.conditions[?\(@.type==\"LastBackupSucceeded\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint'
+```
+
+The archiver's actual error text — `exit status 4` and its cause live here:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=200 2>&1 | grep -i archive | tail -30'
+```
+
+The backlog on disk, per instance:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n keycloak exec keycloak-db-2 -c postgres -- bash -c \
+     "ls /var/lib/postgresql/data/pgdata/pg_wal/archive_status | grep -c ready; du -sh /var/lib/postgresql/data/pgdata/pg_wal; df -h /var/lib/postgresql/data"'
+```
+
+The backups that exist — **fully-qualified, or you will be told there are none**:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get backups.postgresql.cnpg.io -A; echo; \
+   kubectl --context k3s-prod get scheduledbackups.postgresql.cnpg.io -A'
+```
+
+`kubectl get backup -A` prints "No resources found" on this cluster even with 27 CNPG backups present:
+three CRDs register the kind `Backup` (`longhorn.io`, `postgresql.cnpg.io`, `velero.io`) and the short
+name resolves to Longhorn's, which holds zero. Verified on prod 2026-09-13. **Never decide a recovery
+action from the short name.**
+
+The object-store configuration, which is where the fault usually is:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -n keycloak keycloak-db -o yaml | sed -n "/backup:/,/^  [a-z]/p"'
+```
+
+```
+grep -nE "CNPG_BACKUP_PATH|S3_BACKUP_ENDPOINT|S3_BACKUP_REGION|BACKUP_CREDENTIALS_SUFFIX" clusters/base/cluster-vars.yaml clusters/prod/kustomization.yaml
+```
+
+## Likely causes, most likely first
+
+### 1. A wrong or placeholder object-store endpoint / path / credential
+The recorded cause of the 38-hour incident: prod ran with `endpointURL:
+TBD-PROD-S3-BACKUP-ENDPOINT`. The placeholder was **deliberately loud** because *"base defaults are the
+DEV values, so an OMITTED op means prod silently inherits a dev address — far worse than a failed
+apply"*. Live prod values to expect: `CNPG_BACKUP_PATH = s3://k3s-prod-backup/barman`,
+`S3_BACKUP_ENDPOINT = https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com`,
+`S3_BACKUP_REGION = auto`, `BACKUP_CREDENTIALS_SUFFIX = -prod`, retention `30d`, WAL compression gzip,
+`maxParallel: 2`. Remediation: fix it in git and push; Flux applies and the archiver drains on its own.
+
+### 2. R2-specific S3 incompatibilities
+Cloudflare R2 rejects things MinIO tolerates. The two that are mandatory elsewhere in this platform and
+worth checking here: `region` must be exactly `auto`, and AWS-SDK checksum headers are rejected with
+**HTTP 400 InvalidArgument**. See `/runbooks/backup-failure` for the full R2 story, including the Velero
+plugin's `x-amz-tagging` `NotImplemented` failure.
+
+### 3. Credentials expired or mis-scoped
+A SOPS-encrypted Secret selected by `BACKUP_CREDENTIALS_SUFFIX`. Standing rule from the OIDC incident:
+**non-secret, environment-specific values must NOT live inside an encrypted Secret** — `${BASE_DOMAIN}`
+substitution can never reach inside one. Keep endpoints and paths in cluster-vars, tokens in the Secret.
+
+### 4. Egress is blocked
+Default-deny is the baseline. The database namespaces need outbound HTTPS to R2. Cross-read
+`/runbooks/denied-flows`.
+
+### 5. A fresh Cluster that has not made its first base backup
+`firstRecoverabilityPoint` reads null until the first base backup lands; the `for: 6h` exists exactly to
+keep a legitimate cold start quiet. Base backups are staggered: keycloak 01:00, nextcloud 02:00,
+semaphore 04:00 (Velero at 03:00). If a Cluster is hours old, wait.
+
+### 6. Scaling out made it worse
+Noted honestly: scaling a Cluster while `archive_mode=on` with a failing `archive_command`
+**multiplies the disk-full clock**. Do not add instances as a remedy here.
+
+## When to escalate or stop
+
+- **Treat `ContinuousArchiving=False` as an availability incident with a countdown.** Compute the runway:
+  free bytes on the data PVC ÷ WAL growth rate. At ~6.8 MiB/h on a 5 GiB PVC that is weeks; on a busier
+  database it is days.
+- **NEVER delete WAL segments by hand.** It destroys the database. This is the one irreversible mistake
+  available in this runbook.
+- **Do not delete the `.ready` marker files.** Same outcome.
+- **Do not expand the PVC and stop there.** It buys time and hides the fault.
+- **Do not scale the Cluster** while archiving is failing.
+- **Verify before any restore** that `firstRecoverabilityPoint` is non-null and
+  `backups.postgresql.cnpg.io` shows completed backups. Prod's healthy state on 2026-09-13 was:
+  all three Clusters `ContinuousArchiving=True`, `LastBackupSucceeded=True`, `firstRecoverabilityPoint`
+  2026-09-07, 27 backups (9 per cluster) completing daily, with one historical failure
+  (`semaphore-db-daily-20260905130918`). That is what "good" looks like.
+- Escalate to the operator for credential or endpoint changes, and for any restore.
+
+---
+Source of truth: `runbooks/cnpg-archiving.md`. Index: `/runbooks/`
+Records: `platform/cnpg/alerts.yaml` header (the 2026-09-06 incident, the measured fill rate, and the
+2026-09-12 aggregation-scope fix with its live promtool proof);
+`platform/keycloak/postgres-cluster.yaml`; commit `32d6766` (the R2 decision).

runbooks/cnpg-backup-missed.md

@@ -0,0 +1,163 @@
+# cnpg-backup-missed — No recent CNPG base backup for a Postgres cluster
+
+**Alert:** `CNPGNoRecentBaseBackup` (warning, 2h)
+**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg`
+**Expression:**
+```
+(time() - max by (namespace, job) (cnpg_collector_last_available_backup_timestamp)) > (36 * 3600)
+and max by (namespace, job) (cnpg_collector_last_available_backup_timestamp) > 0
+```
+
+## What this means here
+
+The newest completed **base backup** for one Postgres cluster is over 36 hours old — one missed daily run
+plus slack. The daily `ScheduledBackup` has silently stopped producing backups, so the PITR window is
+ageing out against the `30d` retention policy.
+
+**Nothing else covers this.** The Velero rules in `platform/monitoring/alerts.yaml` watch Velero's own
+schedules, and **Velero does not back up these databases** — CNPG owns their DR path, because a Velero
+PVC snapshot of a live Postgres is not a backup. So `/runbooks/backup-missed` being clear says nothing
+about the databases, and vice versa.
+
+The three Clusters and their staggered base-backup times: **keycloak 01:00**, **nextcloud 02:00**,
+**semaphore 04:00** (Velero's own daily runs at 03:00, deliberately between them).
+
+## Two expression details that matter when you read the alert
+
+**The `> 0` guard is load-bearing.** A cluster that has *never* had a backup reports `0`, and
+`time() - 0` is about **57 years** — always true. Without the guard this rule would double-notify with
+`CNPGNoRecoverabilityPoint`, which is the correct alert for "never had a backup". So this rule means
+specifically: *this cluster has had at least one backup, and the newest one is stale.*
+
+**It is aggregated `by (namespace, job)`, not `by (namespace, pod)`** — fixed 2026-09-12 alongside
+`CNPGNoRecoverabilityPoint`. `cnpg_collector_last_available_backup_timestamp` has the same primary-only
+defect: measured on prod the same day, `keycloak-db-2` = 3.1h old, `nextcloud-db-2` = 2.1h,
+`semaphore-db-2` = 0.1h, and **all six replicas = 0**, i.e. `time() - 0` ≈ **56.7 years**. Unlike its
+sibling this rule was *not* firing — because the `> 0` guard was the single thing standing between it and
+six more permanent false positives, which is far too thin a margin for a DR alarm. Aggregating `pod` away
+removes the dependence: `max` picks the primary's timestamp as the cluster's answer (a demoted primary
+drops to 0 and can never win the `max`; a fresher backup always beats a staler one).
+`job` is the grouping label because the metric carries **no `cnpg.io/cluster` label** — the operator
+creates one PodMonitor per Cluster, so prometheus-operator derives `job="<namespace>/<cluster>"`:
+`keycloak/keycloak-db`, `nextcloud-db/nextcloud-db`, `semaphore-db/semaphore-db`.
+
+**Severity is `warning` on purpose** — this repo has been burned by a permanently-firing `critical`
+(the `VPNTunnelDown` post-mortem), and *"noise on a DR alarm is worse than no alarm at all"*.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+**Fully-qualified names, always.** `kubectl get backup -A` prints **"No resources found"** on this
+cluster even with **27 CNPG backups** present: three CRDs register the kind `Backup`
+(`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1`, `velero.io/v1`) and the bare short name resolves to
+**Longhorn's**, which holds zero objects. Verified on prod 2026-09-13. Concluding "the backups are gone"
+from that and acting on it is how a CLI quirk becomes data loss.
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get backups.postgresql.cnpg.io -A \
+     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,CLUSTER:.spec.cluster.name,PHASE:.status.phase,START:.status.startedAt,STOP:.status.stoppedAt --sort-by=.metadata.creationTimestamp | tail -20'
+```
+
+Are the schedules there and enabled?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get scheduledbackups.postgresql.cnpg.io -A \
+     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SCHEDULE:.spec.schedule,SUSPEND:.spec.suspend,CLUSTER:.spec.cluster.name,LAST:.status.lastScheduleTime'
+```
+
+The Cluster's own verdict (and the condition that tells you whether *archiving* is also broken):
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
+     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PRIMARY:.status.currentPrimary,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,LASTBACKUP:.status.conditions[?\(@.type==\"LastBackupSucceeded\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint'
+```
+
+Why the last attempt failed — the barman error is in the primary's logs:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=200 2>&1 | grep -iE "barman|backup|archive" | tail -30; echo ====; \
+   kubectl --context k3s-prod -n cnpg-system logs deploy/cnpg-controller-manager --tail=120 2>&1 | grep -iE "backup|error" | tail -20'
+```
+
+## Likely causes, most likely first
+
+### 1. The ScheduledBackup is suspended, missing, or unapplied
+Check `spec.suspend`, and whether the object exists at all for that Cluster. A `ScheduledBackup` that was
+never applied is the "looks done, is absent" failure class this repo keeps finding; verify the Flux tier
+that owns it is Ready (`/runbooks/flux-reconcile`) and not suspended (`/runbooks/flux-suspended`).
+
+### 2. The object store is rejecting the write
+Same root cause family as `/runbooks/cnpg-archiving`: endpoint, path, region or credentials. Live prod
+values to expect are `CNPG_BACKUP_PATH = s3://k3s-prod-backup/barman`,
+`S3_BACKUP_ENDPOINT = https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com`,
+`S3_BACKUP_REGION = auto`, `BACKUP_CREDENTIALS_SUFFIX = -prod`. **If `ContinuousArchiving` is also
+False, fix that first** — a cluster whose archiver is broken has a disk-full countdown running, which is
+more urgent than a stale base backup.
+
+### 3. A failed run that nobody noticed
+There is precedent: one historical failure, `semaphore-db-daily-20260905130918`. A single failed run does
+not clear itself; the next night's run should. If two consecutive nights fail, the cause is
+configuration, not luck.
+
+### 4. The primary changed and the new primary cannot write
+The backup catalogue is the **primary's** job. After a switchover, a credential or path problem scoped to
+one instance shows up here. Check which pod is `currentPrimary` and read that pod's logs, not another's.
+
+### 5. Node or storage trouble during the backup window
+Base backups are I/O-heavy and run at 01:00-04:00 alongside Velero. Cross-read `/runbooks/node-io`.
+
+### 6. The metric is missing, not the backup
+If `cnpg_collector_last_available_backup_timestamp` has no series, the rule cannot fire. Note that a
+**related** blind spot is real and separate: the CNPG SQL exporter can fail while `up` stays 1 — see
+`/runbooks/cnpg-exporter`.
+
+## When to escalate or stop
+
+- **A stale base backup narrows, but does not eliminate, recoverability** — provided WAL archiving is
+  healthy, PITR still works from the older base. That is why this is a warning. If **both** this and
+  `CNPGNoRecoverabilityPoint` / `CNPGWALArchiveBacklog` are lit, DR for that database is effectively
+  gone: escalate.
+- **Never delete WAL, `.ready` markers, or anything under a Postgres data directory.**
+- **Do not delete old CNPG Backup objects to tidy up.** Retention is `30d` and the operator manages it;
+  deleting a Backup object can orphan objects in R2.
+- **Do not trigger a base backup as a reflex** on a cluster whose archiver is failing — it adds write
+  load to a volume that may already be filling.
+- **Be honest about restore:** the CNPG restore path from R2 has not been rehearsed on this cluster
+  (`dr-restore` is authored and never executed). Free R2 egress makes a rehearsal cheap; it is
+  outstanding. A restore is an operator operation.
+
+---
+Source of truth: `runbooks/cnpg-backup-missed.md`. Index: `/runbooks/`
+Records: `platform/cnpg/alerts.yaml` (the 2026-09-12 aggregation-scope fix with its live measurements,
+the `> 0` guard rationale, and the staggered schedule).

runbooks/cnpg-exporter.md

@@ -0,0 +1,168 @@
+# cnpg-exporter — The CNPG metrics collector is erroring (a database is invisible)
+
+**Alert:** `CNPGMetricsCollectorErroring` (warning, 6h)
+**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg`
+**Expression:** `max by (namespace, pod) (cnpg_last_error) == 1`
+
+## What this means here
+
+**This is the monitor's own blind spot, for databases — a failure that looks exactly like health.**
+
+CloudNativePG publishes **two independent metric families** and only one of them depends on SQL:
+
+- `cnpg_collector_*` comes from the instance manager and is **always** present.
+- `cnpg_pg_stat_*` (archiver, database, replication) comes from the exporter logging in to the
+  application database as the `cnpg_metrics_exporter` role.
+
+When that login fails, `cnpg_last_error` goes to 1 and **the entire `cnpg_pg_stat_*` family blanks for
+that instance — while `up` stays 1 and `cnpg_collector_up` stays 1.** Nothing looks broken. The scrape
+target is green. The dashboards have gaps that read as "no data" rather than "broken".
+
+**This is not hypothetical, and it shaped the whole alert file.** Measured live on 2026-09-06 on both
+clusters, `cnpg_errors_total` carried:
+
+```
+pg_stat_archiver on db nextcloud: failed to connect to `user=cnpg_metrics_exporter database=nextcloud`:
+FATAL: permission denied for database "nextcloud" (SQLSTATE 42501)
+```
+
+`nextcloud-db` had been publishing **no** `pg_stat_archiver` / `pg_stat_database` metrics at all. The
+consequence recorded verbatim: *an archiving alert built on `pg_stat_archiver` — the obvious choice —
+would have been silently blind to one of three prod databases.* That is why every DR rule in
+`platform/cnpg/alerts.yaml` keys on `cnpg_collector_*` instead, and why this rule exists to alert on the
+blind spot itself.
+
+**Scope: `pod` is kept here, deliberately and unchanged.** `cnpg_last_error` is genuinely per-instance —
+each instance's own exporter reports its own SQL-collection health — and prod proved the metric
+discriminates rather than tracking role: all **three** `nextcloud-db` pods (including primary `-2`)
+reported 1 while all six `keycloak`/`semaphore` pods reported 0. There is no primary-only defect to fix,
+and per-pod is the right granularity because the remedy is a **GRANT applied per database** and you want
+to see it clear on every instance.
+
+`for: 6h` keeps a pod restart or a brief connection blip quiet. Severity stays `warning` so it never
+becomes the permanently-firing `critical` this repo has been burned by.
+
+**Live state, measured 2026-09-13:** `max by (namespace,pod) (cnpg_last_error) == 1` returned **empty**
+across the 9 instances that publish `cnpg_last_error` — so the `nextcloud` grant appears to have been
+addressed. Re-measure rather than assuming either way.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+Which instances are erroring:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
+   curl -s -G --data-urlencode "query=max by (namespace,pod) (cnpg_last_error)" localhost:19090/api/v1/query | python3 -m json.tool | grep -E "pod|\"1\""'
+```
+
+**The message itself** — `cnpg_errors_total` carries it in an `errorUserQueries` label. This is the one
+command that tells you the cause outright:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
+   curl -s -G --data-urlencode "query=cnpg_errors_total" localhost:19090/api/v1/query | python3 -m json.tool | head -60'
+```
+
+Or straight from the instance's metrics endpoint:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n nextcloud-db exec nextcloud-db-2 -c postgres -- \
+     curl -s localhost:9187/metrics | grep -E "^cnpg_last_error|^cnpg_errors_total|^cnpg_collector_up" | head -20'
+```
+
+Confirm which family is actually missing — this is what makes the blind spot visible:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
+   for q in "count(cnpg_collector_up)" "count(cnpg_last_error)" "count(cnpg_pg_stat_archiver_archived_count)" "count(cnpg_pg_stat_database_xact_commit)"; do \
+     printf "%-50s " "$q"; curl -s -G --data-urlencode "query=$q" localhost:19090/api/v1/query \
+       | python3 -c "import json,sys;print([x[\"value\"][1] for x in json.load(sys.stdin)[\"data\"][\"result\"]] or \"ABSENT\")"; done'
+```
+
+The exporter role's actual grants:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n nextcloud-db exec nextcloud-db-2 -c postgres -- \
+     psql -tAc "select rolname, rolcanlogin from pg_roles where rolname = \"cnpg_metrics_exporter\";" \
+     -c "select datname, has_database_privilege(\"cnpg_metrics_exporter\", datname, \"CONNECT\") from pg_database where datistemplate = false;"'
+```
+
+## Likely causes, most likely first
+
+### 1. The exporter role lacks CONNECT on the application database
+The recorded cause. Signature: `SQLSTATE 42501`, `permission denied for database "<name>"` in
+`cnpg_errors_total`. Remedy: **grant the `cnpg_metrics_exporter` role CONNECT on the application
+database.** Do it declaratively — CNPG can manage roles and the `managed.roles` / bootstrap surface in the
+Cluster spec is where this belongs, so the grant survives a re-seed. A `psql` GRANT by hand fixes the
+symptom on one instance and vanishes on the next bootstrap.
+
+### 2. Custom monitoring queries reference something that does not exist
+CNPG lets a Cluster attach extra collector queries via a ConfigMap. A query against a missing table,
+extension or column fails the whole collection pass for that instance. `cnpg_errors_total`'s label names
+the failing query.
+
+### 3. The database is not accepting connections
+A read-only/recovery state, `max_connections` exhausted, or the instance mid-restart. Cross-read
+`/runbooks/keycloak-db` if the instance is `keycloak-db-*`. Note the deliberate setting
+`dataDurability: preferred` (never `required`) exists so a degraded standby cannot block writes here.
+
+### 4. An upgrade changed the exporter's expectations
+CNPG **1.30.0** is the verified operator version on both clusters. A major operator or Postgres bump can
+change the collector's queries; if this started at an upgrade, read `/runbooks/update-rollback`.
+
+### 5. The instance is fine and the whole scrape is gone
+Then `up` for that `job` would be 0 and this is the wrong runbook — `/runbooks/targets-down`. The precise
+value of *this* alert is the case where `up` is **1** and the data is still missing.
+
+## When to escalate or stop
+
+- **Understand what you are currently blind to.** While this fires, `cnpg_pg_stat_archiver`,
+  `cnpg_pg_stat_database` and `cnpg_pg_stat_replication` are missing for that instance. The DR alerts in
+  `platform/cnpg/alerts.yaml` are deliberately built on `cnpg_collector_*` so they still work — **but do
+  not build any new alert or dashboard on `pg_stat_*` without checking this first.** That is the trap
+  this alert exists to mark.
+- **Do not silence it.** It is a warning with a 6h `for` precisely so it can be left visible without
+  becoming noise, and it was a **true positive** for months.
+- **Do not grant the exporter role broad privileges** to make it go away. CONNECT on the application
+  database is the whole requirement; anything more widens the blast radius of a metrics credential.
+- **Do not restart the database** to clear it — the error is a grant, not a state.
+- Escalate to the operator if the fix requires changing the Cluster's managed roles or bootstrap, since
+  that touches the database's declarative definition and its DR path.
+
+---
+Source of truth: `runbooks/cnpg-exporter.md`. Index: `/runbooks/`
+Records: `platform/cnpg/alerts.yaml` (the 2026-09-06 `SQLSTATE 42501` measurement on `nextcloud-db`, the
+two-metric-families finding, and the 2026-09-12 scope audit that deliberately kept `pod`).

runbooks/cnpg-primary-spread.md

@@ -0,0 +1,187 @@
+# cnpg-primary-spread — Two or more CNPG primaries share one node
+
+**Alert:** `CNPGPrimariesCoLocated` (warning, 30m)
+**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg`
+**Expression:** `count by (node) (<ready *-rw endpoints joined to cnpg_collector_up>) > 1`
+**and** gated on every node being Ready:
+`count(kube_node_status_condition{condition="Ready",status="true"} == 1) == count(kube_node_info)`
+
+## What this means here
+
+More than one CloudNativePG **primary** has been running on the same node for 30 minutes, while all
+three nodes are Ready. There are exactly three CNPG Clusters on this platform — `keycloak/keycloak-db`,
+`nextcloud-db/nextcloud-db` and `semaphore-db/semaphore-db` — so on a 3-node cluster the healthy shape is
+one primary per node, and this alert says that shape has drifted.
+
+**This is BLAST RADIUS, not an outage.** Each Cluster has `instances: 3` with one pod per node
+(`podAntiAffinityType: required`) and one synchronous standby (`method: any`, `number: 1`,
+`dataDurability: preferred`), so every one of them survives losing the shared node — you just get two or
+three simultaneous PostgreSQL failovers instead of one. Nothing is broken right now, which is why it is a
+`warning` with a long `for`.
+
+**Why it matters anyway: this exact drift has already caused an outage.** All three Clusters used to run
+`instances: 1` and all three primaries had drifted onto one node, and **on 2026-09-07 that node took SSO,
+Nextcloud and Semaphore down together.** `instances: 3` means the same drift no longer causes an outage —
+but if `keycloak-db`'s primary is co-located and that node dies, the SSO failover happens at the same
+moment as two others, on nodes that are also absorbing Longhorn replica rebuilds.
+
+**Why the drift happens and cannot be declared away: CNPG has no declarative primary-placement field.**
+The primary lands wherever the last promotion put it — a node reboot, a switchover, an operator upgrade.
+So this is avoidable drift that must be *corrected*, not prevented, and the alert re-arms itself so it
+will be reported again rather than silently persisting.
+
+**Why the node-readiness gate is in the expression.** During a real node outage, primaries legitimately
+pile onto the survivors and there is nothing to do about it; alerting then would be noise at the worst
+moment. The `and on ()` clause suppresses the rule unless every node is Ready, and re-arms when the node
+returns.
+
+This rule was proven in both directions on live prod data (2026-09-13): the expression returned
+`{node="k3s-prod-1"} => 3` before the rebalance and a **throwaway twin with `for: 1m`** was observed going
+`pending` at +70s and `firing` at +130s; after two switchovers the same expression returned an **empty
+vector while the inner join still resolved all three primaries** — quiet because the condition is false,
+not because the query went blind. It is also unit-tested in `alerts.promtool-test.yaml`
+(run `./run-alert-tests.sh`) with four cases, mutation-checked so that deleting the `cnpg_collector_up`
+join, the readiness gate, or shortening `for:` makes a test FAIL.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+Where the primaries are — the one command this alert is about:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get pods -A -l cnpg.io/instanceRole=primary \
+     -o custom-columns=NS:.metadata.namespace,POD:.metadata.name,NODE:.spec.nodeName'
+```
+
+Are all three Clusters actually healthy enough to move one? (Fully-qualified:
+`clusters.postgresql.cnpg.io` — `cluster` is an ambiguous short name.)
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
+     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,INSTANCES:.status.instances,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,PHASE:.status.phase'
+```
+
+Is every node Ready (the gate the rule itself applies)?
+
+```
+infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes -o wide'
+```
+
+Is there storage or I/O trouble that makes a switchover a bad idea right now?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
+     -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy'
+```
+
+```
+ssh k3s-prod-1 'cat /proc/pressure/io'
+```
+
+And confirm DR is intact before you deliberately move a primary — the primary owns the backup
+catalogue, and **fully-qualified names matter**: `kubectl get backup -A` prints "No resources found" on
+this cluster even with 27 CNPG backups present, because three CRDs register the kind `Backup` and the
+short name resolves to Longhorn's (zero objects):
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get backups.postgresql.cnpg.io -A | tail -10; echo; \
+   kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
+     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint'
+```
+
+## Likely causes, most likely first
+
+### 1. A node rebooted and the primaries never moved back
+The dominant cause. Promotions during the reboot put two primaries on a survivor, and CNPG has no reason
+to move them back afterwards. Remediation: a deliberate switchover, below.
+
+### 2. A rolling operator or Postgres upgrade
+`primaryUpdateMethod: switchover` is set deliberately (the CRD default `restart` is a
+delete-and-recreate of the *primary*, i.e. tens of seconds with no Postgres — an SSO outage on
+`keycloak-db`). Switchovers during an upgrade are correct behaviour and leave the primaries wherever
+they landed.
+
+### 3. One node was cordoned or unschedulable for a while
+Same mechanism. Note that `podAntiAffinityType: required` means an instance stays **Pending** until a
+node returns rather than co-locating — that is the accepted cost of the 2026-09-07 fix, and it is why
+the *instances* stay spread even when the *primaries* do not.
+
+### 4. A Longhorn or I/O event caused a health-check failover
+A primary whose storage stalled gets demoted. Cross-read `/runbooks/node-io` and
+`/runbooks/longhorn-degraded` — if that is why, fix the node before rebalancing.
+
+### 5. The alert is firing during a genuine node outage
+It should not be able to: the readiness gate suppresses it. If it fires anyway with a node NotReady, the
+gate is broken — that is a monitoring bug worth fixing, since a rule that cannot stop is fatigue.
+
+## Remediation — move one primary, deliberately
+
+The cnpg plugin is **NOT in the devcontainer image**; fetch the `kubectl-cnpg` release matching the
+operator version (CNPG **1.30.0** here) before you start.
+
+```
+kubectl cnpg promote -n <namespace> <cluster> <instance-on-an-idle-node>
+```
+
+**Do ONE CLUSTER AT A TIME and wait for `readyInstances: 3` before the next.** A switchover interrupts
+writes to that database for about **9 seconds** — measured on both clusters moved on 2026-09-13. Nine
+seconds of `keycloak-db` unavailability is a brief SSO interruption; doing two at once is an avoidable
+compound one.
+
+Pick the target by reading the primaries table above and choosing an instance that already sits on a node
+holding no other primary. Then re-run the table to confirm the new distribution, and let the alert clear
+on its own.
+
+## When to escalate or stop
+
+- **Do not rebalance during any other incident.** A switchover is a small, deliberate write interruption,
+  and it is the wrong thing to add to a node that is already under I/O pressure or to a cluster with
+  degraded Longhorn volumes. This alert has a 30-minute `for` precisely because it is never urgent.
+- **Never switch over a Cluster whose `readyInstances` is below 3.** You would be promoting into a
+  cluster that has no standby left to fall back on.
+- **Never change `dataDurability` to `required`** while thinking about durability here: it is a strict
+  quorum, and one degraded standby would then **block writes** entirely on the SSO database.
+- **Do not `kubectl delete pod` the primary to force a failover.** That is `restart` semantics — the
+  thing `primaryUpdateMethod: switchover` exists to avoid.
+- **Check `ContinuousArchiving` and `firstRecoverabilityPoint` first** (`/runbooks/cnpg-archiving`). If
+  archiving is broken, that is a real countdown to a disk-full PANIC and it outranks blast-radius
+  tidying completely.
+- Escalate to the operator for the switchover itself if the affected Cluster is `keycloak-db`: the brief
+  write interruption is user-visible across SSO, kubectl OIDC, Grafana, Headlamp and Gitea.
+
+---
+Source of truth: `runbooks/cnpg-primary-spread.md`. Index: `/runbooks/`
+Records: `platform/cnpg/alerts.yaml` (the rule's own both-directions proof on live prod data and its
+promtool tests); commits `f338e91` (instances 3), `e8d8601` (required anti-affinity + switchover),
+`cac413e` (synchronous replication, `dataDurability: preferred`); the 2026-09-07 single-node SSO outage.

runbooks/cnpg-wal-growth.md

@@ -0,0 +1,161 @@
+# cnpg-wal-growth — pg_wal is not being recycled
+
+**Alert:** `CNPGWALVolumeNotRecycling` (warning, 2h)
+**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg`
+**Expression:**
+```
+max by (namespace, pod) (cnpg_collector_pg_wal{value="count"})
+  > max by (namespace, pod) (cnpg_collector_pg_wal{value="max"})
+```
+
+## What this means here
+
+`pg_wal` is holding more WAL segments than `max_wal_size` allows for, which means **checkpoints are no
+longer reclaiming space**. On CNPG the units are **segments of 16 MiB**: `max` is `max_wal_size`
+(**64 = 1 GiB** on this platform) and `count` is what is currently sitting in `pg_wal`.
+
+This is the **storage escalation** of `/runbooks/cnpg-archiving`, not an independent fault.
+`CNPGWALArchiveBacklog` is the day-one signal — it fires within an hour of the archiver getting stuck.
+This one fires when the resulting backlog has grown large enough to be a *disk* problem. With
+`archive_mode=on` a failing `archive_command` is exactly what this looks like after a while, and because
+**`pg_wal` shares the data PVC on every Cluster here** (no `walStorage:` is configured on any of them)
+it ends in the volume filling and the postmaster **PANICking**.
+
+It does **not** fire today and is not meant to: measured, prod `keycloak-db` holds 17 segments against
+max 64 (dev held 36 against 64), and across all nine prod instances counts ran 16-37 against 64. At
+prod's observed ~6.8 MiB/h, `keycloak-db` would reach 64 segments roughly **40 days before** its 5 GiB
+PVC actually ran out — so this alert leads the PVC-fill alerts by weeks.
+
+**Why `pod` is kept here, and is required.** Every instance reports its own `count` and `max`, `pg_wal`
+is per-instance on a per-instance PVC, and the comparison only means anything pod-against-itself —
+aggregating `pod` away would compare one instance's `count` with another's `max`. A `>` shape is also
+safe per-pod: no instance can sit permanently true. (Contrast the `== 0` and staleness rules in the same
+file, which **must** aggregate `pod` away — that is the 2026-09-12 fix.)
+
+**One metric that looks right and is not:** the CNPG-native
+`cnpg_collector_pg_wal{value="volume_size"}` is **NaN on both clusters**, precisely because `pg_wal`
+shares the data PVC rather than having its own. Do not build anything on it.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+Which instance, and how far past `max_wal_size`:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
+   curl -s -G --data-urlencode "query=max by (namespace,pod) (cnpg_collector_pg_wal{value=\"count\"})" localhost:19090/api/v1/query | python3 -m json.tool | grep -E "pod|\"1\""'
+```
+
+The same thing from inside the instance, which also shows you the archive backlog and the free space in
+one view:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n keycloak exec keycloak-db-2 -c postgres -- bash -c \
+     "du -sh /var/lib/postgresql/data/pgdata/pg_wal; \
+      ls /var/lib/postgresql/data/pgdata/pg_wal | grep -c \"^0\"; \
+      ls /var/lib/postgresql/data/pgdata/pg_wal/archive_status | grep -c ready; \
+      df -h /var/lib/postgresql/data; \
+      psql -tAc \"show max_wal_size; show archive_mode; select pg_walfile_name(pg_current_wal_lsn());\""'
+```
+
+Is archiving the cause? (It almost always is.)
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
+     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint; echo; \
+   kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=200 2>&1 | grep -i archive | tail -20'
+```
+
+Is a replication slot or a stale standby holding WAL instead?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n keycloak exec keycloak-db-2 -c postgres -- \
+     psql -x -c "select slot_name, active, wal_status, safe_wal_size, restart_lsn from pg_replication_slots;" \
+     -c "select application_name, state, sync_state, replay_lag from pg_stat_replication;"'
+```
+
+Compute the runway — free bytes on the PVC divided by the observed growth rate:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
+   curl -s -G --data-urlencode "query=predict_linear(kubelet_volume_stats_available_bytes{job=\"kubelet\",namespace=\"keycloak\"}[24h], 7*24*3600)" localhost:19090/api/v1/query'
+```
+
+## Likely causes, most likely first
+
+### 1. WAL archiving is failing (dominant cause)
+Go to `/runbooks/cnpg-archiving` — that is the actual fix. Fix the `archive_command` and Postgres recycles
+the backlog on its own, without any intervention in `pg_wal`. Historical precedent: prod ran 38+ hours
+from bootstrap with `endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`, all three Clusters at
+`ContinuousArchiving=False` with `barman-cloud-wal-archive: exit status 4`.
+
+### 2. An inactive replication slot is pinning WAL
+A standby that has been gone long enough leaves a slot holding `restart_lsn`, and Postgres keeps every
+segment after it. `pg_replication_slots.wal_status` will read `extended` or `lost`. The healthy fix is to
+bring the standby back (CNPG manages the slots), not to drop the slot by hand. Note
+`podAntiAffinityType: required` means a lost instance stays **Pending** until a node returns — expected,
+and accepted.
+
+### 3. Genuinely heavy write load
+Then `count` exceeding `max_wal_size` is checkpointing lagging, not a leak, and it self-corrects. Confirm
+that `archive_status/*.ready` is near zero: a large `pg_wal` with an empty ready queue is a busy
+database, not a stuck archiver.
+
+### 4. `max_wal_size` was lowered
+64 segments = 1 GiB is the configured value on both clusters. A reduction makes this rule fire without
+anything actually being wrong. Check the Cluster's postgresql parameters — and note that under
+`primaryUpdateMethod: switchover` the operator **rejects an update that changes `imageName` and a
+postgresql parameter in the same reconcile**.
+
+### 5. The instance is a replica that fell behind
+Replay lag holds WAL on the primary. `pg_stat_replication.replay_lag` shows it.
+
+## When to escalate or stop
+
+- **NEVER delete WAL segments or `archive_status` marker files by hand.** It destroys the database. This
+  is the single irreversible mistake available here, and the alert's own description says so: *"Fix
+  archiving — do NOT delete WAL by hand."*
+- **Do not drop a replication slot** to free space without the operator: you may make a lagging standby
+  unrecoverable and force a full re-seed.
+- **Do not expand the PVC and stop there.** It buys days and hides the fault.
+- **Do not scale the Cluster out** while archiving is failing — that **multiplies** the disk-full clock.
+- Escalate when the computed runway is under a week, or when `PersistentVolumeAlmostFull`
+  (`/runbooks/pvc-capacity`) is also firing for the same PVC — at that point an outage is imminent and a
+  Postgres data volume under 5% free will PANIC the postmaster.
+
+---
+Source of truth: `runbooks/cnpg-wal-growth.md`. Index: `/runbooks/`
+Records: `platform/cnpg/alerts.yaml` (the measured segment counts, the NaN `volume_size` finding, and the
+2026-09-12 scope audit that deliberately KEPT `pod` here).

runbooks/denied-flows.md

@@ -0,0 +1,154 @@
+# denied-flows — A spike in policy-denied network flows
+
+**Alert:** `HubbleDeniedFlowSpike` (warning, 10m)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.networking`
+**Expression:** `sum(rate(hubble_drop_total{reason="POLICY_DENIED"}[5m])) > 5`
+
+## What this means here
+
+Cilium is dropping more than 5 flows/second on policy. East-west traffic is **default-deny** everywhere
+(R-NET-01): a bare workload reaches DNS and nothing else, and every allow is declared per
+source → destination → port (R-NET-02). So a denied-flow spike is almost always **a NetworkPolicy
+regression** — something legitimate that is no longer allowed — and only rarely a probe.
+
+### Verify the alert can even fire (measured 2026-09-13)
+
+**This alert is currently INERT.** `count(hubble_drop_total)` returned **0 series** on prod: there is no
+Hubble/Cilium scrape job among Prometheus's 19 jobs, so Hubble metrics are not being collected and this
+rule evaluates against nothing. **Policy drops are unmonitored today** — you will hear about them as
+"service X cannot reach service Y", and you diagnose them with `hubble observe`, below. See
+`/runbooks/targets-down` for the full list of inert rule groups.
+
+### The regression class that has actually bitten this cluster
+
+**Cilium's CVE-2025-30162 hairpin change (the expensive one).** After the 1.16 → 1.20 upgrade, the
+shared Gateway's L7 proxy re-checks a pod's egress policy against the destination's **real backend
+identity**. So `toEntities: world` **no longer covers an in-cluster call to a `*.home.keithjackson.dev`
+hostname** — the destination resolves to an in-cluster backend, not to `world`. Every app's SSO call
+started returning **403 with `server: envoy`**. The fix is an explicit `toEndpoints` allow to the
+Keycloak backend, which `catalog/render.py` now emits for every service with
+`auth.mode ∈ {oidc, forward-auth}`.
+
+**The durable rule that follows: any new app → app-by-hostname dependency needs its own explicit allow.**
+If a newly added or newly SSO-enabled service is 403ing or timing out, this is the first thing to check.
+
+A second recorded instance of the same class: Flux's `source-controller` was denied by
+`gitea-allow-gateway-ingress`, whose ingress rule was `fromEntities: [ingress]` only — so an in-cluster
+`wget` to Gitea simply timed out. And a matching diagnostic trap: **testing HTTPS from inside a pod is
+misleading here**; test from outside with `curl --resolve`.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+See the actual drops — this is the tool, since the metric is not collected:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \
+     hubble observe --verdict DROPPED --last 200 -o compact'
+```
+
+Narrow to a suspected pair:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \
+     hubble observe --verdict DROPPED --from-namespace nextcloud --last 100 -o compact'
+```
+
+What policies apply to the source workload, and is the policy actually enforced?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get ciliumnetworkpolicies.cilium.io,networkpolicies.networking.k8s.io -A; echo; \
+   kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg endpoint list | head -30'
+```
+
+Cilium's own drop counters, as a cross-check:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg metrics list | grep -i drop'
+```
+
+Is the 403 an Envoy/L7 verdict rather than an L3/L4 drop? (The hairpin class above shows up here, not in
+`hubble observe`.)
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n <app-ns> logs <pod> --tail=50 | grep -iE "403|envoy|oidc|well-known"'
+```
+
+## Likely causes, most likely first
+
+### 1. A newly added or newly SSO-enabled service has no explicit allow to Keycloak
+The CVE-2025-30162 hairpin class described above. Signature: HTTP **403 with `server: envoy`** on a call
+to `keycloak.${BASE_DOMAIN}` or another `*.home` hostname from inside the cluster, while the same URL
+works from outside. Remediation: the allow is **rendered**, not hand-written — set `auth.mode` in
+`catalog/<service>.yaml` and re-render so `render.py` emits the `toEndpoints` allow to the keycloak
+backend. Do not patch a CiliumNetworkPolicy by hand; it will be overwritten.
+
+### 2. A platform component was moved or a namespace label changed
+`fromEntities: [ingress]`-only ingress rules (the Gitea/source-controller case) break any in-cluster
+client. Remediation: add the specific allow in the component's `networkpolicy.yaml`, in git.
+
+### 3. A missing dependency declaration in the catalog
+`dependencies:` in a ServiceDefinition is what generates app→app allows (R-NET-02). A service talking to
+something it never declared is denied by design. The declaration is the fix.
+
+### 4. A scrape target that nothing allows
+Prometheus selects **every** ServiceMonitor in the cluster
+(`serviceMonitorSelectorNilUsesHelmValues: false`), so a new workload's monitor is scraped immediately
+and will be denied until the component allows ingress from `monitoring`. Cross-read
+`/runbooks/targets-down`.
+
+### 5. An actual probe or scan
+Possible, and it is why the alert is worded as a security signal (docs/07). Distinguish it by the
+**source**: a spike from outside the cluster or from an unexpected identity, rather than one named
+workload repeatedly failing one destination. Escalate rather than tuning policy.
+
+## When to escalate or stop
+
+- **Do not widen a policy to make drops stop.** Default-deny is a safety-critical requirement here
+  (R-NET-01 is flagged `safety_critical: true` in the requirement registry). The correct fix is the
+  narrowest explicit allow, authored in git, usually via the catalog so it is rendered consistently.
+- **Do not hand-edit a rendered CiliumNetworkPolicy** — it comes from `catalog/render.py` and will be
+  reverted on the next reconcile.
+- **Escalate if the source is unexpected.** An unexplained spike from outside the cluster is a security
+  event, not a configuration bug.
+- If drops coincide with a Cilium upgrade, assume the hairpin/identity class above before assuming a
+  policy authoring mistake — the upgrade changed the *evaluation*, not the policy.
+- Note the diagnostic trap when verifying a fix: probe from **outside** with `curl --resolve`, because
+  in-pod HTTPS tests to the platform's own hostnames are misleading.
+
+---
+Source of truth: `runbooks/denied-flows.md`. Index: `/runbooks/`
+Records: `DECISIONS-LOG.md` (the Cilium CVE-2025-30162 world-hairpin / envoy 403 finding and its
+`render.py` fix; source-controller denied by `gitea-allow-gateway-ingress`).

runbooks/flux-reconcile.md

@@ -0,0 +1,164 @@
+# flux-reconcile — A Flux resource has failed to reconcile for 15 minutes
+
+**Alert:** `FluxReconciliationFailure` (warning, 15m)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.flux`
+**Expression:** `max by (namespace, name, kind) (gotk_reconcile_condition{type="Ready", status="False"}) == 1`
+
+## What this means here
+
+Something in the GitOps chain is stuck at Ready=False. On this cluster that is more consequential than
+it sounds, because **`git push origin main` IS the deploy** — there is no other apply step. A stuck
+Kustomization means git and the cluster have diverged and nothing will close the gap.
+
+The chain has **14 tiers**: `flux-system` → `cilium` → `platform` → `platform-config` →
+`platform-config-cdi` → `apps-wave-0/1/2`, plus the non-gating leaves `apps-vms`, `oauth2-proxy`,
+`headlamp`, `homepage`, `crafty`, `runbooks`, `backup`. Which tier is failing decides the blast radius:
+
+- **`flux-system`, `cilium`, `platform`, `platform-config` are gating (`wait: true`).** One Failed
+  HelmRelease in the `platform` tier stalls **everything downstream** on "dependency not ready". That
+  is not theoretical: a crashlooping Headlamp HelmRelease once stalled `platform-config`,
+  `oauth2-proxy` and every apps wave (the "Headlamp gating stall"), and oauth2-proxy twice deadlocked
+  by **blocking its own fix** — it sat in the gating tier while needing Keycloak + trusted TLS +
+  split-horizon DNS, so `platform-config`, which applies the fix, could never reconcile. Both were
+  resolved by moving the leaf into its own `wait: false` tier.
+- **The leaf tiers are `wait: false`** precisely so their unhealth can never gate anything.
+
+Prod's source is `http://gitea.gitea.svc.cluster.local:3000` — the **in-cluster Service**, on purpose.
+It used to be a public hostname, and the apex flip taught the expensive version of that lesson: the
+HTTPRoute serving the source's own hostname lives in a downstream tier that cannot reconcile while the
+source is down, so `srcReady=False` / *"unable to list remote"* became a **deadlock**. If you are
+tempted to point the GitRepository at a public name, don't.
+**This alert is currently INERT.** `count(gotk_reconcile_condition)` and `count(gotk_suspend_status)`
+both returned **0 series** on prod on 2026-09-13: there is no scrape job for the Flux controllers in
+Prometheus's 19 jobs, so every `gotk_*`-based rule (`FluxReconciliationFailure`,
+`FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`,
+`UpdateHealthGateFailed`) evaluates against nothing. **Flux health is therefore unmonitored today** and
+must be read from the CRs directly, with the commands below. See `/runbooks/targets-down` for the full
+list of inert rule groups. Adding the Flux controllers' ServiceMonitor is open work.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+The whole chain at a glance — this is the command to run first, every time:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \
+     -o custom-columns=NAME:.metadata.name,PATH:.spec.path,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,REV:.status.lastAppliedRevision'
+```
+
+Is the SOURCE healthy? If not, nothing downstream can be:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n flux-system get gitrepositories.source.toolkit.fluxcd.io \
+     -o custom-columns=NAME:.metadata.name,URL:.spec.url,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message'
+```
+
+HelmReleases (the usual culprit inside `platform`):
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get helmreleases.helm.toolkit.fluxcd.io -A \
+     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message'
+```
+
+The controller's own view, with the real error text:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n flux-system logs deploy/kustomize-controller --tail=80; echo ====; \
+   kubectl --context k3s-prod -n flux-system logs deploy/helm-controller --tail=80'
+```
+
+## Likely causes, most likely first
+
+### 1. "no matches for kind X" — a CR applied before its CRD
+The recurring structural failure here. Flux server-side-apply fails the **whole** apply, so a tier that
+ships both a CRD-installing HelmRelease and CRs of that CRD can never converge — the CRD never gets
+created because the apply that would create it also contains the CR. Symptoms:
+`no matches for kind "ClusterIssuer"` / `"PrometheusRule"`. The standing rule: **any component shipping
+CRs of a CRD it installs must split the CRs into `platform-config`** (`dependsOn: platform`), and
+second-order CRs whose CRD only appears after a controller reconciles go in `platform-config-cdi`.
+Remediation is a git change moving the resource; `bootstrap/standup.sh crds` is the belt-and-braces
+repair for a cluster already wedged.
+
+### 2. A HelmRelease is genuinely failing its install/upgrade
+Read the message. Known instances: the `headlamp-0.40.1` `-session-ttl` chart/image desync; Grafana 13
+OOM-looping after kps 65→88. Note `platform`'s `timeout: 5m` versus Longhorn's HelmRelease
+`timeout: 15m` — **the tier can time out while a legitimate Longhorn upgrade is still running**. Give
+it another interval before concluding it is broken.
+
+### 3. A `${TOKEN}` problem in post-build substitution
+Substitution **fails open**: an undefined variable becomes the empty string and the Kustomization still
+reports Ready — which is how `k8sServiceHost: ""` broke Cilium on 2 of 3 dev nodes on 2026-09-01. A
+*parse* error is the loud version and does fail the whole Kustomization
+(`unable to parse variable name`). If the message mentions envsubst or a variable, run the L0 check:
+`conformance/tests/l0-static/flux-postbuild-substitution/check.sh`. Note **strict substitution became
+default-on in kustomize-controller v1.9.0**, so a Flux upgrade flips this behaviour whether or not you
+opt in.
+
+### 4. A remote kustomize base timed out
+The kustomize-controller's git fetch has a 27s timeout and remote GitHub bases flake against it.
+`external-snapshotter` was vendored into the repo for this reason; **`platform/kubevirt` still uses
+remote bases**, so a `BuildFail` there is probably this. Remediation: vendor it the same way.
+
+### 5. SOPS decryption
+A tier that carries an encrypted Secret needs its own `decryption` stanza patched in by the overlay, by
+name. A newly split-out tier that inherited a secret but not the patch fails here.
+
+## Remediation notes
+
+- Everything durable is a **git change**. `flux reconcile` only re-triggers; it does not fix a bad
+  manifest, and it is blocked for agents here.
+- Flux is **level-triggered**: an interrupted reconcile is retried, so a restart of a controller is
+  safe and often the fastest way past a wedged in-flight apply (operator action).
+- `dependsOn` is **revision-gated**, not merely Ready-gated: a dependent cannot reconcile revision N
+  until its dependency has applied N. That is what makes the chain safe and also what makes one stuck
+  tier so expensive.
+
+## When to escalate or stop
+
+- **If `flux-system`, `cilium`, `platform` or `platform-config` is the failing tier, treat it as a
+  platform outage in progress** — everything downstream is frozen, including the fix. Get the operator.
+- **Do not `flux suspend` a tier to make the alert stop.** That trades a visible failure for
+  `/runbooks/flux-suspended` and silent drift, which is how prod's whole `platform` tier ended up
+  suspended and left that way after the 2026-09-04 external-dns incident.
+- **Do not enable `driftDetection.mode: enabled` on a CNI.** During the 2026-09-01 Cilium incident the
+  Helm storage manifest *was* the bad render, so drift correction would have actively re-asserted the
+  broken value.
+- If the GitRepository itself is not Ready, check Gitea before Flux — and remember prod deliberately
+  reads from the in-cluster Service so that public DNS can never be in this path.
+
+---
+Source of truth: `runbooks/flux-reconcile.md`. Index: `/runbooks/`
+Records: `DECISIONS-LOG.md` "2026-09-01 — INCIDENT: merging PR #4 briefly broke Cilium" (substitution
+fails open), the Headlamp gating stall, the oauth2-proxy cold-start deadlock, and the 2026-09-13
+GitOps source deadlock.

runbooks/flux-suspended.md

@@ -0,0 +1,134 @@
+# flux-suspended — A Flux resource has been suspended for an hour
+
+**Alert:** `FluxResourceSuspended` (warning, 1h)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.flux`
+**Expression:** `max by (namespace, name, kind) (gotk_suspend_status) == 1`
+
+## What this means here
+
+Someone turned reconciliation off for a resource and it is still off an hour later. While suspended,
+Flux neither applies new commits nor corrects drift — **git stops being the truth** for that subtree,
+silently, and everything downstream of a suspended gating tier is frozen with it.
+
+Suspension is a legitimate emergency valve here and is used deliberately: `cilium` was split into its
+own Kustomization partly so that `flux suspend kustomization cilium` becomes an option that does not
+freeze the other eleven platform components. The problem is never the suspend; it is the **forgotten**
+suspend. Two real cases:
+
+- After the 2026-09-04 external-dns incident, prod's whole `platform` tier and everything downstream
+  were suspended **and left suspended** (`infra/PROD-PROVISIONING.md` §10). That suspension is also
+  the only reason a second bug never fired: `platform-config` would otherwise have requested a
+  `*.home.keithjackson.dev` wildcard from letsencrypt-prod for a certificate dev already held, and the
+  two clusters would have raced each other's `_acme-challenge` TXT records on one zone — neither would
+  reliably solve DNS-01.
+- **Velero was found suspended live, with the suspension never recorded in git**, and resumed with
+  `flux resume`. A backup controller that is not reconciling is the definition of a silent DR gap.
+
+The 1h `for` is the point: short suspensions during maintenance are expected and should not notify.
+**This alert is currently INERT.** `count(gotk_reconcile_condition)` and `count(gotk_suspend_status)`
+both returned **0 series** on prod on 2026-09-13: there is no scrape job for the Flux controllers in
+Prometheus's 19 jobs, so every `gotk_*`-based rule (`FluxReconciliationFailure`,
+`FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`,
+`UpdateHealthGateFailed`) evaluates against nothing. **Flux health is therefore unmonitored today** and
+must be read from the CRs directly, with the commands below. See `/runbooks/targets-down` for the full
+list of inert rule groups. Adding the Flux controllers' ServiceMonitor is open work.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+What is suspended, across every Flux kind:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'for k in kustomizations.kustomize.toolkit.fluxcd.io helmreleases.helm.toolkit.fluxcd.io \
+            gitrepositories.source.toolkit.fluxcd.io helmrepositories.source.toolkit.fluxcd.io \
+            imagerepositories.image.toolkit.fluxcd.io imagepolicies.image.toolkit.fluxcd.io \
+            imageupdateautomations.image.toolkit.fluxcd.io; do \
+     echo "== $k"; kubectl --context k3s-prod get "$k" -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SUSPENDED:.spec.suspend 2>/dev/null | grep -v "<none>"; done'
+```
+
+Is the suspension declared in git, or is it live-only drift? (This is the important question.)
+
+```
+grep -rn "suspend" clusters/ platform/ | grep -v "^Binary"
+```
+
+If it is not in git, it was applied by hand and nothing will restore it after a controller restart —
+which means the resource may un-suspend itself at an unpredictable moment, or stay off forever.
+
+How far behind is it?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \
+     -o custom-columns=NAME:.metadata.name,SUSPENDED:.spec.suspend,APPLIED:.status.lastAppliedRevision; echo; \
+   kubectl --context k3s-prod -n flux-system get gitrepositories.source.toolkit.fluxcd.io \
+     -o jsonpath="{.items[0].status.artifact.revision}{\"\n\"}"'
+```
+
+## Likely causes, most likely first
+
+### 1. A maintenance suspension nobody resumed
+By far the most common, and the Velero case above is the archetype. Remediation (operator):
+`flux resume kustomization <name>` — but **only after** reading the next paragraph.
+
+### 2. It is suspended for a reason that still applies
+Before resuming, find out what would be applied. A tier suspended during an incident may be holding
+back a change that caused the incident. The 2026-09-04 case is the cautionary one: resuming
+`platform-config` while a second cluster still held the same wildcard certificate would have started an
+ACME DNS-01 race on a shared Cloudflare zone. **Diff first**: compare
+`status.lastAppliedRevision` against the GitRepository's current revision and read the commits in
+between.
+
+### 3. A crashlooping leaf was suspended instead of being moved
+The correct fix for "a leaf UI keeps failing and stalls the tier" is a `wait: false` tier of its own —
+that is what happened to `oauth2-proxy`, `headlamp`, `homepage`, `crafty`, `velero` and `runbooks`.
+Suspension as a permanent substitute leaves drift correction off for that component forever.
+
+### 4. Image automation
+`ImageUpdateAutomation`/`ImagePolicy`/`ImageRepository` are **authored and deliberately disabled** here
+(pending a Gitea write credential and `flux install --components-extra`), so if those show up, this is
+expected state, not an incident. See `/runbooks/image-automation`.
+
+## When to escalate or stop
+
+- **Do not resume a gating tier (`cilium`, `platform`, `platform-config`) without the operator.**
+  Resuming applies every commit since the suspension in one pass, to the tier everything depends on.
+- **Do resume `velero` promptly** if it is the suspended resource — a suspended backup controller is a
+  DR gap, and it has happened before.
+- If the suspension is not in git, record it in git (or resume it). A live-only suspension is
+  undocumented state, which is the failure class this repo keeps finding.
+- If you cannot determine *why* something was suspended, stop and ask. That is a safer outcome than
+  either resuming blind or leaving it off.
+
+---
+Source of truth: `runbooks/flux-suspended.md`. Index: `/runbooks/`
+Records: `infra/PROD-PROVISIONING.md` §10 (the 2026-09-04 suspension and the queued ACME race);
+`DECISIONS-LOG.md` (Velero found suspended with the suspension never in git).

runbooks/image-automation.md

@@ -0,0 +1,129 @@
+# image-automation — Flux's image-update-automation cannot write back to git
+
+**Alert:** `ImageUpdateAutomationFailing` (warning, 30m)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.automated-updates`
+**Expression:** `max by (namespace, name) (gotk_reconcile_condition{kind="ImageUpdateAutomation", type="Ready", status="False"}) == 1`
+
+## What this means here
+
+`ImageUpdateAutomation` is the write-back half of hands-off updates (ADR-0009): it commits new image
+tags into git so Flux then deploys them. When it is Ready=False, **stable image advances silently
+stop** — nothing breaks, nothing degrades, the platform just quietly stops receiving updates. That is
+the R-UPD-02 failure mode this alert exists for.
+
+**Read this first: on this cluster, image automation is authored and DELIBERATELY DISABLED.** It has
+been that way since 2026-08-24, pending (a) a Gitea write credential for the automation to push with
+and (b) `flux install --components-extra=image-reflector-controller,image-automation-controller`. The
+line is still commented out in `platform/kustomization.yaml`. So:
+
+- If this alert fires, the most likely explanation is that **the feature was partially enabled** —
+  someone uncommented the component or a rendered app started emitting `ImageRepository`/`ImagePolicy`
+  CRs whose controllers are not installed.
+- Rendered apps emit those CRs by default; `catalog/render.py --no-image-automation` is the documented
+  workaround for exactly this.
+- There is consequently **no real incident history** for this alert. Everything below is mechanism, not
+  post-mortem, and it says so honestly rather than inventing precedent.
+**This alert is currently INERT.** `count(gotk_reconcile_condition)` returned **0 series** on prod on
+2026-09-13 — there is no Flux-controller scrape job among Prometheus's 19 jobs, so every `gotk_*` rule
+evaluates against nothing. Read the CRs directly with the commands below, and see
+`/runbooks/targets-down` for the full list of inert rule groups.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+Are the controllers even installed?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n flux-system get deploy | sort; echo; \
+   kubectl --context k3s-prod api-resources --api-group=image.toolkit.fluxcd.io'
+```
+
+The automation objects and their messages:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get imageupdateautomations.image.toolkit.fluxcd.io -A \
+     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,LASTRUN:.status.lastAutomationRunTime'
+```
+
+The controller log — the git error is always in here:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n flux-system logs deploy/image-automation-controller --tail=100 2>&1 | tail -40'
+```
+
+Is the repo configured with a write path at all?
+
+```
+grep -rn "image-automation\|ImageUpdateAutomation\|components-extra" platform/kustomization.yaml bootstrap/ clusters/ 2>/dev/null | head -20
+```
+
+## Likely causes, most likely first
+
+### 1. The controllers are not installed (expected state here)
+`api-resources --api-group=image.toolkit.fluxcd.io` returns nothing, or the deployments are absent.
+Then the CRs are inert objects and the alert reflects a half-enabled feature. Remediation: either
+finish enabling it (an operator runs `flux install --components-extra=...` — note that Flux's own
+controllers were installed **imperatively** and carry no Flux ownership labels, so **nothing under
+`clusters/**` can install a controller or change a controller flag**), or stop rendering the CRs with
+`catalog/render.py --no-image-automation`.
+
+### 2. No git write credential
+The automation must push. Prod's source is `http://gitea.gitea.svc.cluster.local:3000`, and the
+read-only source credential is not a write credential. The controller log will say authentication
+failed or permission denied. Remediation: a Gitea token in a SOPS-encrypted Secret referenced by the
+`GitRepository`/automation, added in git — and note the standing rule from the OIDC incident:
+**non-secret environment-specific values must not live inside an encrypted Secret**, because
+`${BASE_DOMAIN}` substitution can never reach inside one.
+
+### 3. It cannot push because the branch moved
+Automation commits to a branch; a force-push or a protected branch rejects it. The log names the ref.
+
+### 4. The commit is being made but the Kustomization is not picking it up
+Then this is not the right runbook — go to `/runbooks/flux-reconcile`.
+
+## When to escalate or stop
+
+- **This is a warning with a 30m `for` for a reason: nothing is broken.** Do not treat it as an
+  outage. The consequence is "no new image versions", which is measured in days.
+- **Do not create a Gitea write credential ad hoc.** It is a new secret in the DR path and belongs to
+  the operator, SOPS-encrypted, in git.
+- **Do not enable the extra Flux controllers to make the alert stop.** Enabling automation on a
+  cluster that has never run it will start proposing image bumps across every workload at once; that is
+  a planned change, with `/runbooks/update-rollback` as its companion.
+- If the alert is firing because the feature is half-enabled, the honest fix is to finish or to revert
+  — both are git changes and both want the operator's decision.
+
+---
+Source of truth: `runbooks/image-automation.md`. Index: `/runbooks/`
+Records: `DECISIONS-LOG.md` (image automation authored-and-disabled since 2026-08-24; the Flux
+controllers are outside Flux's own inventory).

runbooks/image-policy-stalled.md

@@ -0,0 +1,137 @@
+# image-policy-stalled — An ImagePolicy or ImageRepository cannot resolve its tag
+
+**Alert:** `ImagePolicyStalled` (warning, 1h)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.automated-updates`
+**Expression:** `max by (namespace, name, kind) (gotk_reconcile_condition{kind=~"ImagePolicy|ImageRepository", type="Ready", status="False"}) == 1`
+
+## What this means here
+
+An `ImageRepository` cannot scan a registry, or an `ImagePolicy` cannot pick a tag out of what was
+scanned. The effect is narrower than `/runbooks/image-automation`: automation itself is healthy, but
+**this one service stops receiving updates** (R-UPD-01), while everything else continues. It is
+therefore easy to miss, which is what the 1h `for` and the per-object labels are for.
+
+The same framing applies as for image-automation: **this feature is authored and deliberately disabled
+on this cluster** and has been since 2026-08-24 (no Gitea write credential, and the
+`image-reflector-controller` / `image-automation-controller` components are not installed — the line is
+still commented out in `platform/kustomization.yaml`). Rendered apps emit `ImageRepository` and
+`ImagePolicy` CRs anyway, so the objects can exist with no controller to reconcile them;
+`catalog/render.py --no-image-automation` is the documented workaround. **There is no incident history
+for this alert** — the section below is mechanism, honestly labelled.
+
+One live constraint worth knowing while reading registry errors: the images in this platform come from
+`docker.io`, `ghcr.io`, `lscr.io`, `quay.io`, `registry.k8s.io` and `registry.gitlab.com`. Docker
+Hub's anonymous rate limit is the most likely real-world cause of a scan failure once automation is
+switched on.
+**This alert is currently INERT.** `count(gotk_reconcile_condition)` returned **0 series** on prod on
+2026-09-13 — there is no Flux-controller scrape job among Prometheus's 19 jobs, so every `gotk_*` rule
+evaluates against nothing. Read the CRs directly with the commands below, and see
+`/runbooks/targets-down` for the full list of inert rule groups.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+Which object, and the message (the message is nearly always the whole answer):
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get imagerepositories.image.toolkit.fluxcd.io -A \
+     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,IMAGE:.spec.image,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,TAGS:.status.lastScanResult.tagCount; echo; \
+   kubectl --context k3s-prod get imagepolicies.image.toolkit.fluxcd.io -A \
+     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,LATEST:.status.latestImage,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message'
+```
+
+Are the controllers installed at all?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n flux-system get deploy | grep -i image || echo "image controllers NOT installed (expected on this cluster)"'
+```
+
+The reflector's log:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n flux-system logs deploy/image-reflector-controller --tail=80 2>&1 | tail -30'
+```
+
+Can the cluster reach the registry at all? (Default-deny is the baseline; a new namespace may simply
+not be allowed out.)
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get pods -A --field-selector status.phase!=Running | head; echo; \
+   kubectl --context k3s-prod get events -A --field-selector reason=Failed --sort-by=.lastTimestamp | grep -i "pull\|registry" | tail -10'
+```
+
+## Likely causes, most likely first
+
+### 1. The image controllers are not installed (expected state here)
+The CRs exist because `catalog/render.py` emits them; nothing reconciles them. Remediation: render with
+`--no-image-automation`, or have the operator finish enabling automation. Note that the Flux
+controllers were installed **imperatively** and are outside Flux's own inventory, so no git change can
+install them.
+
+### 2. Registry rate limit
+Docker Hub anonymous pulls/scans are limited per IP, and all three nodes share one WAN address. The
+message says `toomanyrequests`. Remediation: a registry credential (SOPS-encrypted, in git), or scan
+less often via `spec.interval`.
+
+### 3. Registry auth
+A private repository with no `secretRef`, or an expired token. The message names it.
+
+### 4. The policy's range matches nothing
+`ImagePolicy` semver ranges come from each ServiceDefinition's `update.range` (for example
+`">=2024.0.0"` for pihole, `">=24.0.0"` for keycloak). A range that no published tag satisfies — or a
+repository that publishes only `latest` (this platform has `lscr.io/linuxserver/jackett:latest` and
+`plexinc/pms-docker:latest`) — leaves the policy with nothing to select. Remediation: fix the range in
+`catalog/<service>.yaml` and re-render; a `latest`-only image cannot participate in automation at all
+and should not carry a policy.
+
+### 5. Egress is blocked for that namespace
+Default-deny everywhere (R-NET-01) means outbound registry access is an explicit allow. Cross-read
+`/runbooks/denied-flows`, and in particular the post-Cilium-upgrade finding that `toEntities: world` no
+longer covers a call whose destination resolves to an in-cluster backend.
+
+## When to escalate or stop
+
+- **Warning, 1h, single service: this does not page.** The consequence is one workload pinned at its
+  current version.
+- **Do not widen an `update.range` to make the alert clear.** The range is the service's update policy
+  (R-UPD-01); loosening it to satisfy a scanner is how an unintended major version arrives.
+- **Do not add registry credentials ad hoc** — new secret, DR path, operator's call.
+- If the same object has been stalled for days, the honest outcome is to remove its automation CRs
+  (`--no-image-automation`) rather than leave a permanently-failing object training everyone to ignore
+  the alert. Alert fatigue is a recorded, expensive failure mode on this cluster.
+
+---
+Source of truth: `runbooks/image-policy-stalled.md`. Index: `/runbooks/`
+Records: `DECISIONS-LOG.md` (image automation authored-and-disabled; `--no-image-automation`);
+`catalog/*.yaml` (`update.channel` / `update.range` per service).

runbooks/keycloak-client-secrets.md

@@ -0,0 +1,201 @@
+# keycloak-client-secrets — The OIDC client reconciler is failing or has gone silent
+
+**Alerts:** `KeycloakClientSecretReconcilerFailing` (warning, 15m) and
+`KeycloakClientSecretReconcilerStale` (warning, 2h)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.identity`
+**Expressions:**
+```
+max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"}) > 0
+absent(kube_job_status_succeeded{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"} == 1)
+```
+
+## What this means here
+
+These two rules watch the CronJob (every 15 minutes) that repairs OIDC client **secrets** and
+environment-derived client **URLs** in the Keycloak realm. They exist because of the single worst
+green-looking outage this cluster has had, and they are best understood through it.
+
+### The 2026-09-06 lockout — why "everything is green" means nothing here
+
+The Keycloak Operator resolves each OIDC client's secret from an `(env:NAME)` placeholder in the realm
+import. **When it cannot resolve one, it stores the LITERAL placeholder text as the client secret and the
+import Job still reports SUCCESS.** On prod, **all 12 clients** ended up holding the literal placeholder
+and **every OIDC login on the cluster failed** with *"Invalid client or Invalid client credentials"*.
+Meanwhile: the import Job succeeded, the CR rested at `Done=True / Started=False`,
+`KeycloakDown` and `KeycloakDatabaseDown` were quiet, and **the conformance test that asserts
+`Done=True` passed throughout the outage.** The only observable was a human failing to log in. Dev had
+exactly one broken client (oauth2-proxy — its key was added *after* dev's import completed, and
+**a completed import is never re-run**).
+
+The realm import is a **ONE-SHOT that fails open**: the operator logs *"Job already executed - not
+recreating"*, so a git change to `keycloak-realm-import.yaml` does **not** re-import.
+
+### The second half: redirect URIs, same one-shot, one field over
+
+The realm import carries **26 `${BASE_DOMAIN}` references** (39 files under `platform/` and `apps/`
+reference the knob). Substitution happened **once**, at the old hostname, and then froze. After the apex
+flip, *"every OIDC login fails with a `redirect_uri` mismatch while the CR still reads `Done=True` and
+nothing alerts."* The reconciler was extended with a second pass over `redirectUris`, `webOrigins`,
+`rootUrl`, `baseUrl` and `adminUrl`, with desired state **generated** by
+`ci/render-keycloak-client-urls.py` (gated by an L0 `--check`).
+
+Semantics worth knowing before you read a Job log:
+- URLs are copied **verbatim, including `${BASE_DOMAIN}`**, and rendered by the *same* Flux
+  Kustomization as the import.
+- **List fields compare order-insensitively (`sorted`, not `set`)** — measured, not defensive: Keycloak
+  genuinely reorders lists, and an order-sensitive compare would have rewritten `kubernetes` (8 URIs) and
+  `oauth2-proxy` (2) **every 15 minutes forever while reporting success**.
+- A field the desired state omits is **UNMANAGED, never forced empty**.
+- **Public clients' URLs are managed even though their secrets are not** (`kubernetes` is public, with
+  PKCE, so no kubeconfig holds a secret).
+- The two passes **fail closed independently** — demonstrated live: pass 1 failed on an empty secrets
+  directory while pass 2 completed.
+
+### Why there are exactly two rules, and what each catches
+
+`...Failing` sees a run that *ran and failed* — unrepairable drift, or the admin API unreachable. It
+fires alongside `KeycloakDown` during an IdP outage, deliberately: the reconciler genuinely is not
+running, so treat `KeycloakDown` as the cause when both are lit. Self-clears when failed Jobs age out
+(TTL 1h).
+
+`...Stale` sees the **healer being quietly absent** — the CronJob suspended, never applied by Flux, or
+unable to schedule a pod at all. That is the "looks applied and isn't" failure this whole change exists
+to end. It is also the rule that would catch the healer missing during a domain migration.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+Does the CronJob exist, is it suspended, and when did it last succeed?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n keycloak get cronjobs.batch \
+     -o custom-columns=NAME:.metadata.name,SCHEDULE:.spec.schedule,SUSPEND:.spec.suspend,LAST:.status.lastScheduleTime,LASTSUCCESS:.status.lastSuccessfulTime'
+```
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n keycloak get jobs.batch --sort-by=.metadata.creationTimestamp \
+     -o custom-columns=NAME:.metadata.name,SUCCEEDED:.status.succeeded,FAILED:.status.failed,START:.status.startTime | tail -15'
+```
+
+**Read the Job log — it names the pass, the client, and for a URL the exact before/after, and it never
+prints a secret** (that is a design property, asserted by the L0 check):
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n keycloak logs job/$(kubectl --context k3s-prod -n keycloak get jobs.batch \
+     --sort-by=.metadata.creationTimestamp -o jsonpath="{.items[-1:].metadata.name}") --tail=200'
+```
+
+Is the IdP itself up? (If not, that is the cause; come back here after.)
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n keycloak get pods; echo; \
+   kubectl --context k3s-prod get clusters.postgresql.cnpg.io -n keycloak'
+```
+
+Confirm the declaration is self-consistent **offline**, before touching the cluster — this is the same
+check the merge gate runs unconditionally (Stage 3d):
+
+```
+bash conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh
+python3 ci/render-keycloak-client-urls.py --check
+```
+
+Confirm what a user actually experiences:
+
+```
+curl -s -o /dev/null -w '%{http_code}\n' --resolve keycloak.home.keithjackson.dev:443:192.168.2.20 \
+  https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
+```
+
+```
+infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod auth whoami'
+```
+
+(Expected on a working OIDC path: `oidc:<user>` with `[oidc:platform-admins]`. From this workstation the
+OIDC context hangs on a browser prompt — that is not evidence of breakage.)
+
+## Likely causes, most likely first
+
+### 1. Keycloak or its database is down, so the reconciler cannot reach the admin API
+`...Failing` fires alongside `KeycloakDown` by design. Fix the IdP first: `/runbooks/keycloak-down`,
+`/runbooks/keycloak-db`. Nothing here is actionable until the admin API answers.
+
+### 2. Unresolvable drift on one client
+The log names the client and the pass. Most often a secret key missing from the SOPS Secret, or a client
+in the realm that git does not describe. The contract: keys are **`<clientId>-client-secret`**, and the
+public `kubernetes` client declares **no** secret. Remediation is a git change to the declaration, then
+let the CronJob heal it — the reconciler is compare-then-write and idempotent.
+
+### 3. A hostname migration left the realm behind
+The `redirect_uri` mismatch case. Regenerate desired state with `ci/render-keycloak-client-urls.py` and
+push; the reconciler's second pass applies it. Historical scale: re-rendering at the post-flip apex
+produced **24 drifted fields across 12 clients**, with `kubernetes` correctly `ok` because its localhost
+callbacks are not domain-derived.
+
+### 4. The CronJob is suspended or was never applied (`...Stale`)
+Check `spec.suspend` and the Flux tier. The ordering requirement is a **real Flux `dependsOn` edge with
+`wait: true`**, not a comment — the L0 check asserts that, because a comment is not an ordering guarantee.
+
+### 5. The pod cannot read its own secrets
+A specific, recorded trap: the reconciler needs **`fsGroup: 65532`** with the client-secrets volume at
+**`defaultMode: 0440`**. With `0400` and no fsGroup the container dies with `PermissionError` on **every**
+run. If the log shows that, this is it.
+
+### 6. Everything is green and logins still fail
+Then trust the human, not the dashboard — that is the entire lesson of 2026-09-06. Audit the live realm
+directly with `kcadm.sh` inside `keycloak-0`.
+
+## When to escalate or stop
+
+- **Do not "fix" the realm by re-applying the import.** It is a one-shot: the operator logs *"Job already
+  executed - not recreating"* and nothing happens. The documented path is `kcadm.sh` inside `keycloak-0`
+  (admin credentials in Secret `keycloak-admin-credentials`; `kcadm` is present, `curl` and `python3` are
+  **not**) **and** fixing git so the change is reproducible. Doing only the live half creates drift; doing
+  only the git half changes nothing.
+- **Never print or paste a client secret.** The reconciler deliberately logs URL values in full (they are
+  public hostnames) and never logs secrets. Keep that property.
+- **Do not widen the reconciler's endpoint allow-list.** It is hard-scoped so that `/users`, `/groups`,
+  `/roles` and `/authentication` are unreachable — a blast-radius bound, not an oversight.
+- **Do not disable or suspend the CronJob to stop the alert.** That is the healer, and `...Stale` exists
+  precisely because a suspended healer is invisible.
+- Escalate for: rotating client secrets, editing the live realm, or anything touching MFA (a
+  conditional-OTP misconfiguration once 403'd every platform-admin login with
+  `"Invalid role name: null"`, fixed to `condUserRole: platform-admin`).
+
+---
+Source of truth: `runbooks/keycloak-client-secrets.md`. Index: `/runbooks/`
+Records: `ci/l0.sh` Stage 3d and
+`conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh` (the 2026-09-06 incident and the
+reconciler's contract — 212 L0 assertions, an 84-assertion selftest, 12/12 injected regressions killed);
+`DECISIONS-LOG.md` "Keycloak client REDIRECT URIs reconcile too — the same one-shot, one field over".

runbooks/keycloak-db.md

@@ -0,0 +1,169 @@
+# keycloak-db — Keycloak's PostgreSQL is unavailable
+
+**Alert:** `KeycloakDatabaseDown` (critical, 5m)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.identity`
+**Expression:**
+```
+max(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-db-.*", condition="true"}) == 0
+or absent(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-db-.*"})
+```
+
+## What this means here
+
+No ready `keycloak-db-*` pod for five minutes. `keycloak-db` is a **CloudNativePG Cluster** with
+**3 instances** on Longhorn (3 replicas each), and it is the database the IdP cannot live without. The
+`pod=~"keycloak-db-.*"` scoping matters for the same reason as in `/runbooks/keycloak-down`: the
+namespace also holds the Keycloak server and operator pods, and an unscoped `max()` would be masked by
+those.
+
+**Blast radius is the whole platform's identity:** Keycloak fails → kubectl OIDC, Grafana, Headlamp,
+Gitea and forward-auth all fail. The certificate-based kubeconfig is an independent break-glass path and
+still works. This page is deliberately served **without** forward-auth so it stays readable during
+exactly this outage.
+
+## The HA properties you are relying on, and why they are set the way they are
+
+- **`instances: 3`.** All three CNPG Clusters used to run `instances: 1`, and all three primaries had
+  drifted onto one node — **on 2026-09-07 that node took SSO, Nextcloud and Semaphore down together.**
+  Scale-out to 3 is what gives you a standby to fail over to.
+- **`podAntiAffinityType: required`**, not CNPG's default `preferred`. `preferred` is a scheduler
+  *hint* and on the path that matters — recreating an instance after node loss — it co-locates.
+  Recorded conclusion: *"`preferred` is how a 3-instance cluster decays back into the single-node
+  failure domain."* The accepted cost is that an instance stays **Pending** until a node returns.
+- **`primaryUpdateMethod: switchover`.** The CRD default is `restart`, which is a delete-and-recreate of
+  the **primary** — tens of seconds with no Postgres, i.e. an SSO outage. Constraint to remember: under
+  `switchover` the operator **rejects an update that changes `imageName` and a postgresql parameter in
+  the same reconcile**.
+- **Synchronous replication `method: any`, `number: 1`, `dataDurability: preferred` — never `required`.**
+  `required` is a strict quorum: with fewer than `number` healthy standbys, **writes BLOCK**. On this
+  database one degraded standby (a node reboot, a Longhorn rebuild, a Cilium restart) would become a
+  total write outage. `dataDurability` is only honoured while `standbyNamesPre/Post` are unset.
+
+**And the thing that is easy to get wrong:** *"Standbys replicate corruption and deletions faithfully;
+only the Barman archive is a backup."* Three ready replicas are not a recovery plan.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+The Cluster's own verdict first — fully-qualified, because `cluster` is an ambiguous short name:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
+     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,INSTANCES:.status.instances,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,PHASE:.status.phase'
+```
+
+Where the instances are, and whether they are all on one node (the 2026-09-07 defect):
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n keycloak get pods -l cnpg.io/cluster=keycloak-db -o wide'
+```
+
+Why a pod is not ready:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n keycloak describe pod keycloak-db-2 | sed -n "/Events:/,\$p"; echo ====; \
+   kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=80 2>&1 | tail -40'
+```
+
+The conditions that matter for recoverability — check these **before** any recovery action:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -n keycloak keycloak-db \
+     -o custom-columns=ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,LASTBACKUP:.status.conditions[?\(@.type==\"LastBackupSucceeded\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint'
+```
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get backups.postgresql.cnpg.io -n keycloak'
+```
+
+`kubectl get backup -A` prints "No resources found" here even with 27 CNPG backups present — three CRDs
+register the kind `Backup` and the short name resolves to Longhorn's, which has none. Never decide
+anything from the short name.
+
+Storage underneath:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n keycloak get pvc; \
+   kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy'
+```
+
+## Likely causes, most likely first
+
+### 1. A node is down or was rebooted
+The instances live on Longhorn with hard anti-affinity, so a node loss takes one instance. With
+`instances: 3` the cluster should survive it and promote; with `podAntiAffinityType: required` the lost
+instance stays **Pending** until a node returns, which is expected and accepted. Go to
+`/runbooks/node-down`.
+
+### 2. The data PVC filled up
+The most dangerous cause, because the mechanism is a *backup* fault presenting as a *storage* failure.
+With `archive_mode=on` and a failing `archive_command`, Postgres **refuses to recycle** unarchived WAL;
+`pg_wal` shares the data PVC on every Cluster here (no `walStorage:` configured) and grows until the
+volume is full and the postmaster **PANICs**. Measured on this very database: 17 unarchived segments /
+~268 MiB on a 5 GiB PVC, growing ~6.8 MiB/h — about 29 days from 93% free to dead. Go to
+`/runbooks/cnpg-archiving` and `/runbooks/pvc-capacity`. **Never delete WAL by hand.**
+
+### 3. A Longhorn volume is degraded or faulted
+`/runbooks/longhorn-degraded`, `/runbooks/longhorn-faulted`.
+
+### 4. An upgrade is stuck mid-switchover
+Signature: the operator refusing an update that changes `imageName` and a postgresql parameter together.
+Split the change in git. See `/runbooks/update-rollback`.
+
+### 5. The database is up and Keycloak still fails
+Then read `/runbooks/keycloak-down` and, more likely, `/runbooks/keycloak-client-secrets` — the
+2026-09-06 lockout had all 12 OIDC clients holding literal placeholder secrets behind an entirely green
+cluster.
+
+## When to escalate or stop
+
+- **Escalate immediately.** This is SSO's foundation; a 5m `for` on a `critical` is already generous.
+- **Never change `dataDurability` to `required`** while trying to "make replication safer" — it converts
+  one degraded standby into a total write outage on the SSO database.
+- **Do not delete a CNPG pod or PVC to force a rebuild.** The operator manages instance lifecycle;
+  deleting a PVC destroys that instance's data and, if it is the primary, may lose transactions that the
+  archive does not have.
+- **Do not attempt a restore before checking `firstRecoverabilityPoint` and `ContinuousArchiving`.** Prod
+  ran 38+ hours from bootstrap with `ContinuousArchiving=False` and `firstRecoverabilityPoint: null` on
+  all three Clusters and nothing said so. Know what you actually have before you destroy what you have.
+- **Standbys are not backups.** Recovery is from the Barman archive in R2 (`CNPG_BACKUP_PATH =
+  s3://k3s-prod-backup/barman`, 30d retention), which is an operator operation.
+
+---
+Source of truth: `runbooks/keycloak-db.md`. Index: `/runbooks/`
+Records: commits `f338e91` (instances 3), `e8d8601` (required anti-affinity + switchover), `cac413e`
+(sync replication, `dataDurability: preferred`); `platform/cnpg/alerts.yaml` header (the 2026-09-06
+archiving incident and the measured WAL fill rate).

runbooks/keycloak-down.md

@@ -0,0 +1,166 @@
+# keycloak-down — The Keycloak IdP is down (SSO is unavailable)
+
+**Alert:** `KeycloakDown` (critical, 5m)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.identity`
+**Expression:**
+```
+max(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+", condition="true"}) == 0
+or absent(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+"})
+```
+
+## What this means here
+
+No ready Keycloak **server** pod for five minutes. The regex is deliberately scoped to
+`keycloak-[0-9]+` (the StatefulSet pods `keycloak-0`, `keycloak-1`) because the `keycloak` namespace also
+holds the operator (`keycloak-operator-*`) and the database (`keycloak-db-*`); an unscoped `max()` would
+be satisfied by those being ready and would never fire for the IdP itself. That scoping was a fix — the
+original matched `.*postgres.*`.
+
+## Blast radius — read this before triaging anything else
+
+Keycloak is the root of trust. With SSO down, all of these are down or degraded **at once**:
+
+- **kubectl OIDC login** (the public `kubernetes` client with PKCE, groups → RBAC)
+- **Grafana** (OIDC), **Headlamp** (forwards the user's id_token to the apiserver), **Gitea**
+- **forward-auth** via oauth2-proxy, and the five per-app oauth2-proxy sidecars (pihole, jackett,
+  qbittorrent, radarr, sonarr) — these **fail closed**, 302ing to a Keycloak that is not there
+
+**You can still get in.** The certificate-based kubeconfig is untouched by the OIDC work and remains a
+fully independent break-glass path — that is a deliberate design property, not an accident. From this
+workstation the working non-interactive path is the dev-container kubeconfig; the host `k3s-oidc`
+context will hang on a browser prompt, which is not the cluster being down.
+
+**And you can still read this page.** The runbook route is deliberately **public, with no
+forward-auth**, precisely so that `keycloak-down` is readable by an operator who cannot log in. If that
+ever changes, this runbook becomes unreachable exactly when it is needed.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+Everything in the namespace, in one view — server, operator and database together:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n keycloak get pods -o wide; echo; \
+   kubectl --context k3s-prod -n keycloak get statefulsets,deployments'
+```
+
+Is the database the real cause? (It usually is. Fully-qualified: `clusters.postgresql.cnpg.io`.)
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -n keycloak \
+     -o custom-columns=NAME:.metadata.name,INSTANCES:.status.instances,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,PHASE:.status.phase'
+```
+
+The server's own complaint:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n keycloak logs keycloak-0 --tail=120 2>&1 | tail -50; echo ====; \
+   kubectl --context k3s-prod -n keycloak describe pod keycloak-0 | sed -n "/Events:/,\$p"'
+```
+
+The operator's view of the CR:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n keycloak get keycloaks.k8s.keycloak.org -o yaml | grep -A20 "status:"'
+```
+
+Is it reachable at all, from outside, with an explicit SNI (in-pod HTTPS tests to these hostnames are
+misleading here):
+
+```
+curl -s -o /dev/null -w '%{http_code}\n' --resolve keycloak.home.keithjackson.dev:443:192.168.2.20 \
+  https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
+```
+
+## Likely causes, most likely first
+
+### 1. The CNPG database is not available
+Keycloak cannot start without Postgres. This is the highest-probability cause and it has its own runbook:
+`/runbooks/keycloak-db`. Note that `keycloak-db` is a 3-instance CNPG Cluster with
+`primaryUpdateMethod: switchover` and synchronous replication at `dataDurability: preferred` —
+`preferred` specifically so that one degraded standby does **not** block writes, which on this database
+would be a total SSO write outage.
+
+### 2. A node that held the Keycloak pods went away
+Both Keycloak pods and all three `keycloak-db` instances previously ran on **one node** — all three CNPG
+primaries had drifted onto `k3s-prod-2` — and **on 2026-09-07 that node took SSO, Nextcloud and
+Semaphore down together**. The fixes were `instances: 3` on every Cluster and
+`podAntiAffinityType: required` (CNPG's default `preferred` is only a scheduler hint and co-locates on
+the path that matters — recreating an instance after node loss). Check `-o wide` above: if everything is
+on one node, that is a live re-occurrence of the same defect. Go to `/runbooks/node-down`.
+
+### 3. A HelmRelease / operator upgrade went bad
+Read `/runbooks/update-rollback`. Under `primaryUpdateMethod: switchover` the CNPG operator **rejects an
+update that changes `imageName` and a postgresql parameter in the same reconcile** — a stuck reconcile
+with that message is this.
+
+### 4. TLS to the issuer is broken
+Keycloak's clients validate the issuer's certificate. While the wildcard was LE-**staging**, in-cluster
+clients rejected it with `x509: certificate signed by unknown authority`. If `/runbooks/cert-issuance` or
+`/runbooks/cert-expiry` is also firing, fix that first.
+
+### 5. Logins fail but Keycloak is UP
+Then this is the **wrong alert and the wrong runbook**, and the distinction has cost this cluster dearly.
+On 2026-09-06 every OIDC login on prod failed — *"Invalid client or Invalid client credentials"* — with
+**all 12 clients holding the literal `(env:...)` placeholder as their secret**, while
+`KeycloakDown`/`KeycloakDatabaseDown` were quiet, the import Job reported success, the CR read
+`Done=True`, and the conformance test that asserts `Done=True` **passed throughout**. The only observable
+was a human failing to log in. Go to `/runbooks/keycloak-client-secrets`.
+Similarly, a `redirect_uri` mismatch after a hostname change is not an outage — same runbook.
+And once, MFA broke every platform-admin login on its own: conditional-OTP used a non-existent config
+key, giving `"Invalid role name: null"` → 403, fixed to `condUserRole: platform-admin`.
+
+## When to escalate or stop
+
+- **Escalate immediately.** SSO down is a platform-wide outage; treat it as such.
+- **Use the certificate-based kubeconfig** (the dev-container path) rather than trying to fix OIDC login
+  first. Do not burn the outage on getting `kubectl` to authenticate through the thing that is down.
+- **Do not edit the realm to "reset" anything.** The realm import is a **ONE-SHOT that fails open**: the
+  operator logs *"Job already executed - not recreating"*, so a git change to
+  `keycloak-realm-import.yaml` does **not** re-import. The documented path is `kcadm.sh` inside
+  `keycloak-0` (admin credentials in the Secret `keycloak-admin-credentials`; `kcadm` is present,
+  `curl`/`python3` are **not**) **and** fixing git for reproducibility. Doing only one of the two is how
+  drift is created.
+- **Do not delete the Keycloak CR or the StatefulSet.** Recreating the IdP is a restore operation, not a
+  restart.
+- **Do not touch `keycloak-db` volumes.** If the database is the cause, read `/runbooks/keycloak-db` and
+  `/runbooks/cnpg-archiving`; only the Barman archive is a backup, standbys replicate deletions
+  faithfully.
+- Break-glass local admins exist for gitea, nextcloud and semaphore. Nextcloud has **no native
+  group→admin mapping**, so platform-admins there are promoted by hand.
+
+---
+Source of truth: `runbooks/keycloak-down.md`. Index: `/runbooks/`
+Records: `DECISIONS-LOG.md` (the 2026-09-06 client-secret lockout; the 2026-09-07 single-node SSO
+outage); commits `f338e91`, `e8d8601`, `cac413e` (CNPG instances/anti-affinity/switchover/sync).

runbooks/kubevirt-control-plane.md

@@ -0,0 +1,156 @@
+# kubevirt-control-plane — The KubeVirt control plane is degraded
+
+**Alert:** `KubeVirtControlPlaneDown` (critical, 10m)
+**File:** `platform/kubevirt/alerts.yaml`, group `platform-health.kubevirt`
+**Expression:**
+```
+(sum(kubevirt_virt_controller_ready_status) == 0)
+or (sum(kubevirt_virt_api_ready_status) == 0)
+or (absent(up{job="kubevirt-prometheus-metrics"} == 1))
+```
+
+## What this means here
+
+No `virt-controller` or no `virt-api` replica reports ready, so **VM lifecycle operations — create, start,
+stop, migrate — will fail** (docs/06). Note carefully what this does *not* mean: **already-running VMs
+keep running.** `virt-launcher` pods and their qemu processes are independent of the control plane. So
+this is an "I cannot manage VMs" alert, not "the VMs are down".
+
+The `absent()` clause also fires if the whole KubeVirt metrics target disappears. The scrape job label is
+**`kubevirt-prometheus-metrics`** (the Service name), **not** `kubevirt` — that precision is a fix: this
+very alert once used a non-existent metric with the wrong job label and fired falsely (commit `b57b3e9`).
+There is a sibling precedent too: a broken HTTP CDI ServiceMonitor caused a false `CDIOperatorDown`
+(commit `6a7e9d1`). **So the first question with this alert is always whether it is telling the truth.**
+
+Live state, measured 2026-09-13: `kubevirt-prometheus-metrics` **is** among Prometheus's 19 scrape jobs
+and `kubevirt_virt_api_ready_status` returns **2 series**, so the first two clauses are evaluable and
+quiet. (`kubevirt_vmi_phase_count` was absent at the same moment — that metric only exists when VMIs do,
+which affects `/runbooks/kubevirt-launch-failed`, not this rule.)
+
+Platform context: **KubeVirt v1.9.0 + CDI v1.66.0**, on real nested KVM (`/dev/kvm` present, vmx/svm
+verified; the Proxmox nodes use `cpu_type=host`). The `kubevirt` and `cdi` namespaces deliberately carry
+**no `app-namespace` label**, so Kyverno does not generate the default-deny baseline into them — it would
+sever the virt control plane. Do not "fix" that by adding the label.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+Is the control plane actually unhealthy, or is this a metrics artefact?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n kubevirt get pods -o wide; echo; \
+   kubectl --context k3s-prod get kubevirts.kubevirt.io -A \
+     -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,READY:.status.conditions[?\(@.type==\"Available\"\)].status'
+```
+
+Are the VMs themselves still running? (This is the reassuring half.)
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get vmi -A -o wide; echo; \
+   kubectl --context k3s-prod get vm -A'
+```
+
+Is the scrape target up, and do the two metrics exist?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
+   for q in "up{job=\"kubevirt-prometheus-metrics\"}" "sum(kubevirt_virt_controller_ready_status)" "sum(kubevirt_virt_api_ready_status)"; do \
+     printf "%-55s " "$q"; curl -s -G --data-urlencode "query=$q" localhost:19090/api/v1/query \
+       | python3 -c "import json,sys;print([x[\"value\"][1] for x in json.load(sys.stdin)[\"data\"][\"result\"]] or \"ABSENT\")"; done'
+```
+
+Why a component is unready:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n kubevirt logs deploy/virt-controller --tail=80 2>&1 | tail -30; echo ====; \
+   kubectl --context k3s-prod -n kubevirt logs deploy/virt-api --tail=80 2>&1 | tail -30; echo ====; \
+   kubectl --context k3s-prod -n kubevirt get events --sort-by=.lastTimestamp | tail -20'
+```
+
+Is the Flux tier that owns KubeVirt healthy?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \
+     -o custom-columns=NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message'
+```
+
+## Likely causes, most likely first
+
+### 1. The metrics target, not the control plane
+Check the pods first. If `virt-api` and `virt-controller` are Running and Ready, this is a monitoring
+fault — and there is direct precedent for false positives on this exact alert (`b57b3e9`) and its CDI
+sibling (`6a7e9d1`). Remediation: fix the ServiceMonitor/metric in git. Cross-read
+`/runbooks/targets-down`.
+
+### 2. A node that hosted the control-plane pods went away
+`virt-controller` and `virt-api` are ordinary Deployments. Go to `/runbooks/node-down`.
+
+### 3. The KubeVirt Flux tier failed to reconcile
+Specific known hazard: **`platform/kubevirt` still uses remote kustomize bases**, and the
+kustomize-controller's git fetch has a **27s timeout** that remote GitHub bases flake against — a
+`BuildFail` there is probably this. `external-snapshotter` was vendored into the repo for exactly this
+reason; vendoring KubeVirt the same way is the durable fix. See `/runbooks/flux-reconcile`.
+
+### 4. Node pressure
+Heavy sustained node load — TCG-emulated VMs are the recorded example — can degrade the Cilium datapath
+into intermittent LB-VIP reachability problems, which looks like control-plane flapping. A restart
+cleared it. Cross-read `/runbooks/node-io` and `/runbooks/node-cpu`.
+
+### 5. A bad operator upgrade
+`/runbooks/update-rollback`.
+
+### 6. A NetworkPolicy severed the control plane
+Only if someone added `homelab.keithjackson.dev/app-namespace=true` to `kubevirt` or `cdi`. Those
+namespaces are deliberately unlabelled so Kyverno does not inject a default-deny baseline that would cut
+virt-handler off. Reverting that label is the fix.
+
+## When to escalate or stop
+
+- **Prioritise honestly.** VM *lifecycle* is broken; running VMs are not. The UniFi VM keeps serving.
+  Treat this as urgent-but-not-catastrophic unless a VM also needs to start.
+- **Do not drain a node to move virt pods.** This cluster does not drain, and a VM live-migrates on drain
+  **only** with per-VM `evictionStrategy: LiveMigrate` — a blanket setting would wedge drains for
+  RWO-disk VMs.
+- **Do not delete a VMI or `virt-launcher` pod** while the control plane is down: nothing will bring the
+  VM back until `virt-controller` is ready again.
+- **Do not add the `app-namespace` label to `kubevirt` or `cdi`** to "harden" them.
+- Escalate for vendoring the remote bases or for any operator upgrade. If the alert is a false positive,
+  fixing the rule is itself worth a change — a permanently-firing critical is a recorded, expensive
+  failure mode here.
+
+---
+Source of truth: `runbooks/kubevirt-control-plane.md`. Index: `/runbooks/`
+Records: commits `b57b3e9` (this alert's wrong metric/job label) and `6a7e9d1` (false `CDIOperatorDown`);
+`DECISIONS-LOG.md` (KubeVirt v1.9.0 + CDI v1.66.0 on nested KVM; the remote-base 27s fetch timeout).

runbooks/kubevirt-crashloop.md

@@ -0,0 +1,163 @@
+# kubevirt-crashloop — A VM's virt-launcher pod is restarting repeatedly
+
+**Alert:** `KubeVirtVMCrashLooping` (warning, 10m)
+**File:** `platform/kubevirt/alerts.yaml`, group `platform-health.kubevirt`
+**Expression:** `increase(kube_pod_container_status_restarts_total{pod=~"virt-launcher-.*"}[15m]) > 3`
+
+## What this means here
+
+More than three restarts of a `virt-launcher` pod in 15 minutes. The threshold is chosen to distinguish a
+crash-**loop** from a one-off reboot: a guest that restarts once is not interesting, a guest that restarts
+four times in a quarter of an hour is either panicking, being OOM-killed, or crashing qemu.
+
+Note what layer this watches: the **pod**, not the guest. `virt-launcher` wraps qemu, so a guest kernel
+panic, an OOM kill of the launcher, and a qemu crash all present the same way here. The distinction comes
+from the logs and the events, not from the alert.
+
+This is a `warning`, correctly: the VM tier is deliberately non-gating (`apps-vms` is `wait: false`) and a
+crash-looping VM does not affect the container platform. The one caveat is that on this network the UniFi
+VM is the home network's controller, so *that* VM crash-looping is user-visible.
+
+Two platform facts that shape the likely causes:
+
+- These are **HP EliteDesk 800 G5 Minis with ~22 GiB usable each**, running the whole platform. A VM is
+  the largest discretionary memory consumer on a node, so OOM is a live possibility rather than a
+  theoretical one.
+- **`useEmulation: false` is set deliberately** — on KubeVirt v1.9, `true` silently forces TCG. A VM
+  running under TCG burns a whole core per vCPU, and heavy sustained TCG load has been observed to degrade
+  the Cilium datapath into intermittent LB-VIP unreachability (cleared by a restart). So "the VM is
+  unstable **and** the network is flaky" is usually one cause, not two.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+Which launcher, how many restarts, on which node:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get pods -A -l kubevirt.io=virt-launcher -o wide; echo; \
+   kubectl --context k3s-prod get vmi -A -o wide'
+```
+
+Why it died — the previous container's log is the one that matters:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n unifi logs -l kubevirt.io=virt-launcher --previous --tail=120 2>&1 | tail -60'
+```
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n unifi describe pod -l kubevirt.io=virt-launcher 2>&1 | sed -n "/Last State/,\$p" | head -40'
+```
+
+Was it OOM-killed?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get events -A --field-selector reason=OOMKilling --sort-by=.lastTimestamp | tail -10; echo; \
+   kubectl --context k3s-prod top pods -A --sort-by=memory | head -10'
+```
+
+```
+ssh k3s-prod-1 'dmesg -T | grep -iE "oom|killed process" | tail -10; free -h; swapon --show'
+```
+
+(An empty `swapon --show` is what you want. On 2026-09-07 all three prod nodes came back from a reboot with
+977 MiB of swap **active** while the role reported success — `SwapTotal == 0` is not evidence swap is off.
+See `/runbooks/node-memory`.)
+
+Is it really TCG rather than KVM?
+
+```
+ssh k3s-prod-1 'ls -l /dev/kvm; grep -c -E "vmx|svm" /proc/cpuinfo; ps -eo pid,pcpu,comm | grep -i qemu'
+```
+
+Is the disk healthy underneath?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
+     -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,STATE:.status.state | grep -v healthy'
+```
+
+## Likely causes, most likely first
+
+### 1. The guest is panicking or the guest OS is broken
+Read the serial console via the previous container's log. A guest that panics at boot after a change is
+usually a bad cloud-init or a bad image. Two precise traps here: **KubeVirt caps inline `userData` at 2048
+bytes** (the shipped payload is a deliberately minimal ~1.1 KB inline blob; `userDataSecretRef` was tried
+and reverted), and **`apps-vms` must NOT carry `postBuild.substituteFrom`** — Flux's envsubst rewrote
+`${...}` inside the cloud-init and corrupted it (commit `da1f833`). Do not restore substitution there for
+consistency with the app waves.
+
+### 2. The launcher is being OOM-killed
+On ~22 GiB nodes running the whole platform, a VM with a generous memory request plus Prometheus (3Gi) and
+Grafana (768Mi) is tight. Remediation: lower the VM's memory in git, or address node memory
+(`/runbooks/node-memory`). Check swap is genuinely off first — active swap makes the kubelet's accounting
+lie and eviction misbehave.
+
+### 3. qemu is crashing
+The launcher log will carry the qemu error. Most often a device or firmware mismatch: a UEFI image needs
+`firmware.efi`, and `containerDisk` does not work here at all (it requires the ALPHA `ImageVolume` gate,
+off on k3s — use a CDI DataVolume). See `/runbooks/kubevirt-launch-failed`.
+
+### 4. The disk is unavailable or degraded mid-flight
+A `longhorn-vm` volume that went degraded or faulted takes the guest with it. Cross-read
+`/runbooks/longhorn-degraded` and `/runbooks/longhorn-faulted`. Remember `longhorn-vm` is **Block mode
+only** and `migratable: true`; a VM on plain `longhorn` cannot live-migrate.
+
+### 5. Node I/O pressure is starving the guest
+A guest whose virtual disk cannot complete writes will wedge and be restarted. `/runbooks/node-io` — and
+note the guest's writes land on the same partition as the embedded etcd WAL, so a thrashing VM is a
+control-plane risk, not just its own problem.
+
+### 6. It is running under TCG
+`useEmulation: true` forces it on v1.9. Confirm with `/dev/kvm` presence and qemu CPU usage. Fix in git.
+
+## When to escalate or stop
+
+- **Bound the damage first.** If the VM is thrashing the disk, its restarts are loading the node that
+  etcd writes to. If `/runbooks/node-io` is also firing, stopping the VM (an operator action on the `VM`
+  object, `spec.running: false` in git) is the right move — protecting the control plane beats keeping one
+  VM trying.
+- **Do not delete the DataVolume or the PVC.** A re-import is hours of I/O on the shared partition.
+- **Do not `kubectl drain`** the node to relocate it. This cluster does not drain, and a VM live-migrates
+  on drain **only** with per-VM `evictionStrategy: LiveMigrate`; a blanket setting would wedge drains for
+  RWO-disk VMs.
+- Escalate if the guest needs in-guest repair — that is console work, and for the UniFi VM it is also
+  home-network-visible.
+- If the alert is firing for a `virt-launcher` in a leaked test namespace, deleting that namespace is the
+  fix; leaked TCG test namespaces have caused API blips here before.
+
+---
+Source of truth: `runbooks/kubevirt-crashloop.md`. Index: `/runbooks/`
+Records: `DECISIONS-LOG.md` Session 7 (`useEmulation`, `firmware.efi`, `containerDisk`/ImageVolume);
+commit `da1f833` (cloud-init corrupted by envsubst); the 2026-09-07 swap-persistence finding.

runbooks/kubevirt-launch-failed.md

@@ -0,0 +1,173 @@
+# kubevirt-launch-failed — One or more VirtualMachineInstances are in the Failed phase
+
+**Alert:** `KubeVirtVMLaunchFailing` (critical, 5m)
+**File:** `platform/kubevirt/alerts.yaml`, group `platform-health.kubevirt`
+**Expression:** `sum by (namespace) (kubevirt_vmi_phase_count{phase="Failed"}) > 0`
+
+## What this means here
+
+A VMI reached the `Failed` phase — a scheduling, launch or qemu failure. The IaaS path is broken for that
+workload, which is why it is page-worthy even though the VM tier is deliberately **non-gating**
+(`apps-vms` is `wait: false`, `dependsOn: platform-config`, because a VM's image import and in-guest
+install are legitimately long-not-Ready).
+
+What actually runs here: the `unifi` namespace holds the UniFi VM (rootdisk 40 GiB, Block mode on
+`longhorn-vm`). **UniFi OS Server is not yet installed in the guest** — that is known, documented state,
+not a launch failure.
+
+**Live note, measured 2026-09-13:** `kubevirt_vmi_phase_count` returned **0 series** on prod. That metric
+only exists while VMIs do, so this alert is currently evaluating against nothing. Confirm the real state
+from `kubectl get vmi -A`, not from this alert's silence. (`kubevirt-prometheus-metrics` *is* a live scrape
+job and `kubevirt_virt_api_ready_status` does publish, so the target is up — see
+`/runbooks/kubevirt-control-plane`.)
+
+### The three k3s-specific launch failures this platform already hit and fixed
+
+Recorded from bring-up, and worth checking against any new VM definition because all three were
+non-obvious:
+
+1. **`containerDisk` requires the ALPHA `ImageVolume` feature gate**, which is off on k3s. The fix was to
+   switch to a **CDI DataVolume on Longhorn** instead. A VM authored with `containerDisk` will not launch
+   here.
+2. **A UEFI image needs `firmware.efi`** set on the VM. Without it, boot fails.
+3. **`useEmulation: true` silently forces TCG on KubeVirt v1.9** — it is set to `false` deliberately. A VM
+   that appears to launch but burns a whole core per vCPU is this; it also degrades the Cilium datapath
+   under sustained load.
+
+Also recorded: a `virt-handler` certificate error seen during that work was **unrelated and cosmetic** —
+do not chase it.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+Which VMI, in which namespace, and what phase:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get vmi -A -o wide; echo; \
+   kubectl --context k3s-prod get vm -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,STATUS:.status.printableStatus,READY:.status.ready'
+```
+
+The reason — the VMI status conditions and the launcher pod's events are where it is stated:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n unifi describe vmi unifi 2>&1 | sed -n "/Status:/,\$p" | head -60'
+```
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n unifi get pods -l kubevirt.io=virt-launcher -o wide; echo; \
+   kubectl --context k3s-prod -n unifi describe pod -l kubevirt.io=virt-launcher 2>&1 | sed -n "/Events:/,\$p"'
+```
+
+Is the disk there? (A missing or still-importing DataVolume is the most common cause.)
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get datavolumes.cdi.kubevirt.io -A \
+     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PHASE:.status.phase,PROGRESS:.status.progress; echo; \
+   kubectl --context k3s-prod get pvc -A | grep -Ei "rootdisk|dv|unifi"'
+```
+
+Is nested virtualisation actually available on the node it was scheduled to?
+
+```
+ssh k3s-prod-1 'ls -l /dev/kvm; grep -c -E "vmx|svm" /proc/cpuinfo'
+```
+
+Is the storage class right for a live-migratable VM?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get sc; echo; \
+   kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy'
+```
+
+And whether the KubeVirt control plane can act at all:
+
+```
+infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kubevirt get pods'
+```
+
+## Likely causes, most likely first
+
+### 1. The disk is not ready — a CDI import that has not completed or has stalled
+The VM cannot start without its DataVolume. Go to `/runbooks/cdi-import-stuck`. The specific recorded
+failure: the **3-replica default class failed the UniFi 60Gi import with insufficient storage /
+`ReplicaSchedulingFailure`**, which is why a dedicated single-replica **`longhorn-scratch`** class exists
+and is referenced by the CDI CR's `scratchSpaceStorageClass`.
+
+### 2. The VM is authored in a way k3s does not support
+The three items above: `containerDisk` without the ALPHA `ImageVolume` gate (use a CDI DataVolume),
+missing `firmware.efi` for a UEFI image, or `useEmulation: true` forcing TCG. All are git changes to the
+VM definition.
+
+### 3. The cloud-init payload is too large or corrupted
+Two precise traps. **KubeVirt caps inline `userData` at 2048 bytes** — the original rich cloud-init
+exceeded it; `userDataSecretRef` was tried and reverted, and the shipped state is a minimal ~1.1 KB inline
+payload. And: **`apps-vms` must NOT carry `postBuild.substituteFrom`** — Flux's envsubst rewrote `${...}`
+inside the cloud-init and corrupted it (commit `da1f833`). *Do not restore substitution there for
+consistency with the app waves.* If a VM stopped booting right after someone "tidied up" the Flux
+Kustomizations, this is it.
+
+### 4. It cannot be scheduled
+No node with room, or a storage constraint. A live-migratable VM needs **`longhorn-vm`**: RWX,
+**Block mode only** (migratable RWX is a raw block device, not the share-manager/NFS path), annotated
+`storageclass.kubevirt.io/is-default-virt-class: "true"` — which is *not* the cluster default class.
+A VM pointed at plain `longhorn` will not live-migrate.
+
+### 5. The CDI importer cannot open the raw Block device
+`nonroot-devices: true` is set on the k3s servers because the non-root CDI importer must open a raw Block
+device that Kubernetes never chowns for it (commit `df5979f`). A permission error in the importer on a
+Block PVC is this.
+
+### 6. The KubeVirt control plane is down
+Then nothing can launch. `/runbooks/kubevirt-control-plane`.
+
+## When to escalate or stop
+
+- **Scope it before escalating.** The VM tier is non-gating and does not affect the container platform.
+  One failed VM is not a platform outage — but the UniFi VM is the home network's controller, so treat
+  that one as user-visible.
+- **Do not delete the VM or its DataVolume to "start clean".** A 60 GiB re-import is hours of I/O on disks
+  that share a partition with the embedded etcd WAL, and that is the mechanism behind the 2026-09-07 node
+  outage. Fix the definition, not the data.
+- **Do not set `evictionStrategy: LiveMigrate` cluster-wide.** A blanket setting would **wedge drains**
+  for RWO-disk VMs; it is per-VM on purpose.
+- **Do not add `postBuild.substituteFrom` to `apps-vms`.** It corrupts cloud-init.
+- Escalate for anything that needs a fresh image import during a storage incident, and for changes to the
+  k3s server flags (`nonroot-devices`) — those are Ansible/operator work.
+
+---
+Source of truth: `runbooks/kubevirt-launch-failed.md`. Index: `/runbooks/`
+Records: `DECISIONS-LOG.md` Session 7 (the three k3s-specific launch failures); commits `da1f833`
+(cloud-init corrupted by envsubst) and `df5979f` (`nonroot-devices`); the `longhorn-scratch` /
+UniFi 60Gi import finding.

runbooks/kustomization.yaml

@@ -0,0 +1,86 @@
+---
+# Runbooks tier — reconciled by its OWN non-gating post-platform Flux Kustomization
+# (clusters/base/runbooks.yaml, dependsOn platform-config, wait:false). Same placement and rationale as
+# platform/homepage: a leaf web server must never sit in the gating `platform` tier, where a UI's
+# unhealth once stalled the whole chain (DECISIONS-LOG: "Headlamp gating stall").
+#
+# WHY THE MANIFESTS LIVE HERE, ALONGSIDE THE DOCS, RATHER THAN IN platform/runbooks/.
+# The markdown is the source of truth and the task is to serve it verbatim, which means the ConfigMap must
+# be GENERATED from these files — there must not be a second hand-maintained copy of 34 documents that can
+# drift from the originals. `configMapGenerator` does that, and its `files:` paths are resolved relative to
+# THIS directory. A `platform/runbooks/kustomization.yaml` would therefore need `../../runbooks/<name>.md`,
+# which depends on kustomize load restrictions being relaxed — true for Flux's kustomize-controller,
+# refused by a plain `kustomize build`, and so a divergence between what the gate can verify and what the
+# cluster does. Making this directory the Flux path removes the question entirely: no `../`, identical
+# behaviour locally and in-cluster, one copy of every document. The serving manifests are tucked under
+# `serving/` so `runbooks/*.md` stays exactly the flat set of pages the URLs name.
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: runbooks
+resources:
+  - serving/namespace.yaml
+  - serving/nginx-config.yaml
+  - serving/deployment.yaml
+  - serving/service.yaml
+  # default-deny baseline + Cilium gateway-ingress allow (DNS-only egress from the baseline).
+  - serving/networkpolicy.yaml
+  - serving/httproute.yaml   # apex `/runbooks` prefix (binds the Gateway `https-apex` listener)
+
+configMapGenerator:
+  # ONE ConfigMap holding every runbook, keyed by the EXACT name the alert annotations use.
+  #
+  # THE KEYS ARE EXTENSIONLESS ON PURPOSE. The annotations are `https://${BASE_DOMAIN}/runbooks/<name>`
+  # with NO `.md`, and this repo must not change them (they are spread across four alerts.yaml files that
+  # other work is editing). `<key>=<file>` maps `node-down.md` on disk to the key `node-down`, the volume
+  # mount turns each key into a file under the nginx root, and `/runbooks/node-down` becomes a plain static
+  # lookup — no rewrite rule, nothing to get subtly wrong at 3am. Drop the `<key>=` and every key would
+  # carry `.md`, so every alert link would 404 while this file still looked correct.
+  #
+  # The generated name carries kustomize's content hash, so editing any runbook rolls the server
+  # automatically and a stale projection is impossible.
+  #
+  # ADDING A RUNBOOK: add the file AND its line here. The L0 check
+  # conformance/tests/l0-static/alert-runbooks-resolvable asserts both halves, because a file in git with
+  # no generator entry is reviewed, merged, apparently done — and still a 404 in the cluster.
+  - name: runbooks-content
+    options:
+      labels:
+        app.kubernetes.io/name: runbooks
+    files:
+      # The index, served at /runbooks/ (nginx `try_files /runbooks/README`).
+      - README=README.md
+      - backup-failure=backup-failure.md
+      - backup-missed=backup-missed.md
+      - backup-partial=backup-partial.md
+      - cdi-import-stuck=cdi-import-stuck.md
+      - cert-expiry=cert-expiry.md
+      - cert-issuance=cert-issuance.md
+      - cnpg-archiving=cnpg-archiving.md
+      - cnpg-backup-missed=cnpg-backup-missed.md
+      - cnpg-exporter=cnpg-exporter.md
+      - cnpg-primary-spread=cnpg-primary-spread.md
+      - cnpg-wal-growth=cnpg-wal-growth.md
+      - denied-flows=denied-flows.md
+      - flux-reconcile=flux-reconcile.md
+      - flux-suspended=flux-suspended.md
+      - image-automation=image-automation.md
+      - image-policy-stalled=image-policy-stalled.md
+      - keycloak-client-secrets=keycloak-client-secrets.md
+      - keycloak-db=keycloak-db.md
+      - keycloak-down=keycloak-down.md
+      - kubevirt-control-plane=kubevirt-control-plane.md
+      - kubevirt-crashloop=kubevirt-crashloop.md
+      - kubevirt-launch-failed=kubevirt-launch-failed.md
+      - longhorn-capacity=longhorn-capacity.md
+      - longhorn-degraded=longhorn-degraded.md
+      - longhorn-faulted=longhorn-faulted.md
+      - monitoring-capacity=monitoring-capacity.md
+      - node-cpu=node-cpu.md
+      - node-disk=node-disk.md
+      - node-down=node-down.md
+      - node-io=node-io.md
+      - node-memory=node-memory.md
+      - pvc-capacity=pvc-capacity.md
+      - targets-down=targets-down.md
+      - update-rollback=update-rollback.md
+      - vpn-down=vpn-down.md

runbooks/longhorn-capacity.md

@@ -0,0 +1,158 @@
+# longhorn-capacity — A Longhorn volume or node is near its storage capacity
+
+**Alerts:** `LonghornVolumeCapacityHigh` (warning, 30m) and `LonghornNodeStorageHigh` (warning, 30m)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.longhorn`
+**Expressions:**
+`(longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85`
+`(longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes) > 0.85`
+
+## What this means here
+
+Either one volume is 85% full, or one node's Longhorn disk is 85% allocated. They are different
+problems with different fixes, so read the labels: `volume` means a workload is filling its own PVC;
+`node` means the cluster is running out of places to put replicas.
+
+The node case is bounded by settings that are already conservative:
+`storageReservedPercentageForDefaultDisk: 20`, `storageMinimalAvailablePercentage: 25`,
+`storageOverProvisioningPercentage: 200`, `defaultDataLocality: disabled`, hard node anti-affinity.
+Measured headroom on prod is roughly **269 GiB scheduled against an 829 GiB ceiling**, so the node
+alert firing means something has changed materially, not that the platform has slowly grown.
+
+The severe consequence of running out of node storage is that a degraded volume **cannot finish
+rebuilding** — hard anti-affinity needs three distinct nodes with room, so one full node pins a
+volume at 2 replicas indefinitely. Cross-read `/runbooks/longhorn-degraded`.
+### Verify the alert can even fire (measured 2026-09-13)
+
+Prometheus on prod scrapes 19 jobs: `apiserver`, `cdi-prometheus-metrics`, `coredns`,
+`keycloak-discovery`, `keycloak-service`, `keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`,
+`kube-prometheus-stack-grafana`, `kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`,
+`kube-state-metrics`, `kubelet`, `kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`,
+`node-exporter`, `oauth2-proxy`, `semaphore-db/semaphore-db`, `velero`, `velero/node-agent`.
+There is **no** Longhorn, cert-manager, Flux-controller or Hubble/Cilium job. Check the series this
+rule depends on before you trust its silence:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & \
+   sleep 5; curl -s -G --data-urlencode "query=count by (job) (up)" localhost:19090/api/v1/query'
+```
+
+**These alerts are currently INERT.** `count(longhorn_volume_capacity_bytes)` and
+`count(longhorn_node_storage_capacity_bytes)` both returned **0 series** on prod on 2026-09-13 —
+Longhorn ships no ServiceMonitor here. Volume-fill coverage actually comes from
+`kubelet_volume_stats_*` instead, i.e. `/runbooks/pvc-capacity`, which is where the working alerts
+are. Use this runbook for the Longhorn-side (node allocation) half, and expect to reach it by hand.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+Node-level allocation (the half that has no substitute alert):
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o yaml | grep -E "name:|storageAvailable|storageMaximum|storageScheduled|allowScheduling"'
+```
+
+Volume sizes versus actual usage:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
+     -o custom-columns=NAME:.metadata.name,SPEC:.spec.size,ACTUAL:.status.actualSize,ROBUST:.status.robustness'
+```
+
+Which PVC each one is, so you know who is filling it:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get pv -o custom-columns=PV:.metadata.name,CLAIM:.spec.claimRef.namespace/.spec.claimRef.name,SIZE:.spec.capacity.storage,SC:.spec.storageClassName'
+```
+
+The filesystem view, which is the alert that actually works:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
+   curl -s -G --data-urlencode "query=topk(10, 1 - kubelet_volume_stats_available_bytes{job=\"kubelet\"} / kubelet_volume_stats_capacity_bytes{job=\"kubelet\"})" localhost:19090/api/v1/query'
+```
+
+On the node itself:
+
+```
+ssh k3s-prod-1 'df -h /data; du -xh --max-depth=1 /data/longhorn/replicas 2>/dev/null | sort -h | tail'
+```
+
+## Likely causes, most likely first
+
+### 1. A CNPG database is not recycling WAL (check this FIRST)
+The only cause here that grows without bound. `archive_mode=on` plus a failing `archive_command` means
+Postgres will not recycle unarchived WAL; measured on prod `keycloak-db` at ~6.8 MiB/h with `pg_wal`
+sharing the data PVC (no `walStorage:` is configured on any Cluster) — about 29 days from 93% free to
+disk-full, ending in a postmaster PANIC. Go to `/runbooks/cnpg-archiving`. **Never delete WAL by
+hand.**
+
+### 2. Snapshots and orphaned replicas accumulating
+Longhorn snapshots are not free, and cleanup here is **manual**: `orphanAutoDeletion: true` in
+`platform/longhorn/helmrelease.yaml` has been **silently inert since Longhorn 1.12** — it was renamed
+`orphanResourceAutoDeletion` (now a list) and the chart drops unknown `defaultSettings` keys with no
+error. Verify the key's current state in that file before citing it; then prune orphans and stale
+snapshots from the Longhorn UI.
+
+### 3. A workload genuinely grew
+Plex/Sonarr/Radarr **media** does not live on Longhorn — it is static NFS from
+`192.168.2.102:/mnt/Storage/plex`, via the `bulk` tier. If a media volume is on `longhorn`, that is a
+misconfiguration worth fixing, not a capacity problem: the `bulk` StorageClass is a stub
+(`provisioner: nfs.csi.k8s.io`, `server:/share: REPLACE-ME`) and provisions nothing, so real bulk
+storage is static NFS PVs. Otherwise expand the PVC — all Longhorn classes have
+`allowVolumeExpansion: true`; change the size in git and push.
+
+### 4. Replicas piled onto one node
+`replicaAutoBalance: best-effort` should prevent this, and hard anti-affinity bounds it to one replica
+per volume per node. If one node is much fuller than the others, check whether a rebuild placed
+replicas there while another node was down.
+
+### 5. The cluster has outgrown its disks
+Prod's ceiling is ~829 GiB across three nodes. `docs/plans/longhorn-ssd-tiering.md` is the planned
+answer (a ~51 GB hot tier: Prometheus, CNPG, Loki, Gitea). Operator planning work.
+
+## When to escalate or stop
+
+- **A node over 85% while any volume is degraded is urgent**, because the rebuild cannot complete.
+  Free space or restore the other node before anything else.
+- **Never delete files under `/data/longhorn`** or inside a Postgres `pg_wal`. Both are irreversible.
+- **Do not expand a PVC as a reflex** if the cause is unrecycled WAL — you are buying days and hiding
+  the real fault.
+- Escalate for adding disks, the SSD tiering plan, or anything that needs a node reboot (each reboot
+  degrades all ~18 volumes and triggers a rebuild storm).
+
+---
+Source of truth: `runbooks/longhorn-capacity.md`. Index: `/runbooks/`
+Records: `platform/longhorn/helmrelease.yaml` (the settings above and the inert `orphanAutoDeletion`);
+`platform/cnpg/alerts.yaml` header (the WAL disk-full clock); `docs/BULK-NFS-STORAGE.md`.

runbooks/longhorn-degraded.md

@@ -0,0 +1,170 @@
+# longhorn-degraded — A Longhorn volume has been degraded for 10 minutes
+
+**Alert:** `LonghornVolumeDegraded` (warning, 10m)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.longhorn`
+**Expression:** `longhorn_volume_robustness == 2` (0=unknown 1=healthy 2=degraded 3=faulted)
+
+## What this means here
+
+A volume has fewer healthy replicas than it should. It is **still serving I/O** — that is the
+difference between degraded and faulted (`/runbooks/longhorn-faulted`). What you have lost is
+redundancy, and on a 3-node cluster with hard node anti-affinity
+(`replicaSoftAntiAffinity: false`, `numberOfReplicas: 3`) one degraded volume means one more node
+loss away from data loss.
+
+**Degraded is the NORMAL state after any node reboot on this cluster, and it is expensive.**
+Recorded plainly in the stabilization notes: *"Longhorn on the HDD: every node reboot degrades all 18
+volumes → full rebuild (slow; 30%+ IO-delay). Roll reboots one node at a time; wait
+`robustness=healthy` before the next."* The rebuild is the thing that hurts, not the degradation:
+rebuild writes land on the same partition as the embedded etcd WAL (`/data/k3s` and `/data/longhorn`
+share one device), and that is the mechanism behind the 2026-09-07 `k3s-prod-1` NotReady outage. So
+the question to answer is never just "why is it degraded" but "is the rebuild about to take a node
+out". Cross-read `/runbooks/node-io`.
+### Verify the alert can even fire (measured 2026-09-13)
+
+Prometheus on prod scrapes 19 jobs: `apiserver`, `cdi-prometheus-metrics`, `coredns`,
+`keycloak-discovery`, `keycloak-service`, `keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`,
+`kube-prometheus-stack-grafana`, `kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`,
+`kube-state-metrics`, `kubelet`, `kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`,
+`node-exporter`, `oauth2-proxy`, `semaphore-db/semaphore-db`, `velero`, `velero/node-agent`.
+There is **no** Longhorn, cert-manager, Flux-controller or Hubble/Cilium job. Check the series this
+rule depends on before you trust its silence:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & \
+   sleep 5; curl -s -G --data-urlencode "query=count by (job) (up)" localhost:19090/api/v1/query'
+```
+
+**This alert is currently INERT.** `count(longhorn_volume_robustness)` returned **0 series** on prod
+on 2026-09-13: the whole `platform-health.longhorn` rule group has no data, because Longhorn ships no
+ServiceMonitor here. That is a separate open defect, and the practical consequence is that **you will
+probably learn about a degraded volume from an application symptom or from `volumes.longhorn.io`, not
+from this alert**. Do not infer "no alert, so volumes are fine".
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+The truth about robustness, straight from the CRs (this works whether or not the metric does):
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
+     -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,STATE:.status.state,NODE:.status.currentNodeID,SIZE:.spec.size'
+```
+
+Which replicas are missing, and where:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n longhorn-system get replicas.longhorn.io \
+     -o custom-columns=NAME:.metadata.name,VOL:.spec.volumeName,NODE:.spec.nodeID,RUNNING:.status.currentState | sort -k2'
+```
+
+Which PVC/workload is affected:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get pv -o custom-columns=PV:.metadata.name,CLAIM:.spec.claimRef.namespace/.spec.claimRef.name,SC:.spec.storageClassName'
+```
+
+Are the nodes and their disks schedulable, and is a rebuild in flight?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o wide; echo; \
+   kubectl --context k3s-prod -n longhorn-system get settings.longhorn.io concurrent-replica-rebuild-per-node-limit -o custom-columns=NAME:.metadata.name,VALUE:.value'
+```
+
+And the pressure that rebuild is causing — the reason this alert matters:
+
+```
+ssh k3s-prod-1 'cat /proc/pressure/io'
+```
+
+## Likely causes, most likely first
+
+### 1. A node rebooted or was briefly NotReady (overwhelmingly the most common)
+Expected, and it self-heals by rebuilding. Your job is to keep the rebuild from taking a node down.
+Remediation (operator, mutating) — freeze rebuilds while replicas will **reattach** (an in-place
+bounce):
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n longhorn-system patch settings.longhorn.io \
+     concurrent-replica-rebuild-per-node-limit --type=merge -p "{\"value\":\"0\"}"'
+```
+
+Thaw to `"1"` — never higher — once every volume reads `healthy`. Use `1` rather than `0` when the
+replicas are genuinely gone (an OS re-provision), so replenishment can actually proceed.
+`concurrentReplicaRebuildPerNodeLimit: 1` was **already in force during the 2026-09-07 outage and
+held**; do not "fix" this by setting a knob that is already set. `replicaRebuildingBandwidthLimit` is
+**v2/SPDK-only** and does nothing here; `rebuildConcurrentSyncLimit` is already at its floor of 1.
+What was actually tightened after that incident was every *other* Longhorn I/O source:
+`concurrentVolumeBackupRestorePerNodeLimit` and `snapshotHeavyTaskConcurrentLimit` 5 → 1.
+
+### 2. A node is still down
+Then the rebuild cannot complete and this will not clear. Go to `/runbooks/node-down` first; there is
+nothing to fix in Longhorn.
+
+### 3. A disk was replaced and Longhorn refuses it
+Signature: `DiskFilesystemChanged` — *"record diskUUID doesn't match the one on the disk"*.
+Remediation, in exactly this order (the validating webhook enforces it): **disable the disk → remove
+the disk entry → re-add it**. Seen during the 2026-09-07 drive swap.
+
+### 4. No node can hold the replica (anti-affinity or space)
+Hard anti-affinity means a 3-replica volume needs three distinct schedulable nodes. With one node
+cordoned or its disk over `storageMinimalAvailablePercentage: 25`, the third replica cannot be
+placed and the volume stays degraded indefinitely. Check `nodes.longhorn.io` scheduling and
+`/runbooks/longhorn-capacity`.
+
+### 5. `replicaAutoBalance: best-effort` is shuffling replicas
+Normal background behaviour, transient. If degradation follows a pattern of replicas moving and never
+settling, that is worth an operator's attention.
+
+## When to escalate or stop
+
+- **If more than one volume is degraded and I/O-PSI on any node is climbing, freeze rebuilds first
+  and then get the operator.** That combination is the 2026-09-07 failure mode in progress.
+- **Never reboot, cordon or re-provision a second node while any volume is degraded.** This is the
+  single most important rule in this runbook.
+- **Do not `kubectl drain`.** This cluster deliberately does not drain: there is no Longhorn
+  `nodeDrainPolicy` configured anywhere in the repo, the upgrade plan rejects
+  system-upgrade-controller because "its auto-drain fights this cluster's slow-HDD Longhorn", and the
+  node-loss conformance test evicts only its own writer pod.
+- **Do not delete a degraded volume's replicas to "force a clean rebuild"** unless the operator agrees
+  — with 3 replicas and one already missing you are one mistake from `/runbooks/longhorn-faulted`.
+- Escalate for: a node that will not come back, a disk replacement, or a volume degraded for more than
+  a few hours with all three nodes Ready (that is a Longhorn bug, not an operational state).
+
+---
+Source of truth: `runbooks/longhorn-degraded.md`. Index: `/runbooks/`
+Records: `DECISIONS-LOG.md` "2026-08-31 — Post-upgrade stabilization" (the reboot tax) and
+"2026-09-07 — k3s-prod-1 NotReady"; `docs/plans/cluster-upgrade-plan.md` §6.2/§7 (the freeze/thaw).

runbooks/longhorn-faulted.md

@@ -0,0 +1,164 @@
+# longhorn-faulted — A Longhorn volume is FAULTED (data at risk, volume offline)
+
+**Alert:** `LonghornVolumeFaulted` (critical, 1m)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.longhorn`
+**Expression:** `longhorn_volume_robustness == 3`
+
+## What this means here
+
+Faulted means Longhorn has **no usable replica**. The volume is offline; the workload's writes are
+failing. This is the most serious storage state on the cluster and the `for: 1m` is deliberate — there
+is no benign transient version of this.
+
+Understand what is and is not recoverable before you touch anything:
+
+- The volumes on this cluster are 3-replica with **hard** node anti-affinity, so faulted normally
+  means two or three replicas are gone at once — two nodes lost, or one node lost while another
+  volume's rebuild had not finished, or a disk that ate its replica.
+- **Standbys and replicas are not backups.** Recorded verbatim for the databases and true for all of
+  this: *"Standbys replicate corruption and deletions faithfully; only the Barman archive is a
+  backup."* For a CNPG volume (`keycloak-db`, `nextcloud-db`, `semaphore-db`) the real recovery path
+  is the object-store archive, not Longhorn. For everything else it is Velero.
+- If this is a **`longhorn-vm`** volume (RWX Block, `migratable: true`) the VM is down, not just the
+  disk. If it is **`longhorn-scratch`** (single replica, CDI scratch space) losing it is expected and
+  harmless — re-run the import.
+### Verify the alert can even fire (measured 2026-09-13)
+
+Prometheus on prod scrapes 19 jobs: `apiserver`, `cdi-prometheus-metrics`, `coredns`,
+`keycloak-discovery`, `keycloak-service`, `keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`,
+`kube-prometheus-stack-grafana`, `kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`,
+`kube-state-metrics`, `kubelet`, `kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`,
+`node-exporter`, `oauth2-proxy`, `semaphore-db/semaphore-db`, `velero`, `velero/node-agent`.
+There is **no** Longhorn, cert-manager, Flux-controller or Hubble/Cilium job. Check the series this
+rule depends on before you trust its silence:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & \
+   sleep 5; curl -s -G --data-urlencode "query=count by (job) (up)" localhost:19090/api/v1/query'
+```
+
+**This alert is currently INERT.** `count(longhorn_volume_robustness)` returned **0 series** on prod
+on 2026-09-13 — Longhorn ships no ServiceMonitor here, so the whole `platform-health.longhorn` group
+has no data. A faulted volume today announces itself as an application outage. That is a known open
+defect; treat `volumes.longhorn.io` as the source of truth, not this alert's silence.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+Stop and take inventory before changing anything. Which volume, which workload, which replicas exist:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
+     -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,STATE:.status.state,NODE:.status.currentNodeID | grep -Ev "healthy"'
+```
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n longhorn-system get replicas.longhorn.io \
+     -o custom-columns=NAME:.metadata.name,VOL:.spec.volumeName,NODE:.spec.nodeID,STATE:.status.currentState,FAILED:.spec.failedAt | sort -k2'
+```
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get pv -o custom-columns=PV:.metadata.name,CLAIM:.spec.claimRef.namespace/.spec.claimRef.name,SC:.spec.storageClassName | grep <volume-name>'
+```
+
+What recovery material exists — **fully-qualified names, or you will be told there is nothing**:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get backups.velero.io -A; echo; \
+   kubectl --context k3s-prod get backups.postgresql.cnpg.io -A; echo; \
+   kubectl --context k3s-prod -n longhorn-system get backups.longhorn.io'
+```
+
+`kubectl get backup -A` prints "No resources found" on this cluster even when six Velero and 27 CNPG
+backups exist. Never make a recovery decision from the short name.
+
+Node and disk state:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get nodes; kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o wide'
+```
+
+## Likely causes, most likely first
+
+### 1. Two nodes were unavailable at once
+Usually a second reboot or cordon while the first node's rebuilds were still running. Remediation:
+bring the nodes back. Longhorn will reattach if the replica data survived on disk — which is why you
+must not delete replicas before checking.
+
+### 2. A disk failed and took its replica with it
+`k3s-prod-1`'s original 2011 SSD had 61,440 reallocated sectors before it was replaced; the dev-era
+Seagate had 448 current-pending plus 448 offline-uncorrectable. Check SMART on the node holding the
+failed replica (`/runbooks/node-io` has the command). If the disk is bad, the replica is gone — the
+path forward is restore, not repair.
+
+### 3. All replicas were evicted or deleted
+Check whether a disk was disabled, a node removed from Longhorn, or a `replicas.longhorn.io` object
+deleted. `staleReplicaTimeout: 30` (minutes) means Longhorn eventually discards replicas it considers
+stale.
+
+### 4. It is a single-replica volume by design
+`longhorn-scratch` is `numberOfReplicas: 1`. Losing it is not an incident — delete the DataVolume and
+re-import.
+
+## Remediation, in order
+
+1. **Do not delete the volume, the PVC, or any replica.** A faulted volume with a stale replica is
+   recoverable; a deleted one is not.
+2. If a node is down, restore the node (`/runbooks/node-down`). That is the highest-probability fix.
+3. If Longhorn shows a replica with data but marked failed, an operator can salvage it from the
+   Longhorn UI (*Volume → Salvage*) — this is the documented path for "all replicas failed but the
+   data is still there".
+4. If the volume backs a **CNPG database**, do not try to salvage Postgres files. Recover the Cluster
+   from the Barman archive (`/runbooks/cnpg-archiving` for how to confirm the archive is real first).
+5. Otherwise restore from Velero. Note honestly: `dr-restore` is **authored and has never been
+   executed** (`conformance.home.k3s/never-executed`), and the R2 restore path has not been rehearsed.
+   Expect to be the first person to run it, and read `/runbooks/backup-failure` for the R2 plugin
+   caveats before you rely on a backup's metadata.
+
+## When to escalate or stop
+
+- **Get the operator now.** This is a critical, data-at-risk alert; it is not a runbook-and-carry-on
+  situation. Everything above is diagnosis, and steps 3–5 are operator actions.
+- **Stop before any destructive command.** Specifically: no `delete pvc`, no `delete
+  volumes.longhorn.io`, no `delete replicas.longhorn.io`, no touching files under `/data/longhorn`.
+- **Do not reboot anything** while a volume is faulted.
+- If the affected volume is `keycloak-db`, expect a full SSO outage as a knock-on: kubectl OIDC,
+  Grafana, Headlamp, Gitea and forward-auth all depend on it. The certificate-based kubeconfig is an
+  independent break-glass path and still works.
+
+---
+Source of truth: `runbooks/longhorn-faulted.md`. Index: `/runbooks/`
+Records: `DECISIONS-LOG.md` (drive failures 2026-08-22 and 2026-09-07; `dr-restore` never executed).

runbooks/monitoring-capacity.md

@@ -0,0 +1,133 @@
+# monitoring-capacity — The monitoring stack's own PVC is filling up
+
+**Alert:** `MonitoringVolumeFillingUp` (warning, 15m)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.monitoring-self`
+**Expression:** `kubelet_volume_stats_available_bytes{namespace="monitoring"} / kubelet_volume_stats_capacity_bytes{namespace="monitoring"} < 0.15`
+
+## What this means here
+
+"Don't let the monitor blind itself." Prometheus or Loki is running out of disk. If it fills, you lose
+the ability to see every other alert in this set — which is why `monitoring` gets its own rule instead
+of sharing `/runbooks/pvc-capacity` (the two files partition the namespace space between them; the
+thresholds are deliberately identical, 15% free for 15m).
+
+Live shape on prod:
+
+- `prometheus-...-prometheus-0` PVC **50Gi**, `retention: 30d`, `retentionSize: 45GiB`
+- `storage-loki-0` PVC **50Gi**
+- both on the `longhorn` class, 3 replicas
+
+`retentionSize: 45GiB` against a 50Gi PVC is intentional: Prometheus is *supposed* to plateau near the
+high-water mark and then stay there. **So a Prometheus TSDB approaching 45 GiB is normal, not a leak.**
+That is also why the predictive rule in `pvc-capacity-alerts.yaml` deliberately **excludes** the
+`monitoring` namespace: a TSDB filling toward its retention limit looks exactly like an unbounded leak
+to `predict_linear` until it plateaus.
+
+So the real question is *which* PVC and *whether it should be plateauing*: read the
+`persistentvolumeclaim` label first. Loki has no size cap analogous to `retentionSize`, so a log
+volume growing past 85% is the more likely genuine problem of the two.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring get pvc; echo; \
+   kubectl --context k3s-prod -n monitoring get prometheus -o jsonpath="{.items[0].spec.retention} {.items[0].spec.retentionSize}{\"\n\"}"'
+```
+
+Actual on-disk usage from inside the pods:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- df -h /prometheus; \
+   kubectl --context k3s-prod -n monitoring exec loki-0 -- df -h /var/loki'
+```
+
+Is Prometheus at its retentionSize (normal) or past it (a problem)?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
+   for q in "prometheus_tsdb_storage_blocks_bytes" "prometheus_tsdb_head_series" "rate(prometheus_tsdb_compactions_failed_total[1h])"; do \
+     printf "%s " "$q"; curl -s -G --data-urlencode "query=$q" localhost:19090/api/v1/query | python3 -c "import json,sys;print([x[\"value\"][1] for x in json.load(sys.stdin)[\"data\"][\"result\"]])"; done'
+```
+
+Series growth is the usual driver — find the biggest contributors:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
+   curl -s "localhost:19090/api/v1/status/tsdb" | python3 -m json.tool | head -40'
+```
+
+## Likely causes, most likely first
+
+### 1. Prometheus has reached its retention high-water mark (expected)
+`retentionSize: 45GiB` on a 50Gi PVC leaves 10% headroom, which is close to the 15% trigger. If
+`prometheus_tsdb_storage_blocks_bytes` is near 45 GiB and steady, **this is the design working** and
+the correct response is either to accept it or to raise both the PVC and `retentionSize` together in
+`platform/monitoring/kube-prometheus-stack.yaml`. Raising retention without raising the PVC is how you
+get a genuinely full disk.
+
+### 2. Loki log volume growth
+Loki has no retentionSize equivalent configured. A noisy workload (a crashlooping pod, a chatty
+reconciler) can grow it quickly. Remediation: find the noisy stream in Grafana → Explore, fix the
+source, and set/lower Loki retention in git.
+
+### 3. Series cardinality grew after a chart or workload change
+`serviceMonitorSelectorNilUsesHelmValues` and `podMonitorSelectorNilUsesHelmValues` are both `false`
+here, which means **Prometheus selects every ServiceMonitor/PodMonitor in the cluster regardless of
+labels**. That is convenient (no `additionalLabels` needed anywhere) and it also means a new
+component's monitor is picked up automatically, cardinality and all. Check
+`/api/v1/status/tsdb` for the top label pairs.
+
+### 4. Compaction is failing
+`rate(prometheus_tsdb_compactions_failed_total[1h]) > 0` means blocks are not being merged or expired.
+Usually an I/O problem underneath — cross-read `/runbooks/node-io`.
+
+### 5. The volume is on a node that is itself short of space
+`/runbooks/longhorn-capacity` and `/runbooks/node-disk`.
+
+## When to escalate or stop
+
+- **Do not delete Prometheus TSDB blocks or Loki chunks by hand.** Lower retention and let the
+  component expire them; hand-deleting blocks can leave the TSDB unopenable.
+- **Do not raise `retention` or `retentionSize` without raising the PVC in the same change.**
+- If Prometheus is already full and crashlooping, you are blind: check the alerts you care about
+  directly against the CRs (`clusters.postgresql.cnpg.io`, `volumes.longhorn.io`, `kubectl get nodes`)
+  rather than waiting for alerts that cannot be evaluated.
+- Escalate for PVC expansion — it needs node headroom, and this cluster has been memory- and
+  disk-constrained before (Prometheus was already raised 2 → 3Gi of RAM after the kps 65→88 upgrade).
+
+---
+Source of truth: `runbooks/monitoring-capacity.md`. Index: `/runbooks/`
+Records: `platform/monitoring/pvc-capacity-alerts.yaml` (why monitoring is excluded from the
+predictive rule); `DECISIONS-LOG.md` (Grafana OOM / Prometheus 2→3Gi during kps 65→88).

runbooks/node-cpu.md

@@ -0,0 +1,125 @@
+# node-cpu — CPU pressure or run-queue depth high on a node
+
+**Alerts:** `NodeCPUPressureHigh` (warning, 15m) and `NodeCPULoadHigh` (warning, 15m)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes`
+**Expressions:**
+`rate(node_pressure_cpu_waiting_seconds_total[5m]) > 0.15`
+`node_load5 / count without (cpu, mode) (node_cpu_seconds_total{mode="idle"}) > 1`
+
+## What this means here
+
+`NodeCPUPressureHigh` is PSI "some" for CPU: the fraction of wall-clock time during which at least
+one task was stalled waiting for a CPU. Under 5% is healthy. Averaged CPU% — what `kubectl top`
+shows — **hides this entirely**, which is why the alert uses PSI: it is the metric that surfaces the
+"spikes cause blips of delay" symptom operators actually notice.
+
+`NodeCPULoadHigh` is 5-minute load average divided by core count. Sustained above 1 means the run
+queue is longer than the number of CPUs.
+
+**Read them together.** CPU-PSI high *and* load high = genuine CPU contention. Load high *and* CPU-PSI
+normal = the queue is full of tasks blocked on **I/O**, not CPU — go to `/runbooks/node-io`, which on
+this cluster is the more dangerous of the two (etcd WAL and Longhorn replicas share one partition).
+
+Measured history: steady CPU PSI ran **15–23%** with load at or above core count, which is what drove
+the node resize from **6 to 8 vCPU**; afterwards PSI measured ~4.5–7.7%. So 0.15 is not an arbitrary
+threshold, it is the level the cluster actually lived at before the resize.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+Distinguish CPU contention from I/O contention — do this first:
+
+```
+ssh k3s-prod-1 'cat /proc/pressure/cpu; echo; cat /proc/pressure/io; echo; uptime; nproc'
+```
+
+Who is burning it:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod top nodes; echo; kubectl --context k3s-prod top pods -A --sort-by=cpu | head -20'
+```
+
+```
+ssh k3s-prod-1 'top -b -n1 | head -20'
+```
+
+Are there VMs, and are any of them emulated (the worst case)?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get vmi -A -o wide'
+```
+
+```
+ssh k3s-prod-1 'grep -c -E "vmx|svm" /proc/cpuinfo; ls -l /dev/kvm'
+```
+
+## Likely causes, most likely first
+
+### 1. It is actually I/O, not CPU
+Most common misread. A node blocked on disk has a long run queue and normal CPU-PSI. Confirm with
+`/proc/pressure/io` and go to `/runbooks/node-io`. On 2026-09-07 the node that died had I/O-PSI full
+at 11.36 while CPU looked unremarkable.
+
+### 2. A KubeVirt VM running under TCG emulation
+`useEmulation: true` **silently forces TCG on KubeVirt v1.9** — it is set to `false` on purpose. A VM
+that lost `/dev/kvm` or was authored with emulation burns a whole core per vCPU. Heavy sustained TCG
+load has also degraded the Cilium datapath into intermittent LB-VIP unreachability (cleared by a
+restart), so an "intermittent networking" report alongside this alert is probably one cause, not two.
+Check `grep -c -E "vmx|svm" /proc/cpuinfo` (nested KVM is verified working; Proxmox nodes use
+`cpu_type=host`).
+
+### 3. A Longhorn rebuild's checksumming
+Rebuilds are I/O-heavy but not free on CPU. Same remediation as `/runbooks/longhorn-degraded`: freeze
+with `concurrent-replica-rebuild-per-node-limit` `0`, thaw to `1`.
+
+### 4. Prometheus rule evaluation or a Loki query
+`PersistentVolumeFillingUpPredicted` runs `predict_linear` over a 24h window across every PVC; a
+Grafana panel doing the same interactively is heavier still. Bursty, self-limiting, usually harmless.
+
+### 5. Genuine capacity
+If nothing is anomalous the node is simply undersized again. The 6 → 8 vCPU resize is the precedent.
+A bpg cores change **reboots the VM**, so roll **one node at a time, waiting for Longhorn
+`robustness=healthy` between each**.
+
+## When to escalate or stop
+
+- **If I/O-PSI is also elevated, stop treating this as a CPU problem** and switch to
+  `/runbooks/node-io`. Wrong runbook is the main risk here.
+- Alone and steady, this alert is a capacity signal — schedule it, do not page on it. That is why it
+  is a warning with a 15m `for`.
+- **Do not drain or cordon** to shed CPU load; this cluster does not drain.
+- Escalate to the operator for a vCPU change (it reboots the node) or to kill a VM you did not create.
+
+---
+Source of truth: `runbooks/node-cpu.md`. Index: `/runbooks/`
+Records: `DECISIONS-LOG.md` (node vCPU 6→8 and the post-resize PSI measurements; KubeVirt
+`useEmulation` finding).

runbooks/node-disk.md

@@ -0,0 +1,131 @@
+# node-disk — A node filesystem is filling up
+
+**Alert:** `NodeDiskFillingUp` (warning, 15m)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes`
+**Expression:** `node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"} / node_filesystem_size_bytes{...} < 0.10`
+
+## What this means here
+
+Less than 10% free on a real filesystem on a node, for 15 minutes. Read the `mountpoint` label
+first, because on these nodes it tells you which of two very different problems you have.
+
+**`/data` is the one that matters.** Both the embedded **etcd WAL** (`/data/k3s`) and **every
+Longhorn replica** (`/data/longhorn`) live on that single partition —
+`node_baseline_etcd_source: shared-data-partition`, not a dedicated device, and the exit condition
+for splitting them is still open (`infra/PROD-PROVISIONING.md` §2). If `/data` fills:
+
+- etcd cannot write its WAL. A control-plane member with a full WAL partition does not degrade
+  gracefully; it stops, and this is a 3-member cluster.
+- Longhorn stops being able to hold replicas and volumes go degraded, which triggers rebuilds, which
+  need *more* space.
+
+So `/data` at 10% free is not a warning, it is the last comfortable moment to act.
+
+**`/` filling up** is usually logs, container images or a stuck journal. Lower stakes, but there is
+history: in the dev era the host root filled to ~99% because VM disks sat under a snapper-snapshotted
+subvolume, and the hypervisor **paused the k3s VMs on I/O error**. Recovery was deleting 11 snapper
+timeline snapshots plus `virsh destroy` on the paused VMs (which were holding rsync-unlinked inodes):
+99% → 30%.
+
+Longhorn's own guard rails are already tight and are why this alert usually fires before anything
+breaks: `storageReservedPercentageForDefaultDisk: 20`,
+`storageMinimalAvailablePercentage: 25`, `storageOverProvisioningPercentage: 200`.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+Which filesystem, on which node:
+
+```
+ssh k3s-prod-1 'df -h -x tmpfs -x overlay; echo; du -xh --max-depth=1 /data 2>/dev/null | sort -h | tail -10; echo; du -xh --max-depth=1 / 2>/dev/null | sort -h | tail -10'
+```
+
+What Longhorn thinks it has:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o custom-columns=NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,SCHED:.spec.allowScheduling; echo; \
+   kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,SIZE:.spec.size,ACTUAL:.status.actualSize,ROBUST:.status.robustness'
+```
+
+Journal and image bloat on `/`:
+
+```
+ssh k3s-prod-1 'sudo journalctl --disk-usage; sudo crictl images | wc -l; sudo du -sh /var/lib/rancher/k3s/agent/containerd 2>/dev/null'
+```
+
+## Likely causes, most likely first
+
+### 1. Longhorn replicas grew, or replicas landed here after a rebuild
+Hard node anti-affinity (`replicaSoftAntiAffinity: false`) means one replica per node per volume, but
+`replicaAutoBalance: best-effort` moves them. Prod sits around 269 GiB scheduled against an 829 GiB
+ceiling, so this is rarely tight — but a rebuild storm plus snapshots can spike it.
+Remediation: check for orphaned replicas and stale snapshots in the Longhorn UI. Note
+`orphanAutoDeletion: true` in `platform/longhorn/helmrelease.yaml` **has been silently inert since
+Longhorn 1.12** — it was renamed `orphanResourceAutoDeletion` (a list) and the chart drops unknown
+`defaultSettings` keys with no error. So orphan cleanup is manual. Verify the key's current state in
+the file before you trust it.
+
+### 2. A CNPG database is not recycling WAL
+The single most dangerous cause, because it is unbounded. With `archive_mode=on` and a failing
+`archive_command`, Postgres **refuses to recycle** unarchived WAL segments and `pg_wal` grows forever
+— on prod's `keycloak-db` that was measured at ~6.8 MiB/h, 17 segments / ~268 MiB, about 29 days to
+fill a 5 GiB PVC. `pg_wal` shares the data PVC (no `walStorage:` is configured on any Cluster).
+Check `/runbooks/cnpg-archiving` **before** you free any space, and **never delete WAL by hand** —
+that destroys the database.
+
+### 3. Journal / container images on `/`
+`journalctl --vacuum-size=200M` and pruning unused images are safe. Do not prune images for
+workloads that may need to restart while the registry is unreachable.
+
+### 4. A Velero or CDI staging write
+Velero's kopia uploads and CDI's scratch space both stage locally. `longhorn-scratch`
+(single-replica) exists precisely because the 3-replica default failed the UniFi 60Gi import on
+insufficient storage.
+
+### 5. The disk is smaller than the platform now needs
+`docs/plans/longhorn-ssd-tiering.md` holds the planned hot tier (~51 GB: Prometheus, CNPG, Loki,
+Gitea). Working set measured ~79 GB against ~323 GB free on SSD, so the "won't fit" blocker was a
+measurement artifact. This is operator planning work, not an incident fix.
+
+## When to escalate or stop
+
+- **If the mountpoint is `/data` and free space is under 5%, treat it as an imminent control-plane
+  outage** and get the operator. etcd and Longhorn are both on it.
+- **Never delete anything under `/data/longhorn` or a Postgres `pg_wal` directory by hand.** Both are
+  irreversible. Free space elsewhere, or fix the cause.
+- **Do not `kubectl drain`** the node to move load off it — this cluster does not drain.
+- If the cause is a CNPG archive failure, this runbook is the wrong one: go to
+  `/runbooks/cnpg-archiving`, which is the actual fix.
+
+---
+Source of truth: `runbooks/node-disk.md`. Index: `/runbooks/`
+Records: `DECISIONS-LOG.md` "Disk/drive incident + k3s relocated to NVMe (2026-08-22)";
+`platform/cnpg/alerts.yaml` header (the WAL disk-full clock).

runbooks/node-down.md

@@ -0,0 +1,156 @@
+# node-down — A node's node-exporter is unscrapable (the node may be gone)
+
+**Alert:** `NodeExporterDown` (critical, 5m)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes`
+**Expression:** `up{job="node-exporter"} == 0`
+
+## What this means here
+
+Prometheus has not scraped node-exporter on that instance for 5 minutes. The alert is worded
+carefully — it says the *exporter* is down — because on this cluster the two cases look identical
+from Prometheus and have opposite urgency:
+
+- **The node is genuinely gone** (powered off, kernel panic, network loss).
+- **The node is up and the kubelet stopped posting status.** This is what actually happened on
+  **2026-09-07 at 06:05:21Z**: `k3s-prod-1` went NotReady with "Kubelet stopped posting node status"
+  while the host was up, pingable and SSH-able. Root cause was disk-I/O starvation of the embedded
+  etcd, not a dead machine. If this alert is firing alongside `node-io`, read `/runbooks/node-io`
+  first — that is the cause and this is the symptom.
+
+**Why it is critical rather than a warning.** All three nodes are control-plane + etcd members
+(`k3s-prod-1` .10, `k3s-prod-2` .11, `k3s-prod-3` .12, kube-vip API VIP `192.168.2.9`). Losing one
+leaves a 2-member etcd with no fault tolerance: the next loss ends quorum and the API server with it.
+Losing a node also degrades every Longhorn volume that had a replica there, and every degraded volume
+starts rebuilding — which loads the same disks etcd fsyncs to. One node down is not a routine event
+on a 3-node cluster; it is one step from an outage.
+
+Two node-specific roles matter when you read which node it is:
+- `k3s-prod-3` is the **sole VPN egress gateway**. Cilium OSS has **no egress-gateway failover**, so
+  while that node is down, VPN-classed egress is an outage, not a degradation. See `/runbooks/vpn-down`.
+- `k3s-prod-2` is the cluster **founder** (`[k3s_prod_primary]` in `inventory/hosts.ini`). Do not
+  re-provision it from the inventory without reading `/runbooks/node-io` — re-provisioning a node
+  still listed as primary founds a **second cluster**.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+Is the node actually NotReady, or is only the exporter unscrapable?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get nodes -o wide; echo; kubectl --context k3s-prod -n monitoring get pods -l app.kubernetes.io/name=prometheus-node-exporter -o wide'
+```
+
+Is it the API server or the node? (If this hangs, you have a bigger problem than one node.)
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get --raw /readyz?verbose 2>&1 | tail -20'
+```
+
+Is the host alive at all? Prod is directly on `192.168.2.0/24` — no NAT hop:
+
+```
+ping -c3 192.168.2.10 ; ssh k3s-prod-1 'uptime; systemctl is-active k3s; cat /proc/pressure/io'
+```
+
+The three lines that distinguish "etcd starved" from "machine dead":
+
+```
+ssh k3s-prod-1 'journalctl -u k3s --since "30 min ago" | grep -Ei "fdatasync|apply request took|too many requests|Kubelet stopped|leader"'
+```
+
+Etcd membership and leader — do this before touching anything:
+
+```
+ssh k3s-prod-2 'curl -s http://127.0.0.1:2381/metrics | grep -E "etcd_server_is_leader|etcd_server_has_leader"'
+```
+
+What did the node take with it?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
+     -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy; \
+   kubectl --context k3s-prod get pods -A --field-selector spec.nodeName=k3s-prod-1 -o wide | head -30'
+```
+
+## Likely causes, most likely first
+
+### 1. Disk I/O starvation wedged the kubelet (host still up)
+The signature is `slow fdatasync took=3.56s` / `apply request took too long took=8.30s` /
+`etcdserver: too many requests` in `journalctl -u k3s`, with `/proc/pressure/io` full avg300 well
+above 1. On 2026-09-07 node 1 read **11.36** against 0.75 on node 2 and 1.29 on node 3.
+Remediation: follow `/runbooks/node-io`. Recovery that day required a manual
+`systemctl restart k3s` on the node (operator, on-host), after which 18 Longhorn volumes rebuilt and
+pressure only fell when the rebuilds finished.
+
+### 2. The node rebooted (planned or otherwise)
+Check `uptime`. A reboot is not harmless here: **every node reboot degrades all ~18 Longhorn volumes
+and triggers a full rebuild**. If a reboot is intentional, roll one node at a time and wait for
+`robustness=healthy` on every volume before the next, and freeze rebuilds first (see
+`/runbooks/longhorn-degraded`). Note two hardware quirks recorded in `infra/PROD-PROVISIONING.md`
+§0b: `k3s-prod-2` **hangs on a warm reboot while staying powered**, and all three nodes carry the EFI
+memattr kernel bug that once presented `k3s-prod-1` as having 2.6 GiB usable of 16 GB (needs kernel
+≥ 6.13).
+
+### 3. node-exporter itself is unhealthy, node fine
+If `kubectl get nodes` says Ready and the pod is CrashLoopBackOff or Pending, this is a monitoring
+fault, not a node fault. Downgrade urgency, but do not ignore it — while that exporter is down you
+are blind to `node-io`, `node-memory`, `node-disk` and `node-cpu` on that node, which is exactly the
+blindness that let 2026-09-07 arrive as a surprise. Related: `/runbooks/targets-down`.
+
+### 4. Network loss to the node
+`ping` fails, SSH fails. Check the switch and the node's `eno1` (prod-3's
+`/etc/network/interfaces` also holds a commented-out wireless stanza — inert, but worth scrubbing).
+A node reachable by ping but not by the API is usually Cilium, not the network: heavy sustained node
+load has been observed to degrade the Cilium datapath into intermittent LB-VIP unreachability,
+cleared by a restart.
+
+### 5. Hardware
+`k3s-prod-1`'s original 2011 SSD had 61,440 reallocated sectors and was replaced 2026-09-07. Check
+SMART (`/runbooks/node-io` has the command). Hardware is an operator job.
+
+## When to escalate or stop
+
+- **STOP if two nodes are affected.** Two of three etcd members down means no quorum, no API server,
+  and no GitOps. Nothing in this runbook applies; get the operator.
+- **Do not reboot, drain or cordon a second node** while any Longhorn volume is not `healthy`. This
+  cluster deliberately does not `kubectl drain` — the auto-drain of system-upgrade-controller was
+  rejected precisely because it fights slow-disk Longhorn.
+- **Do not delete the Node object** to "clean up". That triggers replica deletion in Longhorn.
+- Escalate to the operator for: a power-cycle, a disk replacement, or re-provisioning a node (the
+  inventory's `[k3s_prod_primary]` trap above makes re-provisioning a two-person job).
+- If the node returns but volumes stay degraded for more than an hour, go to
+  `/runbooks/longhorn-degraded`.
+
+---
+Source of truth: `runbooks/node-down.md` in `home-network-k3s-impl`. Index: `/runbooks/`
+Incident record: `DECISIONS-LOG.md` "2026-09-07 — k3s-prod-1 NotReady"; `infra/PROD-PROVISIONING.md` §0b/§0c.

runbooks/node-io.md

@@ -0,0 +1,183 @@
+# node-io — Disk I/O pressure / starvation on a node
+
+**Alerts:** `NodeIOPressureHigh` (warning, 10m) and `NodeIOPressureCritical` (critical, 5m)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes`
+**Expression:** `rate(node_pressure_io_stalled_seconds_total[5m]) > 0.05` (warning) / `> 0.10` (critical)
+
+## What this means here
+
+PSI **full** for I/O: the fraction of wall-clock time in which *every* runnable task on the node was
+blocked on disk, i.e. the machine did no useful work. It is not "the disk is busy" — it is "the node
+is stalled".
+
+This alert exists because of a real outage it would have caught and could not. On **2026-09-07 at
+06:05:21Z, `k3s-prod-1` went NotReady** ("Kubelet stopped posting node status") while the host stayed
+up, pingable and SSH-able. The k3s journal showed embedded etcd starved of disk:
+
+```
+slow fdatasync                took=3.56s / 6.71s / 4.95s / 3.40s   (etcd expects 1s)
+apply request took too long   took=8.30s                           (expects 100ms)
+etcdserver: too many requests
+Failed to check local etcd status for learner management: context deadline exceeded
+```
+
+`/proc/pressure/io` **full avg300** at the time: `k3s-prod-1` **11.36**, `k3s-prod-2` (PM9A1 NVMe)
+**0.75**, `k3s-prod-3` (WD Blue) **1.29**, against a ~1% steady state. The stack alerted on CPU-PSI
+and memory and had **no I/O alert at all**, so the first signal the operator got was a node already
+out of the cluster. The thresholds here are those measurements, not guesses: 0.05 is 5x steady
+state, 0.10 is the level node 1 actually held while etcd stalled.
+
+**Why this is self-reinforcing, and why it is urgent.** On every node the etcd WAL (`/data/k3s`) and
+all Longhorn replicas (`/data/longhorn`) sit on **one partition** — `node_baseline_etcd_source:
+shared-data-partition`, not a dedicated device, and §2's exit condition in
+`infra/PROD-PROVISIONING.md` is still **open**. So: node stalls → drops out → its volumes go degraded
+→ Longhorn rebuilds them → more writes to the same spindle etcd fsyncs to → worse. Recovery on
+2026-09-07 needed a manual `systemctl restart k3s`, and I/O pressure only fell when the 18 rebuilds
+finished.
+
+**There is no etcd fsync metric to check.** `etcd_disk_wal_fsync_duration_seconds` is not collected —
+`kubeEtcd` is deliberately disabled in `platform/monitoring/kube-prometheus-stack.yaml` for k3s. PSI
+is the only signal that exists. Do not go looking for the etcd dashboard; it is not there.
+
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+Which node, and how bad:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get nodes -o wide'
+```
+
+Read PSI and per-device queue depth on the node itself (SSH direct — prod is on
+`192.168.2.0/24` with no NAT hop; `.10` = prod-1, `.11` = prod-2, `.12` = prod-3):
+
+```
+ssh k3s-prod-1 'cat /proc/pressure/io; echo; iostat -x 1 3; echo; lsblk -o NAME,MOUNTPOINT,ROTA'
+```
+
+Is etcd already stalling? This is the line that decides urgency:
+
+```
+ssh k3s-prod-1 'journalctl -u k3s --since "30 min ago" | grep -Ei "fdatasync|apply request took|too many requests|leader"'
+```
+
+Is Longhorn rebuilding (the usual writer)?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
+     -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,NODE:.status.currentNodeID | grep -v healthy'
+```
+
+Is the disk itself dying? (This is what it was in September.)
+
+```
+ssh k3s-prod-1 'sudo smartctl -a /dev/sda | grep -Ei "Reallocated|Pending|Uncorrectable|Wear|Power_On_Hours|Model"'
+```
+
+## Likely causes, most likely first
+
+### 1. A Longhorn replica rebuild storm (most common by far)
+Every node reboot degrades all ~18 volumes and triggers a full rebuild. That is the normal cost of
+Longhorn on these disks, and it is why reboots are rolled **one node at a time, waiting for
+`robustness=healthy` on every volume before the next**.
+
+Remediation — freeze rebuilds while the node recovers (operator, mutating):
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n longhorn-system patch settings.longhorn.io \
+     concurrent-replica-rebuild-per-node-limit --type=merge -p "{\"value\":\"0\"}"'
+```
+
+Thaw to `"1"` — never higher — once all volumes read `healthy`. Use `0` when replicas will
+**reattach** (an in-place bounce); use `1` when they are genuinely gone (an OS re-provision).
+`concurrentReplicaRebuildPerNodeLimit: 1` was **already in force during the 2026-09-07 outage and
+held** — do not "fix" this by setting a knob that is already set.
+
+### 2. The node's disk is failing
+`k3s-prod-1`'s original 2011 `C400-MTFDDAC512MAM` had **61,440 reallocated sectors**, 31% endurance
+consumed and no power-loss protection. The operator replaced it with a Samsung 850 EVO on 2026-09-07
+and the same 18-replica rebuild measured **3.79** instead of 11.36 — ~3x better, node stayed Ready.
+The earlier dev-era drive (Seagate `ST4000DM000`: 29,408 reallocated, 448 pending, 448 offline
+uncorrectable) drove etcd to 48% iowait. If SMART shows reallocated or pending sectors climbing,
+**the fix is hardware**; nothing in software recovers it.
+
+When replacing a disk, two traps recorded from that rebuild:
+- Longhorn refuses the new disk with `DiskFilesystemChanged` ("record diskUUID doesn't match the one
+  on the disk"). Remedy: **disable → remove the disk entry → re-add, in that order** (the validating
+  webhook enforces it).
+- `inventory/hosts.ini` decides `--cluster-init` via `[k3s_prod_primary]`. Re-provisioning a node
+  that is still listed there **founds a second cluster**. The founder now sits on `k3s-prod-2`.
+
+### 3. A Velero backup or restore running against the same spindle
+Velero's daily runs at `0 3 * * *` and reads Longhorn PVCs through kopia. Check
+`backups.velero.io` (fully qualified — see above) for one in flight, and the Longhorn limits
+`concurrentVolumeBackupRestorePerNodeLimit` / `snapshotHeavyTaskConcurrentLimit`, both pinned to 1
+for exactly this reason.
+
+### 4. A CDI image import or a TCG-emulated VM
+A CDI import writes a whole disk image; heavy sustained node load from TCG-emulated VMs has also been
+observed to degrade the Cilium datapath into intermittent LB-VIP unreachability (cleared by a
+restart). Check `kubectl --context k3s-prod get vmi -A` and importer pods in `cdi`.
+
+### 5. Structural: etcd and Longhorn share one partition
+This is the standing cause underneath the other four, and it is open work. `roles/etcd_io_priority`
++ `playbooks/etcd-io-priority.yml` apply BFQ on the shared disk, `nr_requests` 64→32 and
+`IOWeight=10000` on `system.slice` and `k3s.service`, reading every value back. **It applies nothing
+until an operator runs it** — run `k3s-prod-3` first, with `--limit` and `--list-hosts`.
+
+Four mechanisms are **proven inert on these nodes** — do not re-try them:
+`IOSchedulingClass=realtime` on `k3s.service` is a footgun (ioprio is inherited across fork/exec and
+containerd is a child of the k3s server process, so realtime for k3s means realtime for Longhorn
+too); `IOSchedulingPriority` is a no-op under `mq-deadline`; `io.latency` is unavailable
+(`CONFIG_BLK_CGROUP_IOLATENCY` is not built in Debian trixie); a bare `IOWeight=` drop-in is
+decoration (blk-iocost compiled-but-disabled, BFQ not loaded). And no cgroup or scheduler control
+arbitrates the **jbd2 journal** that etcd's `fdatasync` actually waits on.
+
+## When to escalate or stop
+
+- **Stop and get the operator** if `NodeIOPressureCritical` is firing *and* the journal shows
+  `slow fdatasync` or `apply request took too long`. The node is minutes from leaving the cluster.
+  With one of three etcd members already gone, a second loss ends quorum.
+- **Do not reboot or drain a second node** while any volume is not `healthy`. That is how a
+  single-node problem becomes a data problem on a 3-replica cluster.
+- **Do not `kubectl drain`.** This cluster does not drain: the posture is deliberate (the upgrade
+  plan rejects system-upgrade-controller because "its auto-drain fights this cluster's slow-HDD
+  Longhorn"), and the node-loss conformance test evicts only its own writer pod.
+- Hand over to the operator for anything physical (disk replacement) or for the Ansible role — both
+  are outside what can be done from git.
+
+---
+Source of truth: `runbooks/node-io.md` in `home-network-k3s-impl`. Index of all runbooks: `/runbooks/`
+Incident record: `DECISIONS-LOG.md` "2026-09-07 — k3s-prod-1 NotReady: etcd starved of disk I/O by
+Longhorn", and `infra/PROD-PROVISIONING.md` §0c / §2.

runbooks/node-memory.md

@@ -0,0 +1,123 @@
+# node-memory — Memory pressure high on a node
+
+**Alert:** `NodeMemoryPressureHigh` (warning, 10m)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes`
+**Expression:** `(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 0.90`
+
+## What this means here
+
+Available memory has been under 10% for ten minutes. These are HP EliteDesk 800 G5 Minis with
+~22 GiB usable, running the whole platform plus KubeVirt VMs on three nodes, so the headroom is
+genuinely thin and this alert is a capacity signal rather than a bug signal most of the time.
+
+Two cluster-specific facts change how you read it:
+
+**`SwapTotal == 0` is not evidence swap is off.** On 2026-09-07 all three prod nodes came back from a
+reboot with **977 MiB of swap ACTIVE** (`/dev/sda4` on prod-1 and prod-3, `/dev/nvme0n1p4` on prod-2)
+while the Ansible role's commented-out fstab line reported success.
+`systemd-gpt-auto-generator` claims any GPT partition typed
+`0657fd6d-a4ab-43c4-84e5-0933c84b4f4f` straight from the partition table and synthesises a `.swap`
+unit into `/run/systemd/generator.late/` — it never reads `/etc/fstab`. The fix masks `swap.target`
+(the single choke point for gpt-auto, fstab-generator and cryptsetup-generator). If a node is
+swapping, the kubelet's memory accounting is lying to you and eviction will misbehave. Check
+explicitly with `swapon --show`, never `SwapTotal`.
+
+**There is a kernel memory-reporting bug on all three nodes.** `infra/PROD-PROVISIONING.md` §0b
+records an EFI memattr defect that once presented `k3s-prod-1` as having **2.6 GiB usable of 16 GB**.
+It needs kernel ≥ 6.13 and all three nodes are exposed (prod-3 publishes `MEMATTR=0xbeece018` too).
+If `MemTotal` looks absurd rather than merely tight, suspect this before suspecting a leak.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod top nodes; echo; kubectl --context k3s-prod get nodes -o custom-columns=NAME:.metadata.name,MEM:.status.capacity.memory,PRESSURE:.status.conditions[?\(@.type==\"MemoryPressure\"\)].status'
+```
+
+Who is using it — and is swap actually off?
+
+```
+ssh k3s-prod-1 'free -h; echo; swapon --show; echo "(empty output above = swap really off)"; echo; cat /proc/pressure/memory; echo; ps -eo pid,rss,comm --sort=-rss | head -15'
+```
+
+Biggest consumers in the cluster, and anything that has been OOM-killed:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod top pods -A --sort-by=memory | head -20; echo; \
+   kubectl --context k3s-prod get events -A --field-selector reason=OOMKilling --sort-by=.lastTimestamp | tail -20'
+```
+
+## Likely causes, most likely first
+
+### 1. A monitoring component grew past its limit
+This is the one with real history. When kube-prometheus-stack went 65 → 88 it shipped Grafana 13 and
+**Grafana OOM-looped** — "the hidden blocker that stalled the whole kps upgrade and the platform
+tier". Limits went 256 → 512 → **768Mi** for Grafana and Prometheus 2 → **3Gi** (commit `b2b1aa5`).
+Prod keeps 30d retention / 45GiB retentionSize on a 50Gi PVC, so Prometheus is legitimately the
+largest single consumer. Remediation: raise the limit in
+`platform/monitoring/kube-prometheus-stack.yaml` and push; do not restart it in a loop.
+
+### 2. A KubeVirt VM, especially a TCG-emulated one
+VMs are the largest discretionary consumer. `useEmulation: false` is set deliberately (on v1.9
+`true` silently forces TCG), but leaked test namespaces running emulated VMs have caused API blips
+before and were deleted. Check `kubectl --context k3s-prod get vmi -A` and the `virt-launcher-*` pods'
+memory.
+
+### 3. Swap silently back on after a reboot
+See above. Verify with `swapon --show` and `systemctl is-enabled swap.target` (should be `masked`).
+`node_health` now reports `memory.swap_persistence` for this. The fix is the Ansible role; an
+operator runs it.
+
+### 4. Genuine capacity exhaustion
+If nothing is anomalous, the node is simply full. The node vCPU count was already raised 6 → 8 for
+CPU pressure; memory is next. A bpg/OpenTofu memory change **reboots the VM**, so roll **one node at
+a time and wait for Longhorn `robustness=healthy` between each** — the same discipline as every other
+reboot here.
+
+### 5. A leak in an app-tier workload
+Nextcloud, Plex and the *arr stack are the usual suspects. Compare `top pods` against the previous
+day in Grafana rather than guessing.
+
+## When to escalate or stop
+
+- **Escalate immediately if this is firing together with `node-io` or `node-down`** on the same node.
+  Memory pressure plus I/O pressure on an etcd member is the shape of the 2026-09-07 outage.
+- **Do not add swap** to relieve this. Kubernetes does not want it here and the platform has already
+  been bitten by swap re-enabling itself.
+- **Do not evict or drain** to free memory: this cluster does not drain (Longhorn rebuild cost). Kill
+  the specific offending pod instead, and only if you know what it is.
+- Hand to the operator for anything requiring a VM resize or an Ansible run.
+
+---
+Source of truth: `runbooks/node-memory.md`. Index: `/runbooks/`
+Records: `DECISIONS-LOG.md` (Grafana OOM, kps 65→88); commits `0e92d25`/`9d00a3f` (swap persistence);
+`infra/PROD-PROVISIONING.md` §0b (EFI memattr).

runbooks/pvc-capacity.md

@@ -0,0 +1,144 @@
+# pvc-capacity — A PersistentVolumeClaim is filling up, nearly full, or predicted to fill
+
+**Alerts:** `PersistentVolumeFillingUp` (warning, 15m), `PersistentVolumeAlmostFull` (critical, 15m),
+`PersistentVolumeFillingUpPredicted` (warning, 1h)
+**File:** `platform/monitoring/pvc-capacity-alerts.yaml`, group `platform-health.pvc-capacity`
+**Expressions:** `< 0.15` free / `< 0.05` free / `< 0.50` free **and**
+`predict_linear(kubelet_volume_stats_available_bytes[24h], 7*24*3600) < 0`, each `and
+kubelet_volume_stats_inodes{job="kubelet"} > 0`
+
+## What this means here
+
+These are the PVC fill alerts for everything **outside** the `monitoring` namespace
+(`/runbooks/monitoring-capacity` owns that half). Before this file existed, **no PVC outside
+`monitoring` had any fill alert at all**. Unlike the Longhorn rule group, these work: they are built
+on `kubelet_volume_stats_*`, which is live (56 series on prod).
+
+Three design facts you need in order to read the labels correctly:
+
+- **Every rule pins `job="kubelet"`.** `kubelet_volume_stats_*` is published by *two* jobs with
+  different label shapes: `job="kubelet"` carries the PVC's real namespace, while `job="apiserver"`
+  reports `namespace="default"` with the real one in `exported_namespace`. Without the pin,
+  `namespace!="monitoring"` double-fires on monitoring's own PVCs and names the wrong namespace.
+- **Every rule carries `and kubelet_volume_stats_inodes > 0`.** Block-mode PVCs — KubeVirt VM disks
+  on `longhorn-vm` — report available=0, used=0, inodes=0 against a real capacity. Without the guard,
+  `unifi/unifi-rootdisk` (40 GiB Block) would fire both warning **and** critical permanently. If you
+  are ever tempted to remove that clause, this is why it is there.
+- **The predicted rule is the point of the exercise.** A level threshold is useless for a slow leak on
+  a mostly-empty volume: prod's `keycloak-db` sat at **93% free** while leaking ~6.8 MiB/h of
+  unrecyclable WAL, so `<15%` was ~29 days away and would have arrived as a surprise. The `< 0.50`
+  gate keeps it quiet on volumes that are not plausibly about to fill, and the 24h fit / 7d horizon is
+  chosen so a daily backup or log rotation does not make it flap.
+
+**If the PVC name matches `*-db-N`, read `/runbooks/cnpg-archiving` before doing anything else.**
+That is by far the most likely cause on this cluster and the one where the obvious action is fatal.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+Which PVC, how full, on which node:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
+   curl -s -G --data-urlencode "query=topk(10, 1 - kubelet_volume_stats_available_bytes{job=\"kubelet\"} / kubelet_volume_stats_capacity_bytes{job=\"kubelet\"})" localhost:19090/api/v1/query | python3 -m json.tool | grep -E "persistentvolumeclaim|namespace|\"1\"" '
+```
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get pvc -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SIZE:.spec.resources.requests.storage,SC:.spec.storageClassName,STATUS:.status.phase'
+```
+
+Is it a CNPG data volume? (Fully-qualified `clusters.postgresql.cnpg.io` — the short name `cluster` is
+ambiguous on clusters carrying more than one `Cluster` kind.)
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A'
+```
+
+If yes, the one question that decides everything — is WAL archiving working?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
+     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,LASTBACKUP:.status.conditions[?\(@.type==\"LastBackupSucceeded\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint'
+```
+
+What is actually consuming the space, from inside the pod:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n keycloak exec keycloak-db-2 -c postgres -- bash -c "df -h /var/lib/postgresql/data; du -sh /var/lib/postgresql/data/pgdata/pg_wal; ls /var/lib/postgresql/data/pgdata/pg_wal/archive_status | wc -l"'
+```
+
+## Likely causes, most likely first
+
+### 1. A CNPG database is not recycling WAL
+The dominant cause. With `archive_mode=on` and a failing `archive_command`, Postgres **refuses to
+recycle** unarchived WAL segments, so `pg_wal` grows until the data PVC is full and the postmaster
+**PANICs** — taking the database, and for `keycloak-db` all of SSO, down. `pg_wal` shares the data
+PVC on every Cluster here (no `walStorage:` is configured), which is why a *backup* fault presents as
+a *capacity* alert. This actually happened: prod ran from bootstrap to 2026-09-07 with
+`endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`, all three Clusters at `ContinuousArchiving=False` with
+`barman-cloud-wal-archive: exit status 4`, unnoticed for 38+ hours.
+**Remediation: fix archiving (`/runbooks/cnpg-archiving`). Do NOT delete WAL by hand — that destroys
+the database. Do NOT expand the PVC and call it done; that buys days.**
+
+### 2. Log or cache growth in an app-tier workload
+Nextcloud, the *arr apps, Gitea. Prune inside the app, or expand the PVC in git (every Longhorn class
+has `allowVolumeExpansion: true`).
+
+### 3. Media on the wrong storage class
+Media is meant to be static NFS PVs from `192.168.2.102:/mnt/Storage/plex` (8.32 TiB measured), not
+Longhorn. A media volume on `longhorn` is a misconfiguration; fix the class, do not grow the disk.
+
+### 4. A Velero or CDI staging write
+Transient. CDI uses `longhorn-scratch` deliberately (single replica) because the 3-replica default
+failed the UniFi 60Gi import on insufficient storage.
+
+### 5. The predicted rule caught a genuine slow leak
+This is the rule working as designed and giving you a week. Find what is growing before the level
+thresholds arrive.
+
+## When to escalate or stop
+
+- **`PersistentVolumeAlmostFull` on a `*-db-N` volume is an imminent database outage.** Get the
+  operator. A Postgres data volume at under 5% free will PANIC the postmaster.
+- **Never delete WAL segments, and never delete anything under a Postgres data directory.**
+- **Do not delete the PVC** to reclaim space.
+- If `unifi-rootdisk` or another Block-mode volume appears in these alerts, the inodes guard has been
+  removed or broken — that is a monitoring bug, not a storage incident.
+- Escalate for PVC expansion beyond what the node can hold (see `/runbooks/longhorn-capacity`).
+
+---
+Source of truth: `runbooks/pvc-capacity.md`. Index: `/runbooks/`
+Records: `platform/monitoring/pvc-capacity-alerts.yaml` (the two-job and inodes findings);
+`platform/cnpg/alerts.yaml` header (the 2026-09-06 archiving incident and the measured fill rate).

runbooks/serving/deployment.yaml

@@ -0,0 +1,151 @@
+---
+# The runbook static server. Deliberately the dumbest thing that can work.
+#
+# WHY A POD AT ALL, AND NOT A REDIRECT TO GITEA. The obvious cheaper option is an HTTPRoute whose
+# only job is a RequestRedirect into the in-cluster Gitea web UI, which already renders these files.
+# It was rejected for three reasons, in order of weight:
+#   1. DEPENDENCY DIRECTION. A runbook is read when the platform is broken. Gitea is an app-tier
+#      workload on Longhorn; its own DR path and several of these runbooks' subjects sit UNDER it.
+#      `runbooks/longhorn-faulted` must be readable while Longhorn is faulted, and
+#      `runbooks/keycloak-db` while a CNPG database is down. Pointing the operator at Gitea makes
+#      the documentation fail in exactly the conditions that produce the alert.
+#   2. NO SUFFIX REWRITE EXISTS. Gateway API's RequestRedirect path modifier offers ReplaceFullPath
+#      and ReplacePrefixMatch only. There is no "append .md", so mapping 34 extensionless names onto
+#      34 `.md` blobs needs 34 hand-maintained ReplaceFullPath rules — MORE surface than this pod,
+#      and a silent 404 for every runbook someone forgets to add a rule for.
+#   3. A 302 is not a document. The annotation's contract is that the URL returns the runbook; a
+#      redirect chain through a second app's auth and rendering is a worse contract.
+# This pod reads only from a ConfigMap projected out of git. It depends on the CNI, the kubelet and
+# the Gateway, and on nothing else in the platform.
+#
+# WHY NOT THE SERVICE CATALOG (catalog/*.yaml + render.py). The catalog is the idiomatic onboarding
+# surface for a SERVICE, and it was the first thing checked — but its schema cannot express this
+# route. `ingressRoutes[].host` is pattern-bound to `^[a-z0-9-]+\.(\$\{BASE_DOMAIN\}|home\.keithjackson\.dev)$`:
+# a single DNS label plus the apex, so the BARE apex is unrepresentable, and there is no `path` field
+# at all, so a path-prefix route cannot be authored. Bending the schema for one docs server would be
+# the tail wagging the dog. The repo's OTHER established pattern is exactly right instead: a small
+# hand-authored platform component reconciled by its own non-gating Flux Kustomization — what
+# platform/homepage does for the apex `/` prefix, which this mirrors file for file.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: runbooks
+  namespace: runbooks
+  labels:
+    app.kubernetes.io/name: runbooks
+spec:
+  # TWO replicas with anti-affinity, unlike homepage's one. This is the only workload in the
+  # cluster whose whole purpose is to be available during an incident, and a single replica is
+  # unavailable for the minutes a node reboot or a drain takes — which is when it is wanted. It
+  # costs 2 x 16Mi.
+  replicas: 2
+  revisionHistoryLimit: 3
+  strategy:
+    type: RollingUpdate
+    rollingUpdate:
+      maxUnavailable: 0
+      maxSurge: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: runbooks
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: runbooks
+    spec:
+      automountServiceAccountToken: false
+      # Spread across nodes so losing one node never takes the runbooks with it. `preferred`, not
+      # `required`: on a 3-node cluster with one node cordoned for the very incident being debugged,
+      # a hard rule would leave a replica Pending instead of running somewhere.
+      affinity:
+        podAntiAffinity:
+          preferredDuringSchedulingIgnoredDuringExecution:
+            - weight: 100
+              podAffinityTerm:
+                topologyKey: kubernetes.io/hostname
+                labelSelector:
+                  matchLabels:
+                    app.kubernetes.io/name: runbooks
+      # Tolerate a node that is under memory/disk pressure or unreachable for a while. An operator
+      # reading `runbooks/node-disk` is by definition dealing with a node under pressure, and the
+      # default 300s unreachable toleration is long enough; these exist so eviction pressure does
+      # not silently remove the last replica while the cluster is sick.
+      tolerations:
+        - key: node.kubernetes.io/unreachable
+          operator: Exists
+          effect: NoExecute
+          tolerationSeconds: 900
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 101            # `nginx` in nginxinc/nginx-unprivileged
+        runAsGroup: 101
+        fsGroup: 101
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: nginx
+          # Renovate maintains this pin. nginx-unprivileged (not nginx) so the container can honour
+          # runAsNonRoot: the stock nginx image's entrypoint needs root to drop privileges and to
+          # bind :80. `stable-alpine` branch, pinned exactly.
+          image: nginxinc/nginx-unprivileged:1.30.4-alpine
+          imagePullPolicy: IfNotPresent
+          ports:
+            - name: http
+              containerPort: 8080
+              protocol: TCP
+          resources:
+            requests:
+              cpu: 10m
+              memory: 16Mi
+            limits:
+              memory: 64Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop:
+                - ALL
+            # Achievable here, unlike homepage/headlamp: nginx only needs /tmp (its pid file, on the
+            # unprivileged image) and /var/cache/nginx writable, and both are emptyDirs below.
+            readOnlyRootFilesystem: true
+          livenessProbe:
+            httpGet: { path: /healthz, port: http }
+            initialDelaySeconds: 3
+            periodSeconds: 15
+          readinessProbe:
+            # Probe a REAL runbook, not just /healthz. nginx answers /healthz from its own config
+            # even if the ConfigMap volume failed to project, so /healthz alone would report a pod
+            # that serves 404 for every runbook as Ready. `node-down` is the canonical alert link
+            # (NodeExporterDown, platform/monitoring/alerts.yaml), so if this path 404s the pod is
+            # useless and must not take traffic.
+            httpGet: { path: /runbooks/node-down, port: http }
+            initialDelaySeconds: 3
+            periodSeconds: 15
+          volumeMounts:
+            - name: nginx-config
+              mountPath: /etc/nginx/conf.d/default.conf
+              subPath: default.conf
+              readOnly: true
+            # The runbooks themselves, one extensionless file per key. Whole-volume mount (NOT
+            # subPath): subPath mounts do not update when the ConfigMap changes, and here the
+            # ConfigMap name carries kustomize's content hash so a content change rolls the pod
+            # anyway — the whole-volume mount just means a stale projection is impossible.
+            - name: runbooks
+              mountPath: /usr/share/nginx/html/runbooks
+              readOnly: true
+            - name: tmp
+              mountPath: /tmp
+            - name: cache
+              mountPath: /var/cache/nginx
+      volumes:
+        - name: nginx-config
+          configMap:
+            name: runbooks-nginx
+        - name: runbooks
+          configMap:
+            # Generated by ../kustomization.yaml's configMapGenerator from runbooks/*.md. kustomize
+            # rewrites this reference to the hashed name, so the pods roll whenever a runbook changes.
+            name: runbooks-content
+        - name: tmp
+          emptyDir: {}
+        - name: cache
+          emptyDir: {}

runbooks/serving/httproute.yaml

@@ -0,0 +1,87 @@
+---
+# North-south ingress for the runbooks at https://<apex>/runbooks/ — the exact prefix the alert
+# annotations already point at (`runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"`, 42 of them
+# across platform/monitoring/alerts.yaml, platform/monitoring/pvc-capacity-alerts.yaml,
+# platform/cnpg/alerts.yaml and platform/kubevirt/alerts.yaml). Nothing about those annotations
+# changes; this route is what finally makes them resolve.
+#
+# NO `hostnames:` — DELIBERATE, and the reason this manifest needs no Flux variable substitution.
+# An HTTPRoute with no hostnames inherits the hostname of every listener it attaches to, and this
+# route attaches to exactly one: `https-apex`, whose hostname is `${BASE_DOMAIN}`
+# (platform/gateway/gateway.yaml). So the route is correct on dev and on prod with no
+# `postBuild.substituteFrom` anywhere in this tier — which matters a great deal here, because the
+# SAME Flux Kustomization renders the runbook markdown into a ConfigMap. With substitution enabled,
+# Flux's envsubst would run over the runbook prose and silently EMPTY every `${...}` in every shell
+# example a runbook contains (it fails open — DECISIONS-LOG 2026-09-01, the Cilium k8sServiceHost
+# incident). Declaring no hostname removes the last reason to substitute, so the docs ship verbatim.
+# See clusters/base/runbooks.yaml.
+#
+# PATH PRECEDENCE AGAINST HOMEPAGE. platform/homepage/httproute.yaml already binds `https-apex` with
+# PathPrefix `/`. Gateway API resolves a conflict between rules on the same hostname by preferring
+# the LONGEST path match, so `/runbooks` wins over `/` for these requests and homepage keeps
+# everything else. The two routes coexist by specification, not by luck — and the live 200s recorded
+# in the commit prove it on this Gateway implementation (Cilium).
+#
+# ============================ AUTH: THIS ROUTE IS PUBLIC ==============================
+# auth-mode=none, deliberately, and it is the one route in this repo where that is the SAFE choice.
+#
+# THE CIRCULARITY. Every other exposed UI here declares forward-auth (oauth2-proxy -> Keycloak).
+# Put these pages behind that and `runbooks/keycloak-down` becomes unreadable precisely when
+# Keycloak is down; `runbooks/keycloak-db` unreadable when its CNPG database is down;
+# `runbooks/keycloak-client-secrets` unreadable during the exact failure mode that broke all 12
+# OIDC clients on 2026-09-06 while every health signal stayed green. An on-call who cannot log in
+# is the primary reader of this route. Gating documentation on the identity provider it documents
+# is a loop with no exit.
+#
+# WHAT IS ACTUALLY EXPOSED. Operational procedure: metric names, thresholds, kubectl invocations,
+# component topology. No credentials, no secrets, no tokens, no internal IPs beyond the LAN ranges
+# already written in the public spec repo. The content is the same text any reader of the git repo
+# can see. The real cost is reconnaissance value — it names what runs here and how it fails — and
+# that is accepted knowingly, against the certainty of a locked-out operator staring at a login page
+# during an outage. If that trade is ever revisited, the correct fix is NOT forward-auth on this
+# route; it is to keep it public and move sensitive specifics out of the runbooks.
+#
+# HOW forward-auth IS ENFORCED HERE — and why this route is genuinely different. Enforcement is NOT
+# a Gateway ext-authz attachment; it is PER-POD. Each forward-auth service runs an oauth2-proxy
+# sidecar, its Service publishes BOTH the app port and 4180, and the service's HTTPRoute targets
+# :4180 — verified on prod for all five that declare `auth: forward-auth` in the catalog (pihole,
+# jackett, qbittorrent, radarr, sonarr). Ingress traffic therefore always traverses oauth2-proxy,
+# which performs the Keycloak exchange and the group check. The app port stays on the Service so
+# in-cluster clients can talk service-to-service without SSO (that is how sonarr/radarr reach
+# qBittorrent's API), which is deliberate, not a leak: Cilium default-deny decides who may use it.
+#
+# So this route is NOT "no more open than its neighbours" — its neighbours are closed and this one is
+# open, on purpose, for the lockout reason argued above. It has no oauth2-proxy sidecar and targets
+# its app port directly. If a Gateway-level ext-authz attachment is ever introduced, this route must
+# be left OUT of it ON PURPOSE, and this comment is the record of that decision so nobody "fixes" it
+# later and re-closes the loop. (Headlamp is also excluded from forward-auth, for a different and
+# equally deliberate reason: it runs native Keycloak OIDC and forwards the logged-in user's id_token
+# to the apiserver for per-user RBAC, with clusterRoleBinding.create=false so its ServiceAccount can
+# do nothing on its own.)
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: runbooks
+  namespace: runbooks
+  labels:
+    app.kubernetes.io/name: runbooks
+  annotations:
+    homelab.keithjackson.dev/auth-mode: none
+    homelab.keithjackson.dev/auth-rationale: >-
+      Public on purpose. An operator locked out of Keycloak is this route's primary reader;
+      gating it behind Keycloak would make runbooks/keycloak-down unreadable exactly when it is
+      needed. Content is operational procedure only — no secrets. Do NOT add this route to the
+      Gateway ext-authz attachment when forward-auth enforcement lands.
+spec:
+  parentRefs:
+    - name: shared
+      namespace: gateway
+      sectionName: https-apex     # the apex listener; the wildcard `https` listener won't match it
+  rules:
+    - matches:
+        - path:
+            type: PathPrefix
+            value: /runbooks
+      backendRefs:
+        - name: runbooks
+          port: 80

runbooks/serving/namespace.yaml

@@ -0,0 +1,16 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: runbooks
+  labels:
+    # Serve the runbooks at https://<apex>/runbooks/<name> — the APEX host, on a `/runbooks` path
+    # prefix. The apex has its own `https-apex` listener on the shared Gateway
+    # (platform/gateway/gateway.yaml); this label opts the namespace in to attach HTTPRoutes to it
+    # (serving/httproute.yaml). Same opt-in as homepage, which owns the apex `/` prefix.
+    gateway-access: "shared"
+    # NOT homelab.keithjackson.dev/app-namespace=true. This is a PLATFORM component (like homepage and
+    # headlamp), so Kyverno does not generate the app default-deny baseline here and the app-scoped
+    # auth-mode / egress-class policies do not apply. The default-deny NetworkPolicy and the
+    # Gateway-only CiliumNetworkPolicy are shipped explicitly in serving/networkpolicy.yaml instead.
+    app.kubernetes.io/name: runbooks

runbooks/serving/networkpolicy.yaml

@@ -0,0 +1,57 @@
+---
+# Default-deny baseline (R-NET-01): deny all ingress + egress, allow only DNS. `runbooks` is a
+# PLATFORM namespace (no app-namespace label) so Kyverno does not auto-generate this — it is shipped
+# explicitly, defence in depth, same shape as the Kyverno-generated app baseline and as
+# platform/homepage/networkpolicy.yaml. Cilium enforces the UNION of this NetworkPolicy and the
+# CiliumNetworkPolicy below (allow if ANY policy allows).
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: default-deny
+  namespace: runbooks
+  labels:
+    homelab.keithjackson.dev/baseline: default-deny
+spec:
+  podSelector: {}
+  policyTypes:
+    - Ingress
+    - Egress
+  ingress: []
+  egress:
+    - to:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: kube-system
+      ports:
+        - protocol: UDP
+          port: 53
+        - protocol: TCP
+          port: 53
+---
+# Gateway ingress allow (R-NET-01/02): only the shared Gateway (the Cilium `ingress` entity) may
+# reach the runbook server, on container port 8080. The server makes NO outbound calls whatsoever —
+# every byte it serves comes from a ConfigMap projected by the kubelet — so the baseline's DNS-only
+# egress is already more than it needs and no egress CNP is added.
+#
+# CONSEQUENCE WORTH STATING: because egress is DNS-only, these pages cannot pull a stylesheet, a
+# font or a markdown renderer from anywhere. That is why the content is served as text/plain
+# (serving/nginx-config.yaml) rather than as HTML with a CDN dependency, and it is a feature: the
+# runbooks render with zero network beyond the request that fetched them.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: runbooks-allow-gateway-ingress
+  namespace: runbooks
+  labels:
+    app.kubernetes.io/name: runbooks
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: runbooks
+  ingress:
+    - fromEntities:
+        - ingress
+      toPorts:
+        - ports:
+            - port: "8080"
+              protocol: TCP

runbooks/serving/nginx-config.yaml

@@ -0,0 +1,81 @@
+---
+# nginx server config for the runbook static server. Hand-authored ConfigMap (same plain-manifest style
+# as platform/homepage), mounted over /etc/nginx/conf.d/default.conf.
+#
+# WHY THE URI NEEDS NO REWRITE. The alert annotations are `.../runbooks/<name>` with NO `.md` suffix, and
+# this repo must not change them (they are spread across four alerts.yaml files that other work is
+# editing). Rather than rewriting the URI, the ConfigMap generated from runbooks/*.md maps each file to an
+# EXTENSIONLESS key (`node-down=node-down.md` in ../kustomization.yaml) and is mounted at <root>/runbooks/.
+# So `/runbooks/node-down` is a plain static file lookup: no rewrite, no fallback chain, nothing to get
+# subtly wrong at 3am.
+#
+# CONTENT TYPE. An extensionless file has no MIME mapping, so nginx would serve it as
+# application/octet-stream and a browser would DOWNLOAD it instead of showing it. The empty `types {}`
+# block removes the mapping table entirely inside these locations and `default_type` forces
+# text/plain;charset=utf-8 — markdown read as plain text, which is what markdown is for, and which works
+# identically in a browser, in `curl`, and in a terminal with no JavaScript, no CDN and no egress. This
+# namespace has DNS-only egress, so a client-side markdown renderer fetched from a CDN could not load
+# anyway; plain text is the honest choice here, not a shortcut.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: runbooks-nginx
+  namespace: runbooks
+  labels:
+    app.kubernetes.io/name: runbooks
+data:
+  default.conf: |
+    # Replaces the image's stock conf.d/default.conf. Port 8080 because the base image is
+    # nginx-unprivileged (runs as uid 101 and cannot bind 80).
+    server {
+        listen       8080;
+        server_name  _;
+        root         /usr/share/nginx/html;
+
+        # Never leak the server version on an error page.
+        server_tokens off;
+        # Keep the 301 below relative so it works through the Gateway without knowing the public host.
+        absolute_redirect off;
+
+        # Probes. Kept OUT of the access log so a 15s kubelet probe does not bury the one request an
+        # operator actually made.
+        location = /healthz {
+            access_log off;
+            default_type text/plain;
+            return 200 "ok\n";
+        }
+
+        # /runbooks and /runbooks/ both land on the index (runbooks/README.md, mounted as the
+        # extensionless key `README`).
+        location = /runbooks {
+            return 301 /runbooks/;
+        }
+        location = /runbooks/ {
+            types { }
+            default_type "text/plain; charset=utf-8";
+            try_files /runbooks/README =404;
+        }
+
+        # The runbooks themselves. One static file per alert runbook_url.
+        location /runbooks/ {
+            types { }
+            default_type "text/plain; charset=utf-8";
+            # A ConfigMap volume is a tree of symlinks into ..data/; nginx must follow them.
+            disable_symlinks off;
+            try_files $uri =404;
+            add_header Cache-Control "public, max-age=300";
+            # An unknown name gets a 404 with a POINTER, not a blank page: the likeliest cause is an
+            # alert whose runbook was never written, and the reader needs to know where to look next.
+            error_page 404 = @no_such_runbook;
+        }
+        location @no_such_runbook {
+            default_type "text/plain; charset=utf-8";
+            return 404 "No runbook by that name.\n\nIndex of all runbooks: /runbooks/\nSource of truth: runbooks/<name>.md in the home-network-k3s-impl repo.\nIf an alert linked you here, its runbook_url has no matching runbooks/<name>.md, and the\nconformance check alert-runbooks-resolvable should have caught it before merge.\n";
+        }
+
+        # Nothing else is served from this pod. The apex `/` prefix belongs to homepage.
+        location / {
+            default_type "text/plain; charset=utf-8";
+            return 404 "This host serves /runbooks/ only.\n";
+        }
+    }

runbooks/serving/service.yaml

@@ -0,0 +1,19 @@
+---
+# ClusterIP fronting the runbook server. The shared Gateway routes the apex `/runbooks` prefix here
+# (serving/httproute.yaml); Service port 80 -> container port 8080.
+apiVersion: v1
+kind: Service
+metadata:
+  name: runbooks
+  namespace: runbooks
+  labels:
+    app.kubernetes.io/name: runbooks
+spec:
+  type: ClusterIP
+  selector:
+    app.kubernetes.io/name: runbooks
+  ports:
+    - name: http
+      port: 80
+      targetPort: http
+      protocol: TCP

runbooks/targets-down.md

@@ -0,0 +1,138 @@
+# targets-down — More than 10% of Prometheus scrape targets are down
+
+**Alert:** `PrometheusTargetsMissing` (warning, 15m)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.monitoring-self`
+**Expression:** `(count(up == 0) / count(up)) > 0.10`
+
+## What this means here
+
+A tenth of everything Prometheus is supposed to scrape is unreachable. This is a **blind-spot** alert:
+its purpose is to tell you that other alerts may be silent because their data is missing, not because
+the thing they watch is healthy.
+
+Live baseline on prod, measured 2026-09-13: **55 targets across 19 jobs** —
+`apiserver`, `cdi-prometheus-metrics`, `coredns`, `keycloak-discovery`, `keycloak-service`,
+`keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`, `kube-prometheus-stack-grafana`,
+`kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`, `kube-state-metrics`, `kubelet`,
+`kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`, `node-exporter`, `oauth2-proxy`,
+`semaphore-db/semaphore-db`, `velero`, `velero/node-agent`. So 10% is roughly **6 targets** — one
+node's worth of exporters, or one CNPG cluster's three pods.
+
+**The far more dangerous failure this alert CANNOT see: a target that was never configured.** `up`
+only counts targets Prometheus knows about. A missing ServiceMonitor produces no `up == 0` series and
+no alert at all. This cluster has been bitten by that repeatedly, and these are the known live gaps as
+of 2026-09-13 — every one of them means the corresponding alerts are **inert**:
+
+| Missing scrape job | Alerts that cannot fire |
+|---|---|
+| Longhorn | `LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `LonghornVolumeCapacityHigh`, `LonghornNodeStorageHigh` |
+| cert-manager | `CertificateExpiringSoon`, `CertificateExpiryImminent`, `CertificateNotReady` |
+| Flux controllers (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` |
+| Hubble / Cilium (`hubble_drop_total`) | `HubbleDeniedFlowSpike` |
+| etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all |
+
+Precedent for how costly this is: Velero shipped with three backup alerts and
+`metrics.serviceMonitor.enabled: false`, so *"a nightly backup could fail — or silently stop running —
+for weeks with nothing to say so."* That ServiceMonitor now exists. The rest of the table does not.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+Which targets are down, grouped by job:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
+   curl -s -G --data-urlencode "query=up == 0" localhost:19090/api/v1/query | python3 -m json.tool | grep -E "\"job\"|\"instance\"|\"namespace\"" '
+```
+
+The scrape-error text, which usually names the cause outright:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
+   curl -s "localhost:19090/api/v1/targets?state=active" | python3 -c "
+import json,sys
+for t in json.load(sys.stdin)[\"data\"][\"activeTargets\"]:
+    if t[\"health\"]!=\"up\": print(t[\"labels\"].get(\"job\"), t[\"scrapeUrl\"], t[\"lastError\"][:120])"'
+```
+
+Which monitors exist at all (the gap this alert cannot see):
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get servicemonitors,podmonitors -A'
+```
+
+## Likely causes, most likely first
+
+### 1. A node is down or NotReady
+One node takes out its `node-exporter` and `kubelet` targets together — roughly the 10% threshold on
+its own. Go to `/runbooks/node-down`; this alert is the echo.
+
+### 2. A workload was rolled, scaled to zero, or crashlooped
+Targets follow pods. A CNPG cluster restarting takes three targets with it. Transient; if it persists,
+the workload is the problem, not the monitoring.
+
+### 3. A NetworkPolicy blocks the scrape
+Default-deny is the baseline everywhere (R-NET-01) and a new component's monitor will be scraped the
+moment it exists — `serviceMonitorSelectorNilUsesHelmValues: false` means Prometheus selects every
+monitor in the cluster — so a newly added workload can produce a permanently-down target purely
+because nothing allows ingress from `monitoring`. The `lastError` will be a timeout. Fix is an
+explicit allow in the component's CiliumNetworkPolicy, in git.
+
+### 4. TLS or auth on the scrape endpoint
+`lastError` will say so (certificate or 401/403). Note a related trap from this cluster: after the
+Cilium upgrade, an in-cluster call to a `*.home` hostname is re-checked against the destination's real
+backend identity, so `toEntities: world` no longer covers it — see `/runbooks/denied-flows`.
+
+### 5. The exporter is genuinely broken
+CDI once had a broken HTTP ServiceMonitor that caused a false `CDIOperatorDown`, and
+`KubeVirtControlPlaneDown` once used a non-existent metric with the wrong job label. Both were fixed.
+An exporter that is up but publishing the wrong series is the hardest case — see
+`/runbooks/cnpg-exporter` for the canonical example (the CNPG exporter can be `up=1` while its whole
+`cnpg_pg_stat_*` family is missing).
+
+## When to escalate or stop
+
+- **Treat this alert as "trust nothing else for now".** Before acting on the *silence* of any other
+  alert, confirm its series exists (`count(<metric>)`), because the table above shows five whole rule
+  groups are currently evaluating against nothing.
+- Escalate if the down targets are `apiserver` or `kubelet` — that is a control-plane problem, not a
+  monitoring one.
+- Adding a missing ServiceMonitor is a git change (a new `ServiceMonitor` next to the component, or a
+  chart value). Do not "fix" a blind spot by silencing the alert.
+- The Longhorn, cert-manager and Flux monitor gaps are known open work. Note them; do not try to
+  hand-apply monitors into the cluster — every durable change goes through git.
+
+---
+Source of truth: `runbooks/targets-down.md`. Index: `/runbooks/`
+Records: `DECISIONS-LOG.md` / commit `2d5171e` (Velero had no ServiceMonitor, so three critical backup
+alerts could never fire); commits `6a7e9d1`, `b57b3e9` (the false CDI/KubeVirt alerts).

runbooks/update-rollback.md

@@ -0,0 +1,150 @@
+# update-rollback — A HelmRelease failed its health gate and is rolling back
+
+**Alert:** `UpdateHealthGateFailed` (critical, 10m)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.automated-updates`
+**Expression:** `max by (namespace, name) (gotk_reconcile_condition{kind="HelmRelease", type="Ready", status="False"}) == 1`
+
+## What this means here
+
+helm-controller holds a HelmRelease at Ready=False while a failed upgrade remediates and rolls back to
+the last-good revision. This is the **R-UPD-03 safety rail firing** — the bad-update auto-rollback
+working — so the alert is a report that the platform protected itself, not necessarily that something
+is broken right now.
+
+Read the expression honestly, though: it matches **any** HelmRelease at Ready=False, not only one that
+is mid-rollback. The rule's own comment flags this as a `TODO(bring-up)`: tighten it with the `reason`
+label (`UpgradeFailed` / `RolledBack`) once confirmed against the deployed helm-controller. So a
+HelmRelease that simply cannot install — a bad value, a missing CRD, a crashlooping pod — lights this
+`critical` too. Your first job is to tell those two cases apart, because they need opposite responses:
+a rollback is self-healing and wants a git revert; a failed *install* is a stall and may be gating an
+entire tier (see `/runbooks/flux-reconcile`).
+
+Real precedent that rollback does work here: during the 2026-09-01 Cilium incident the Helm release
+history shows `v8 e83bf695 → v9 eebd6c88 (the bad render) → v10 e83bf695 (identical to v8)`, and the
+release secret for v9 reads `failed`. Helm's wait genuinely failed and the release was remediated — a
+claim that remediation never engaged was **measured wrong**. That is why the cluster self-healed in
+about six minutes.
+**This alert is currently INERT.** `count(gotk_reconcile_condition)` returned **0 series** on prod on
+2026-09-13 — there is no Flux-controller scrape job among Prometheus's 19 jobs, so every `gotk_*` rule
+evaluates against nothing. Read the HelmRelease CRs directly with the commands below, and see
+`/runbooks/targets-down` for the full list of inert rule groups.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+Which release, and what happened:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get helmreleases.helm.toolkit.fluxcd.io -A \
+     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,REASON:.status.conditions[?\(@.type==\"Ready\"\)].reason,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,LASTSUCCESS:.status.lastAppliedRevision'
+```
+
+The Helm revision history — this is what distinguishes "rolled back" from "never installed":
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n <namespace> get secrets -l owner=helm --sort-by=.metadata.creationTimestamp \
+     -o custom-columns=NAME:.metadata.name,STATUS:.metadata.labels.status,VERSION:.metadata.labels.version'
+```
+
+The controller's reasoning:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n flux-system logs deploy/helm-controller --tail=120 2>&1 | grep -iE "<release>|rollback|remediat|upgrade failed" | tail -30'
+```
+
+What the workload itself is doing:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n <namespace> get pods -o wide; \
+   kubectl --context k3s-prod -n <namespace> get events --sort-by=.lastTimestamp | tail -20'
+```
+
+If the release touches backups or databases, verify them with **fully-qualified names** before deciding
+anything is fine — `kubectl get backup -A` prints "No resources found" on this cluster even with six
+Velero and 27 CNPG backups present:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get backups.velero.io -A; echo; \
+   kubectl --context k3s-prod get backups.postgresql.cnpg.io -A; echo; \
+   kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A'
+```
+
+## Likely causes, most likely first
+
+### 1. A chart upgrade shipped an incompatible image or value
+The recorded instances: **kube-prometheus-stack 65 → 88 brought Grafana 13, which OOM-looped** and
+became "the hidden blocker that stalled the whole kps upgrade and the platform tier" (fixed by raising
+Grafana 256 → 512 → 768Mi and Prometheus 2 → 3Gi, commit `b2b1aa5`); and **headlamp-0.40.1** shipped a
+`-session-ttl` chart/image desync that crashlooped. Remediation: pin the previous version in git
+(`update.pin` is the documented escape hatch, docs/11) and push. The rollback has already restored
+service; the git change stops it happening again next interval.
+
+### 2. Not a rollback at all — a first install that cannot succeed
+`REASON` will be an install/dependency failure and there will be no earlier `deployed` Helm secret. If
+the release is inside the gating `platform` tier, everything downstream is frozen behind it — go to
+`/runbooks/flux-reconcile`, which covers the "no matches for kind" / CRD-ordering class.
+
+### 3. A timeout, not a failure
+`platform` has `timeout: 5m` while Longhorn's HelmRelease has `timeout: 15m` — **the tier can time out
+while a legitimate Longhorn upgrade is still running.** Give it another interval before declaring a
+failure, and check whether pods are still progressing.
+
+### 4. The rollback itself is stuck
+Helm cannot roll back cleanly if the upgrade already mutated something immutable (a StatefulSet
+selector, a PVC). Then Ready=False is permanent until an operator intervenes.
+
+### 5. A post-upgrade health check is failing for an unrelated reason
+A node under I/O pressure makes pods slow to become Ready and the upgrade's wait expires. Cross-read
+`/runbooks/node-io`.
+
+## When to escalate or stop
+
+- **Get the operator if the release is `cilium` or anything in the gating `platform` tier.** A CNI
+  rollback affects every node's datapath.
+- **Do NOT enable `driftDetection.mode: enabled` on a CNI.** During the 2026-09-01 incident the Helm
+  storage manifest *was* the bad render, so drift correction would have actively re-asserted the broken
+  value. This is a recorded conclusion, not a preference.
+- **Do not `helm rollback` by hand.** helm-controller owns the release; a manual rollback creates drift
+  it will fight. Use git.
+- **Do not suspend the HelmRelease to silence the alert** — that trades a visible failure for
+  `/runbooks/flux-suspended` and silent drift.
+- If the rolled-back release is CNPG or Keycloak, expect knock-on SSO effects and read
+  `/runbooks/keycloak-down`; note also that a CNPG update under `primaryUpdateMethod: switchover`
+  cannot change `imageName` and a postgresql parameter in the same reconcile — the operator rejects it.
+
+---
+Source of truth: `runbooks/update-rollback.md`. Index: `/runbooks/`
+Records: `DECISIONS-LOG.md` "2026-09-01 — INCIDENT: merging PR #4 briefly broke Cilium" (the measured
+Helm revision history and the drift-detection conclusion); the Grafana OOM and headlamp desync.

runbooks/vpn-down.md

@@ -0,0 +1,187 @@
+# vpn-down — The egress VPN tunnel is down
+
+**Alert:** `VPNTunnelDown` (critical, 5m)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.networking`
+**Expression:**
+```
+absent(node_network_receive_bytes_total{device="tun0"})
+and on() (
+    (count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
+  or (count(count_over_time(node_network_receive_bytes_total{device="tun0"}[1h])) > 0)
+)
+```
+
+## What this means here
+
+There is **no in-cluster egress-gateway pod** on this platform (ADR-0015). The tunnel is a **host
+systemd service** — `vpn-egress-tunnel`, OpenVPN to PIA — running on one node, and steering is Cilium
+eBPF via a `CiliumEgressGatewayPolicy` (`vpn-egress-steering`). OpenVPN creates and destroys `tun0` with
+the connection, so the `tun0` byte-counter series exists **if and only if** the tunnel is up. The rule
+keys on that presence rather than `node_network_up`, because `tun0` reports `operstate=unknown` and
+`node_network_up` is therefore 0 even when the tunnel is fine.
+
+Live state, measured 2026-09-13: **the tunnel is up** (`node_network_receive_bytes_total{device="tun0"}`
+= 1 series) and the gateway node is labelled (`kube_node_labels{...egress_gateway="true"}` = 1 series),
+so this alert is armed and quiet. The gateway is **`k3s-prod-3`** (192.168.2.12, `eno1`).
+
+**The two facts that decide your response:**
+
+1. **Cilium OSS has no egress-gateway failover.** Gateway-node loss is an outage, not a degradation.
+   The ADR-0016 supersede drafted on 2026-09-06 is still open. Worse, Cilium OSS picks *"the first node
+   in lexical ordering"* among nodeSelector matches — so labelling `k3s-prod-1` would **silently steal**
+   the gateway role and the public egress IP from `k3s-prod-3`. `[vpn_egress_gateway_prod]` is a
+   separate inventory group holding only `k3s-prod-3` for this reason; run the role as
+   `-e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3`, with `--list-hosts` first,
+   every time.
+2. **The `absent()` was originally evaluated cluster-wide and became a permanent false alarm.** On a
+   cluster with no gateway node it could never clear, and from 2026-09-05T21:23Z it fired continuously
+   into `email-critical` — unbounded noise on the one channel that must stay trustworthy. The two
+   guards OR'd above are the fix: the alert arms itself the moment a node is labelled, and stays armed
+   through a kube-state-metrics outage. Before that, the very first version keyed on
+   `absent(gluetun_vpn_up)`, a metric gluetun's control API never exported at all — a pre-existing
+   perma-false-alarm. **If this alert is firing, verify the tunnel by hand before believing it.**
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+Confirm on the host, not from Prometheus:
+
+```
+ssh k3s-prod-3 'systemctl status vpn-egress-tunnel --no-pager | head -20; echo; ip -br addr show tun0; echo; ip link show tun0 | head -2'
+```
+
+Which node is the gateway, and is it labelled?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get nodes -L homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink; echo; \
+   kubectl --context k3s-prod get ciliumegressgatewaypolicies.cilium.io -o yaml | grep -A6 "nodeSelector\|egressIP\|interface"'
+```
+
+**The check that catches the silent failure** (see cause 1 below) — does Cilium's datapath point at the
+interface that actually exists?
+
+```
+ssh k3s-prod-3 'ip -o link | awk -F: "{print \$1, \$2}" | grep tun0'
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg bpf egress list'
+```
+
+If the `Egress Ifindex` in that map is not the live `tun0` ifindex, egress is broken **even though this
+alert is quiet**.
+
+Is traffic actually leaving through the tunnel?
+
+```
+ssh k3s-prod-3 'curl -s --interface tun0 --max-time 10 https://ipinfo.io/ip; echo; nft list table inet vpn_killswitch 2>/dev/null | grep -E "counter|killswitch_" | head'
+```
+
+## Likely causes, most likely first
+
+### 1. The tunnel is UP but Cilium is steering at a dead interface (the silent one)
+This is the richest failure on the platform and it does **not** set off this alert.
+**Cilium resolves a `CiliumEgressGatewayPolicy`'s `interface: tun0` exactly once**, caches
+(egress IP, ifindex), and never re-resolves. On `k3s-dev-3`, `cilium-dbg bpf egress list` showed
+`Egress Ifindex 3` — **there is no ifindex 3 on that node**; the live `tun0` was **1780**. Egress was
+dead for about 80 minutes with nothing alerting and the conformance ledger still reading green.
+`egress-gateway-reconciliation-trigger-interval: 1s` does **not** cover it (netlink link events are not
+Kubernetes object events).
+
+Measured behaviour, so you can pick a lever that works:
+| action | result |
+|---|---|
+| annotate the CEGP | map → 1780 in 8s, PIA IP returns |
+| `systemctl restart vpn-egress-tunnel` | tun0 1780→1985, **map stays 1780**, still broken |
+| the exact `vpn-killswitch` stop/30s/start sequence | still broken 20s+ later |
+| set **any** unrelated label on the gateway **node** | recovers within 10s |
+
+The durable fix is in `roles/vpn_egress_gateway`: the OpenVPN up-script publishes a node label
+`homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on every connect (via
+`systemctl start --no-block`, so it can never block the tunnel), plus a 2-minute convergence timer for
+boot. A node label, not a CEGP annotation — a node writing to GitOps state is a drift fight.
+**Nothing monitors this coupling yet**; that gap is exactly why dev sat dead for 80 minutes.
+
+### 2. PIA handed two gateways the same tunnel address
+Standing rule: **one PIA server per gateway.** The vaulted `.ovpn` pins one server, so when prod came up
+green, dev went down: both gateways dialled `158.173.241.112:1198` and PIA gave both the tunnel address
+`10.72.192.7`. Each reconnect steals it — a measured ~2-minute mutual-eviction cycle (dev restarting at
+22:57:05, 22:59:03, 23:01:05, 23:03:05, 23:05:06 UTC). Fix: the role var
+`vpn_egress_gateway_pia_remote` ("`<host> <port>`") rewrites only the `.ovpn` `remote` line; prod is
+pinned to **`158.173.241.135`** (pool `158.173.241.{34,55,112,135,239}`). *"A single green probe is not
+a stability check."*
+
+### 3. PIA "unreachable" — suspect a stale server list first
+The 2026-08-22/23 chain went port collision → UDP/53 → 8080 → "MTU" (Cilium VXLAN pod MTU 1450 is real
+but irrelevant to a **14-byte** handshake) → "the network blackholes PIA" (**wrong**) before the actual
+cause: **gluetun's stale bundled server list**, dialling retired `46.246.x`/`212.112.x` addresses when
+live SE Stockholm is `158.173.241.x`. Lesson recorded verbatim: when a client "can't reach" the
+provider, suspect the stale server list **first**.
+
+### 4. The gateway node is down
+`absent()` also fires when the gateway node or its node-exporter is gone, so this alert covers
+gateway-node-down by design. Go to `/runbooks/node-down`. Remember: **there is no failover** — while
+`k3s-prod-3` is down, VPN-classed egress is an outage.
+
+### 5. Nothing is wrong and the alert is lying
+Given the two false-alarm post-mortems above, always confirm `ip -br addr show tun0` on the host before
+acting.
+
+## What the kill-switch does and does not do
+
+Proven with counters on `k3s-prod-3`, because the intuitive reading is wrong: with
+**`bpf.masquerade: true`** (which Egress Gateway requires) SNAT and egress happen in eBPF and
+`bpf_redirect()` **skips netfilter POSTROUTING**. Measured: `forward` saddr=pod-CIDR **0**,
+`postrouting` oif WAN saddr=pod-CIDR **0**. **What actually fails closed is Cilium's own drop** —
+tcpdump during tunnel stop and restart showed **0 packets leaving either device**. The nft table is
+kept as defence-in-depth for node-originated tunnel-sourced traffic and was proven both ways:
+`ping -I 10.72.192.7` with the ip-rule → 3/3 at 144 ms with `killswitch_accept` advancing; rule removed
+→ `killswitch_drop` 0→3 packets / 252 bytes, 100% loss. Every rule carries a `counter`, so:
+**if `killswitch_drop` ever counts in normal operation, the datapath has changed.** A reported "leak
+window" was **measured as no WAN leak** — a stale egress IP follows the cached ifindex, not a routing
+lookup. Note the `.nft` file uses `table` + `flush table`, which clears rules but leaves named-set
+elements intact.
+
+## When to escalate or stop
+
+- **Escalate immediately if the gateway node is down.** No failover exists; this is an outage with no
+  in-cluster remedy.
+- **Never label a second node `egress-gateway=true`.** Lexical ordering means `k3s-prod-1` would
+  silently take the role and the public egress IP.
+- **Do not restart the tunnel expecting Cilium to follow** — measured above, it does not. If the map is
+  stale, touching a node label is the lever that works (operator action).
+- **Do not remove or "simplify" the nft kill-switch** because the counters read zero. Zero on the pod
+  path is the expected, measured result of eBPF masquerade; the table guards a different path.
+- A leak concern is a security matter (docs/07) — escalate rather than experimenting.
+
+---
+Source of truth: `runbooks/vpn-down.md`. Index: `/runbooks/`
+Records: `DECISIONS-LOG.md` "2026-09-12 — VPN egress on PROD (k3s-prod-3), and three things the repo
+believed that are not true"; the 2026-08-22/23 gluetun chain; commit `155df8e` (R-NET-06 re-proven on
+prod, recovery ≤3s).