fix(runbooks): declare whole-repo scope, drop the bespoke l0 stage, +cnpg-primary-spread

Three corrections after rebasing onto main. 1. THE GUARD CAUGHT A REAL ONE ON ITS FIRST CONTACT WITH main. main had gained `CNPGPrimariesCoLocated` (platform/cnpg/alerts.yaml) pointing at /runbooks/cnpg-primary-spread, with no runbook. The check failed the rebase exactly as designed — 46 annotations, 35 names, 34 present — naming both the missing file AND the missing configMapGenerator entry. runbooks/cnpg-primary-spread.md now exists: what co-located primaries actually cost here (blast radius, not an outage, because instances: 3 + required anti-affinity), why the drift is uncorrectable-by-declaration (CNPG has no primary-placement field), the readiness gate that keeps the rule quiet during a real node outage, and the ~9s-per-database switchover rebalance, one cluster at a time. The 2026-09-07 outage in which all three primaries sat on one node is the reason the alert exists at all. 2. WHOLE-REPO SCOPE, NOT A HAND-WRITTEN STAGE. main's refactor replaced ci/l0.sh's per-check "UNCONDITIONAL" stages with a declaration: `conformance.home.k3s/scope: "whole-repo"`, unioned into every gate run by Stage 3b's `--include-scope`. My +38-line Stage 3f is removed (ci/l0.sh is now byte-identical to main) and the annotation is declared instead. Assertion (7) of the check was rewritten to match: it no longer asserts its own path appears in ci/l0.sh — it asserts the test DECLARES the scope and that ci/l0.sh still passes --include-scope, since neither half alone makes the check load-bearing. That matters here because ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02", not R-OBS-04, and maps platform/kubevirt/* and runbooks/* to nothing at all. 3. A FACTUAL CORRECTION IN ALL 35 RUNBOOKS. Every page claimed `dev.sh` defaults its current-context to k3s-dev. main's `9b4744d fix(devcontainer): dev.sh defaults to prod` made that false, and a runbook that misstates which cluster a command talks to is worse than one that says nothing. The pages now say what is durably true: the default has already changed once, so naming `--context k3s-prod` is what makes a command mean the same thing next month, and the per-invocation banner is there to be read. The check's own failure message was corrected the same way. Re-verified after all three: check green at 374 assertions (46 annotations, 35/35 runbooks present, 35/35 projected), re-applied to prod, and re-probed in-cluster through the Gateway — node-down, keycloak-down, keycloak-client-secrets, cnpg-archiving, longhorn-faulted, vpn-down, node-io and cnpg-primary-spread all HTTP 200 with text/plain; charset=utf-8 and 8.9-12.1 KB of real body; /runbooks/ index 200; an unknown name 404s with a pointer rather than a blank page; and homepage still answers the apex `/`, confirming Gateway API longest-path precedence. Full merge gate: PASS (spec-immutability, lint, l0, render-golden, l1 all green; prom-rules SKIPPED for want of promtool in the image; L2/L3 SKIPPED, no cluster). Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 00:00:24 -05:00
1 parent 44983095e1
commit b7341b329b
@@ -136,44 +136,6 @@ else
  emit_offline_summary
 fi

-# ---- Stage 3f: every alert runbook_url resolves to a served runbook (UNCONDITIONAL) ----
-# Every platform-health alert rule carries `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` —
-# 42 annotations over 34 names. For three roadmap phases there was no runbooks/ directory and no
-# HTTPRoute serving that path, so a live in-cluster probe of .../runbooks/node-down returned HTTP 404:
-# every alert that reached a human handed them a dead link at the one moment the documentation is read,
-# while the annotation made it look solved. The check asserts the whole chain — the URL's shape, the
-# runbook's existence, a quality floor that a stub cannot clear, the configMapGenerator entry that
-# actually projects it into the server, and the route/listener/nginx prefix agreeing with the
-# annotations — because "the file exists in git" and "the URL resolves in the cluster" are different
-# claims and only the second one helps an operator.
-#
-# WHY ITS OWN UNCONDITIONAL STAGE, same reasoning as 3c/3d/3e: Stage 3b scopes the conformance run to
-# the TOUCHED requirements (R-TEST-03), and ci/touched-requirements.sh maps platform/monitoring/* to
-# "R-OBS-01 R-OBS-02" — NOT R-OBS-04 — and maps platform/kubevirt/* and runbooks/* to nothing at all.
-# So the single most likely breaking change (adding an alert with a runbook_url, or adding a runbook
-# without its generator entry) attributes a requirement set that would FILTER THIS CHECK OUT of exactly
-# the change that introduces the bug. The invariant spans platform/, runbooks/ and clusters/, so it runs
-# on EVERY gate invocation.
-RUNBOOK_CHECK="${RUNBOOK_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/alert-runbooks-resolvable/check.sh}"
-log "Stage: alert runbook links resolve (${RUNBOOK_CHECK})"
-if [ ! -f "$RUNBOOK_CHECK" ]; then
-  skip "alert runbook check not present at '${RUNBOOK_CHECK}' — UNWIRED, not green"
-  summary "- alert-runbooks: **SKIPPED** (check missing)"
-  if [ "$rc" -eq 0 ]; then rc=3; fi
-else
-  rb_rc=0
-  bash "$RUNBOOK_CHECK" || rb_rc=$?
-  case "$rb_rc" in
-    0) ok "Alert runbook links resolve"
-       summary "- alert-runbooks: **passed**" ;;
-    3) skip "alert runbook check SKIPPED (no PrometheusRule present) — not a pass"
-       summary "- alert-runbooks: **SKIPPED** (nothing to audit)"
-       if [ "$rc" -eq 0 ]; then rc=3; fi ;;
-    *) err "Alert runbook links FAILED — an alert links to a runbook that does not exist or is not served"
-       summary "- alert-runbooks: **FAILED**"; rc=1 ;;
-  esac
-fi
-
 case "$rc" in
  0) ok "L0 stage complete" ;;
  3) skip "L0 stage incomplete (unwired seam / missing tool) — not a pass" ;;
@@ -35,8 +35,11 @@
 #           shell example in it (it fails open: DECISIONS-LOG 2026-09-01, the emptied Cilium
 #           k8sServiceHost), and the flux-postbuild-substitution check cannot see it because that check
 #           scans files reachable through `resources:`, not configMapGenerator inputs;
-#   AND (7) the check is LOAD-BEARING: ci/l0.sh runs it on every gate run (Stage 3f) and ci/gate.sh sets
-#           the merge block when the l0 stage fails.
+#   AND (7) the check is LOAD-BEARING: this Test DECLARES
+#           `conformance.home.k3s/scope: whole-repo`, so ci/l0.sh Stage 3b's
+#           `--include-scope whole-repo` unions it into EVERY gate invocation rather than only the runs
+#           whose touched requirements happen to include R-OBS-04; and ci/gate.sh sets the merge block
+#           when the l0 stage fails. A SKIP of a scope-declaring check is UNWIRED, never green.
 #
 # STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library only);
 # conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test wraps the SAME
@@ -59,6 +62,16 @@ metadata:
  annotations:
    conformance.home.k3s/requirements: "R-OBS-04"
    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE — this is what makes the check unconditional. Stage 3b of ci/l0.sh runs
+    # `conformance/run.sh --layer l0 --include-scope whole-repo`, which UNIONS every test carrying this
+    # annotation into the selection regardless of which requirements the change touched. That union is
+    # load-bearing here: ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02" —
+    # NOT R-OBS-04 — and maps platform/kubevirt/* and runbooks/* to nothing at all, so the single most
+    # likely breaking change (adding an alert with a runbook_url, or adding a runbook without its
+    # configMapGenerator entry) attributes a requirement set that would otherwise FILTER THIS CHECK OUT
+    # of exactly the change that introduces the bug. Declaring the scope replaces the hand-written
+    # per-test ci/l0.sh stage that this repo used to need (and that nobody remembered to add).
+    conformance.home.k3s/scope: "whole-repo"
    conformance.home.k3s/safety-critical: "false"
    conformance.home.k3s/executable: "true"
    conformance.home.k3s/intent: >-
@@ -56,7 +56,8 @@
 # maps platform/kubevirt/* and runbooks/* to nothing at all. So the single most likely breaking
 # change (adding an alert with a runbook_url to platform/monitoring/alerts.yaml, or adding a
 # runbook and forgetting the generator entry) attributes a requirement set that would FILTER THIS
-# CHECK OUT. Same reasoning as stages 3c/3d/3e; the invariant is whole-repo, so it runs every time.
+# CHECK OUT. Declaring `conformance.home.k3s/scope: whole-repo` makes ci/l0.sh Stage 3b union it into
+# EVERY gate invocation instead; a SKIP of a scope-declaring check is UNWIRED, never green.
 #
 # WHAT THIS DOES *NOT* PROVE (honest gaps — this is a STATIC check):
 #   - It does not fetch the URL. That it returns HTTP 200 with real content was proven by an
@@ -122,6 +123,7 @@ def read(p: Path) -> str:
 # `kind: PrometheusRule` are scanned, so a runbook_url quoted inside this check or inside prose
 # elsewhere in the repo is not mistaken for an alert annotation. Commented-out lines are skipped:
 # platform/monitoring/alerts.yaml carries a `# runbook_url points at a docs base` header comment.
+SCOPE_DECL_RE = r"^\s*conformance\.home\.k3s/scope:\s*[\"\']?whole-repo"
 RUNBOOK_URL_RE = re.compile(r"""^\s*runbook_url:\s*["']?([^"'\s#]+)["']?\s*$""")

 rule_files = sorted(
@@ -217,8 +219,10 @@ for name, sites in sorted(names.items()):
          f"`infra/devcontainer/dev.sh run bash -c '...'` and the runbook must say so")
    check("--context k3s-prod" in body,
          f"{md.relative_to(ROOT)} never passes --context k3s-prod — dev.sh merges both clusters "
-          f"and defaults to k3s-DEV, so a context-less command silently answers about the wrong "
-          f"cluster in the middle of an incident")
+          f"and picks a current-context PER INVOCATION — a default that has already changed once "
+          f"(k3s-dev until dev was decommissioned on 2026-09-13, k3s-prod now). A context-less command "
+          f"silently answers about whichever cluster the default happens to be, in the middle of an "
+          f"incident")

 # ADVISORY, not a failure: a runbook nothing links to yet. Writing the runbook BEFORE the alert is
 # the right order, and the repo's precedent for "authored but unreferenced" is advisory (see the
@@ -353,13 +357,29 @@ check(re.search(rf"^\s*-\s*{re.escape(PREFIX)}\.yaml\b", read(BASE_KUST), re.M)
 # ============================================ (7) this check is load-bearing in the merge gate
 L0 = read(ROOT / "ci" / "l0.sh")
 GATE = read(ROOT / "ci" / "gate.sh")
-check("alert-runbooks-resolvable/check.sh" in L0,
-      "ci/l0.sh does not run alert-runbooks-resolvable/check.sh. The requirement-scoped conformance "
-      "selection in Stage 3b would filter this check out of exactly the change that breaks it: "
-      "ci/touched-requirements.sh maps platform/monitoring/* to 'R-OBS-01 R-OBS-02' (not R-OBS-04) "
-      "and maps platform/kubevirt/* and runbooks/* to nothing, so adding an alert with a new "
-      "runbook_url attributes no requirement this test carries. It must be invoked directly and "
-      "unconditionally, like stages 3c/3d/3e")
+# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism, in two halves.
+# It is not enough that check.sh exists: the merge gate must SELECT it on every run. The repo used to
+# get that with a hand-written "UNCONDITIONAL" stage per whole-repo check in ci/l0.sh, hardcoding the
+# path; those stages are gone, because the next whole-repo invariant only got one if somebody
+# REMEMBERED to write it. The property is now DECLARED by the test and HONOURED by the gate, so both
+# halves are asserted — neither alone is sufficient. The mechanism itself (that the union really
+# selects, and that a scope-declared SKIP exits 3 rather than passing green) is proven end-to-end and
+# falsifiably by conformance/tests/l0-static/whole-repo-invariants-wired.
+SELF_YAML = (ROOT / "conformance" / "tests" / "l0-static" / "alert-runbooks-resolvable"
+             / "chainsaw-test.yaml")
+SELF = read(SELF_YAML)
+check(re.search(SCOPE_DECL_RE, SELF, re.M) is not None,
+      "alert-runbooks-resolvable/chainsaw-test.yaml does not declare "
+      "conformance.home.k3s/scope: whole-repo, so the merge gate selects this check only when a "
+      "change happens to touch a requirement it is labelled with. ci/touched-requirements.sh maps "
+      "platform/monitoring/* to 'R-OBS-01 R-OBS-02' (NOT R-OBS-04) and maps platform/kubevirt/* and "
+      "runbooks/* to nothing at all — so adding an alert with a new runbook_url, or adding a runbook "
+      "without its configMapGenerator entry, would be filtered out of exactly the change that "
+      "introduces the dead link")
+check("--include-scope" in L0,
+      "ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is a "
+      "purely requirement-scoped run again and the scope annotation above is inert decoration — this "
+      "check would then run only by coincidence")
 check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
      "ci/gate.sh does not set the merge block when the l0 stage fails")

@@ -6,7 +6,7 @@ Every alert rule in this platform carries an annotation of the form

    runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"

-42 of them, over the 34 names below. Until this directory existed, that path returned **HTTP 404** on the
+46 of them, over the 35 names below. Until this directory existed, that path returned **HTTP 404** on the
 live cluster: every alert that reached a human handed them a dead link at the exact moment they needed
 help. These pages are what makes the link resolve.

@@ -18,10 +18,12 @@ is not the cluster being down.

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod`: `dev.sh` merges both clusters into one kubeconfig and defaults its
-current-context to k3s-**dev**, so a command without `--context` silently answers about the wrong cluster.
-`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is the equivalent form. Do not run two `dev.sh`
-invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod`. `dev.sh` merges BOTH clusters into one kubeconfig and picks a
+current-context per invocation, and **that default has already changed once** — k3s-dev until dev was
+decommissioned on 2026-09-13, k3s-prod now. A command that names its context still means the same thing
+next month; one that relies on the default does not, and the failure is silent. `CLUSTER=prod
+infra/devcontainer/dev.sh run kubectl ...` selects it explicitly, and every invocation prints a banner
+naming the cluster — read it. Do not run two `dev.sh` invocations at once; they share one container.

 **Use fully-qualified resource names, always.** Three CRDs on this cluster register the kind `Backup` —
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1`, `velero.io/v1` — and the bare short name resolves to
@@ -92,6 +94,8 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
 - **`cnpg-backup-missed`** — no recent CNPG base backup. Velero does not cover these databases.
 - **`cnpg-wal-growth`** — `pg_wal` not being recycled. Never delete WAL by hand.
 - **`cnpg-exporter`** — the CNPG SQL exporter erroring: a blind spot that looks exactly like health.
+- **`cnpg-primary-spread`** — two or more CNPG primaries on one node. Blast radius, not an outage,
+  and the rebalance costs ~9s of writes per database — one cluster at a time.

 ### Identity
 - **`keycloak-down`** — the IdP is down. Blast radius, and the break-glass path that still works.
@@ -44,10 +44,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -43,10 +43,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -60,10 +60,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -37,10 +37,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -44,10 +44,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -45,10 +45,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -69,10 +69,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -51,10 +51,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -53,10 +53,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -0,0 +1,187 @@
+# cnpg-primary-spread — Two or more CNPG primaries share one node
+
+**Alert:** `CNPGPrimariesCoLocated` (warning, 30m)
+**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg`
+**Expression:** `count by (node) (<ready *-rw endpoints joined to cnpg_collector_up>) > 1`
+**and** gated on every node being Ready:
+`count(kube_node_status_condition{condition="Ready",status="true"} == 1) == count(kube_node_info)`
+
+## What this means here
+
+More than one CloudNativePG **primary** has been running on the same node for 30 minutes, while all
+three nodes are Ready. There are exactly three CNPG Clusters on this platform — `keycloak/keycloak-db`,
+`nextcloud-db/nextcloud-db` and `semaphore-db/semaphore-db` — so on a 3-node cluster the healthy shape is
+one primary per node, and this alert says that shape has drifted.
+
+**This is BLAST RADIUS, not an outage.** Each Cluster has `instances: 3` with one pod per node
+(`podAntiAffinityType: required`) and one synchronous standby (`method: any`, `number: 1`,
+`dataDurability: preferred`), so every one of them survives losing the shared node — you just get two or
+three simultaneous PostgreSQL failovers instead of one. Nothing is broken right now, which is why it is a
+`warning` with a long `for`.
+
+**Why it matters anyway: this exact drift has already caused an outage.** All three Clusters used to run
+`instances: 1` and all three primaries had drifted onto one node, and **on 2026-09-07 that node took SSO,
+Nextcloud and Semaphore down together.** `instances: 3` means the same drift no longer causes an outage —
+but if `keycloak-db`'s primary is co-located and that node dies, the SSO failover happens at the same
+moment as two others, on nodes that are also absorbing Longhorn replica rebuilds.
+
+**Why the drift happens and cannot be declared away: CNPG has no declarative primary-placement field.**
+The primary lands wherever the last promotion put it — a node reboot, a switchover, an operator upgrade.
+So this is avoidable drift that must be *corrected*, not prevented, and the alert re-arms itself so it
+will be reported again rather than silently persisting.
+
+**Why the node-readiness gate is in the expression.** During a real node outage, primaries legitimately
+pile onto the survivors and there is nothing to do about it; alerting then would be noise at the worst
+moment. The `and on ()` clause suppresses the rule unless every node is Ready, and re-arms when the node
+returns.
+
+This rule was proven in both directions on live prod data (2026-09-13): the expression returned
+`{node="k3s-prod-1"} => 3` before the rebalance and a **throwaway twin with `for: 1m`** was observed going
+`pending` at +70s and `firing` at +130s; after two switchovers the same expression returned an **empty
+vector while the inner join still resolved all three primaries** — quiet because the condition is false,
+not because the query went blind. It is also unit-tested in `alerts.promtool-test.yaml`
+(run `./run-alert-tests.sh`) with four cases, mutation-checked so that deleting the `cnpg_collector_up`
+join, the readiness gate, or shortening `for:` makes a test FAIL.
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
+that hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.
+
+USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
+`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
+Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
+a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
+"the backups are gone" and acting on it turns a false negative into real data loss. Write
+`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
+`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
+
+Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
+agents here: every durable change goes through git (`git push origin main` **is** the deploy).
+Where this runbook gives a mutating command it is for an operator, and it says so.
+
+## First commands
+
+Where the primaries are — the one command this alert is about:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get pods -A -l cnpg.io/instanceRole=primary \
+     -o custom-columns=NS:.metadata.namespace,POD:.metadata.name,NODE:.spec.nodeName'
+```
+
+Are all three Clusters actually healthy enough to move one? (Fully-qualified:
+`clusters.postgresql.cnpg.io` — `cluster` is an ambiguous short name.)
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
+     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,INSTANCES:.status.instances,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,PHASE:.status.phase'
+```
+
+Is every node Ready (the gate the rule itself applies)?
+
+```
+infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes -o wide'
+```
+
+Is there storage or I/O trouble that makes a switchover a bad idea right now?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
+     -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy'
+```
+
+```
+ssh k3s-prod-1 'cat /proc/pressure/io'
+```
+
+And confirm DR is intact before you deliberately move a primary — the primary owns the backup
+catalogue, and **fully-qualified names matter**: `kubectl get backup -A` prints "No resources found" on
+this cluster even with 27 CNPG backups present, because three CRDs register the kind `Backup` and the
+short name resolves to Longhorn's (zero objects):
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get backups.postgresql.cnpg.io -A | tail -10; echo; \
+   kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
+     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint'
+```
+
+## Likely causes, most likely first
+
+### 1. A node rebooted and the primaries never moved back
+The dominant cause. Promotions during the reboot put two primaries on a survivor, and CNPG has no reason
+to move them back afterwards. Remediation: a deliberate switchover, below.
+
+### 2. A rolling operator or Postgres upgrade
+`primaryUpdateMethod: switchover` is set deliberately (the CRD default `restart` is a
+delete-and-recreate of the *primary*, i.e. tens of seconds with no Postgres — an SSO outage on
+`keycloak-db`). Switchovers during an upgrade are correct behaviour and leave the primaries wherever
+they landed.
+
+### 3. One node was cordoned or unschedulable for a while
+Same mechanism. Note that `podAntiAffinityType: required` means an instance stays **Pending** until a
+node returns rather than co-locating — that is the accepted cost of the 2026-09-07 fix, and it is why
+the *instances* stay spread even when the *primaries* do not.
+
+### 4. A Longhorn or I/O event caused a health-check failover
+A primary whose storage stalled gets demoted. Cross-read `/runbooks/node-io` and
+`/runbooks/longhorn-degraded` — if that is why, fix the node before rebalancing.
+
+### 5. The alert is firing during a genuine node outage
+It should not be able to: the readiness gate suppresses it. If it fires anyway with a node NotReady, the
+gate is broken — that is a monitoring bug worth fixing, since a rule that cannot stop is fatigue.
+
+## Remediation — move one primary, deliberately
+
+The cnpg plugin is **NOT in the devcontainer image**; fetch the `kubectl-cnpg` release matching the
+operator version (CNPG **1.30.0** here) before you start.
+
+```
+kubectl cnpg promote -n <namespace> <cluster> <instance-on-an-idle-node>
+```
+
+**Do ONE CLUSTER AT A TIME and wait for `readyInstances: 3` before the next.** A switchover interrupts
+writes to that database for about **9 seconds** — measured on both clusters moved on 2026-09-13. Nine
+seconds of `keycloak-db` unavailability is a brief SSO interruption; doing two at once is an avoidable
+compound one.
+
+Pick the target by reading the primaries table above and choosing an instance that already sits on a node
+holding no other primary. Then re-run the table to confirm the new distribution, and let the alert clear
+on its own.
+
+## When to escalate or stop
+
+- **Do not rebalance during any other incident.** A switchover is a small, deliberate write interruption,
+  and it is the wrong thing to add to a node that is already under I/O pressure or to a cluster with
+  degraded Longhorn volumes. This alert has a 30-minute `for` precisely because it is never urgent.
+- **Never switch over a Cluster whose `readyInstances` is below 3.** You would be promoting into a
+  cluster that has no standby left to fall back on.
+- **Never change `dataDurability` to `required`** while thinking about durability here: it is a strict
+  quorum, and one degraded standby would then **block writes** entirely on the SSO database.
+- **Do not `kubectl delete pod` the primary to force a failover.** That is `restart` semantics — the
+  thing `primaryUpdateMethod: switchover` exists to avoid.
+- **Check `ContinuousArchiving` and `firstRecoverabilityPoint` first** (`/runbooks/cnpg-archiving`). If
+  archiving is broken, that is a real countdown to a disk-full PANIC and it outranks blast-radius
+  tidying completely.
+- Escalate to the operator for the switchover itself if the affected Cluster is `keycloak-db`: the brief
+  write interruption is user-visible across SSO, kubectl OIDC, Grafana, Headlamp and Gitea.
+
+---
+Source of truth: `runbooks/cnpg-primary-spread.md`. Index: `/runbooks/`
+Records: `platform/cnpg/alerts.yaml` (the rule's own both-directions proof on live prod data and its
+promtool tests); commits `f338e91` (instances 3), `e8d8601` (required anti-affinity + switchover),
+`cac413e` (synchronous replication, `dataDurability: preferred`); the 2026-09-07 single-node SSO outage.
@@ -43,10 +43,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -44,10 +44,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -43,10 +43,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -40,10 +40,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -35,10 +35,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -35,10 +35,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -69,10 +69,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -49,10 +49,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -41,10 +41,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -39,10 +39,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -35,10 +35,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -44,10 +44,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -58,6 +58,7 @@ configMapGenerator:
      - cnpg-archiving=cnpg-archiving.md
      - cnpg-backup-missed=cnpg-backup-missed.md
      - cnpg-exporter=cnpg-exporter.md
+      - cnpg-primary-spread=cnpg-primary-spread.md
      - cnpg-wal-growth=cnpg-wal-growth.md
      - denied-flows=denied-flows.md
      - flux-reconcile=flux-reconcile.md
@@ -50,10 +50,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -49,10 +49,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -50,10 +50,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -34,10 +34,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -31,10 +31,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -38,10 +38,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -38,10 +38,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -47,10 +47,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -34,10 +34,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -40,10 +40,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -42,10 +42,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -36,10 +36,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
@@ -48,10 +48,14 @@ that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

-Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
-defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
-about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
-Do not run two `dev.sh` invocations at once — they share one container and interfere.
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
+until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
+the same thing next month as it does today; one that relies on the default does not, and the failure is
+silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
+kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
+to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
+interfere.

 USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
 `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to