From b7341b329b71fa86e13eff61e50e1452748df448 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 13 Sep 2026 00:00:24 -0500 Subject: [PATCH] fix(runbooks): declare whole-repo scope, drop the bespoke l0 stage, +cnpg-primary-spread MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three corrections after rebasing onto main. 1. THE GUARD CAUGHT A REAL ONE ON ITS FIRST CONTACT WITH main. main had gained `CNPGPrimariesCoLocated` (platform/cnpg/alerts.yaml) pointing at /runbooks/cnpg-primary-spread, with no runbook. The check failed the rebase exactly as designed — 46 annotations, 35 names, 34 present — naming both the missing file AND the missing configMapGenerator entry. runbooks/cnpg-primary-spread.md now exists: what co-located primaries actually cost here (blast radius, not an outage, because instances: 3 + required anti-affinity), why the drift is uncorrectable-by-declaration (CNPG has no primary-placement field), the readiness gate that keeps the rule quiet during a real node outage, and the ~9s-per-database switchover rebalance, one cluster at a time. The 2026-09-07 outage in which all three primaries sat on one node is the reason the alert exists at all. 2. WHOLE-REPO SCOPE, NOT A HAND-WRITTEN STAGE. main's refactor replaced ci/l0.sh's per-check "UNCONDITIONAL" stages with a declaration: `conformance.home.k3s/scope: "whole-repo"`, unioned into every gate run by Stage 3b's `--include-scope`. My +38-line Stage 3f is removed (ci/l0.sh is now byte-identical to main) and the annotation is declared instead. Assertion (7) of the check was rewritten to match: it no longer asserts its own path appears in ci/l0.sh — it asserts the test DECLARES the scope and that ci/l0.sh still passes --include-scope, since neither half alone makes the check load-bearing. That matters here because ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02", not R-OBS-04, and maps platform/kubevirt/* and runbooks/* to nothing at all. 3. A FACTUAL CORRECTION IN ALL 35 RUNBOOKS. Every page claimed `dev.sh` defaults its current-context to k3s-dev. main's `9b4744d fix(devcontainer): dev.sh defaults to prod` made that false, and a runbook that misstates which cluster a command talks to is worse than one that says nothing. The pages now say what is durably true: the default has already changed once, so naming `--context k3s-prod` is what makes a command mean the same thing next month, and the per-invocation banner is there to be read. The check's own failure message was corrected the same way. Re-verified after all three: check green at 374 assertions (46 annotations, 35/35 runbooks present, 35/35 projected), re-applied to prod, and re-probed in-cluster through the Gateway — node-down, keycloak-down, keycloak-client-secrets, cnpg-archiving, longhorn-faulted, vpn-down, node-io and cnpg-primary-spread all HTTP 200 with text/plain; charset=utf-8 and 8.9-12.1 KB of real body; /runbooks/ index 200; an unknown name 404s with a pointer rather than a blank page; and homepage still answers the apex `/`, confirming Gateway API longest-path precedence. Full merge gate: PASS (spec-immutability, lint, l0, render-golden, l1 all green; prom-rules SKIPPED for want of promtool in the image; L2/L3 SKIPPED, no cluster). Co-Authored-By: Claude Opus 5 --- ci/l0.sh | 38 ---- .../chainsaw-test.yaml | 17 +- .../alert-runbooks-resolvable/check.sh | 40 +++- runbooks/README.md | 14 +- runbooks/backup-failure.md | 12 +- runbooks/backup-missed.md | 12 +- runbooks/backup-partial.md | 12 +- runbooks/cdi-import-stuck.md | 12 +- runbooks/cert-expiry.md | 12 +- runbooks/cert-issuance.md | 12 +- runbooks/cnpg-archiving.md | 12 +- runbooks/cnpg-backup-missed.md | 12 +- runbooks/cnpg-exporter.md | 12 +- runbooks/cnpg-primary-spread.md | 187 ++++++++++++++++++ runbooks/cnpg-wal-growth.md | 12 +- runbooks/denied-flows.md | 12 +- runbooks/flux-reconcile.md | 12 +- runbooks/flux-suspended.md | 12 +- runbooks/image-automation.md | 12 +- runbooks/image-policy-stalled.md | 12 +- runbooks/keycloak-client-secrets.md | 12 +- runbooks/keycloak-db.md | 12 +- runbooks/keycloak-down.md | 12 +- runbooks/kubevirt-control-plane.md | 12 +- runbooks/kubevirt-crashloop.md | 12 +- runbooks/kubevirt-launch-failed.md | 12 +- runbooks/kustomization.yaml | 1 + runbooks/longhorn-capacity.md | 12 +- runbooks/longhorn-degraded.md | 12 +- runbooks/longhorn-faulted.md | 12 +- runbooks/monitoring-capacity.md | 12 +- runbooks/node-cpu.md | 12 +- runbooks/node-disk.md | 12 +- runbooks/node-down.md | 12 +- runbooks/node-io.md | 12 +- runbooks/node-memory.md | 12 +- runbooks/pvc-capacity.md | 12 +- runbooks/targets-down.md | 12 +- runbooks/update-rollback.md | 12 +- runbooks/vpn-down.md | 12 +- 40 files changed, 514 insertions(+), 191 deletions(-) create mode 100644 runbooks/cnpg-primary-spread.md diff --git a/ci/l0.sh b/ci/l0.sh index e824cb0..a298a59 100755 --- a/ci/l0.sh +++ b/ci/l0.sh @@ -136,44 +136,6 @@ else emit_offline_summary fi -# ---- Stage 3f: every alert runbook_url resolves to a served runbook (UNCONDITIONAL) ---- -# Every platform-health alert rule carries `runbook_url: "https://${BASE_DOMAIN}/runbooks/"` — -# 42 annotations over 34 names. For three roadmap phases there was no runbooks/ directory and no -# HTTPRoute serving that path, so a live in-cluster probe of .../runbooks/node-down returned HTTP 404: -# every alert that reached a human handed them a dead link at the one moment the documentation is read, -# while the annotation made it look solved. The check asserts the whole chain — the URL's shape, the -# runbook's existence, a quality floor that a stub cannot clear, the configMapGenerator entry that -# actually projects it into the server, and the route/listener/nginx prefix agreeing with the -# annotations — because "the file exists in git" and "the URL resolves in the cluster" are different -# claims and only the second one helps an operator. -# -# WHY ITS OWN UNCONDITIONAL STAGE, same reasoning as 3c/3d/3e: Stage 3b scopes the conformance run to -# the TOUCHED requirements (R-TEST-03), and ci/touched-requirements.sh maps platform/monitoring/* to -# "R-OBS-01 R-OBS-02" — NOT R-OBS-04 — and maps platform/kubevirt/* and runbooks/* to nothing at all. -# So the single most likely breaking change (adding an alert with a runbook_url, or adding a runbook -# without its generator entry) attributes a requirement set that would FILTER THIS CHECK OUT of exactly -# the change that introduces the bug. The invariant spans platform/, runbooks/ and clusters/, so it runs -# on EVERY gate invocation. -RUNBOOK_CHECK="${RUNBOOK_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/alert-runbooks-resolvable/check.sh}" -log "Stage: alert runbook links resolve (${RUNBOOK_CHECK})" -if [ ! -f "$RUNBOOK_CHECK" ]; then - skip "alert runbook check not present at '${RUNBOOK_CHECK}' — UNWIRED, not green" - summary "- alert-runbooks: **SKIPPED** (check missing)" - if [ "$rc" -eq 0 ]; then rc=3; fi -else - rb_rc=0 - bash "$RUNBOOK_CHECK" || rb_rc=$? - case "$rb_rc" in - 0) ok "Alert runbook links resolve" - summary "- alert-runbooks: **passed**" ;; - 3) skip "alert runbook check SKIPPED (no PrometheusRule present) — not a pass" - summary "- alert-runbooks: **SKIPPED** (nothing to audit)" - if [ "$rc" -eq 0 ]; then rc=3; fi ;; - *) err "Alert runbook links FAILED — an alert links to a runbook that does not exist or is not served" - summary "- alert-runbooks: **FAILED**"; rc=1 ;; - esac -fi - case "$rc" in 0) ok "L0 stage complete" ;; 3) skip "L0 stage incomplete (unwired seam / missing tool) — not a pass" ;; diff --git a/conformance/tests/l0-static/alert-runbooks-resolvable/chainsaw-test.yaml b/conformance/tests/l0-static/alert-runbooks-resolvable/chainsaw-test.yaml index 18fed29..b9b006a 100644 --- a/conformance/tests/l0-static/alert-runbooks-resolvable/chainsaw-test.yaml +++ b/conformance/tests/l0-static/alert-runbooks-resolvable/chainsaw-test.yaml @@ -35,8 +35,11 @@ # shell example in it (it fails open: DECISIONS-LOG 2026-09-01, the emptied Cilium # k8sServiceHost), and the flux-postbuild-substitution check cannot see it because that check # scans files reachable through `resources:`, not configMapGenerator inputs; -# AND (7) the check is LOAD-BEARING: ci/l0.sh runs it on every gate run (Stage 3f) and ci/gate.sh sets -# the merge block when the l0 stage fails. +# AND (7) the check is LOAD-BEARING: this Test DECLARES +# `conformance.home.k3s/scope: whole-repo`, so ci/l0.sh Stage 3b's +# `--include-scope whole-repo` unions it into EVERY gate invocation rather than only the runs +# whose touched requirements happen to include R-OBS-04; and ci/gate.sh sets the merge block +# when the l0 stage fails. A SKIP of a scope-declaring check is UNWIRED, never green. # # STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library only); # conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test wraps the SAME @@ -59,6 +62,16 @@ metadata: annotations: conformance.home.k3s/requirements: "R-OBS-04" conformance.home.k3s/layer: "L0" + # WHOLE-REPO SCOPE — this is what makes the check unconditional. Stage 3b of ci/l0.sh runs + # `conformance/run.sh --layer l0 --include-scope whole-repo`, which UNIONS every test carrying this + # annotation into the selection regardless of which requirements the change touched. That union is + # load-bearing here: ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02" — + # NOT R-OBS-04 — and maps platform/kubevirt/* and runbooks/* to nothing at all, so the single most + # likely breaking change (adding an alert with a runbook_url, or adding a runbook without its + # configMapGenerator entry) attributes a requirement set that would otherwise FILTER THIS CHECK OUT + # of exactly the change that introduces the bug. Declaring the scope replaces the hand-written + # per-test ci/l0.sh stage that this repo used to need (and that nobody remembered to add). + conformance.home.k3s/scope: "whole-repo" conformance.home.k3s/safety-critical: "false" conformance.home.k3s/executable: "true" conformance.home.k3s/intent: >- diff --git a/conformance/tests/l0-static/alert-runbooks-resolvable/check.sh b/conformance/tests/l0-static/alert-runbooks-resolvable/check.sh index 48a0cdb..8777529 100755 --- a/conformance/tests/l0-static/alert-runbooks-resolvable/check.sh +++ b/conformance/tests/l0-static/alert-runbooks-resolvable/check.sh @@ -56,7 +56,8 @@ # maps platform/kubevirt/* and runbooks/* to nothing at all. So the single most likely breaking # change (adding an alert with a runbook_url to platform/monitoring/alerts.yaml, or adding a # runbook and forgetting the generator entry) attributes a requirement set that would FILTER THIS -# CHECK OUT. Same reasoning as stages 3c/3d/3e; the invariant is whole-repo, so it runs every time. +# CHECK OUT. Declaring `conformance.home.k3s/scope: whole-repo` makes ci/l0.sh Stage 3b union it into +# EVERY gate invocation instead; a SKIP of a scope-declaring check is UNWIRED, never green. # # WHAT THIS DOES *NOT* PROVE (honest gaps — this is a STATIC check): # - It does not fetch the URL. That it returns HTTP 200 with real content was proven by an @@ -122,6 +123,7 @@ def read(p: Path) -> str: # `kind: PrometheusRule` are scanned, so a runbook_url quoted inside this check or inside prose # elsewhere in the repo is not mistaken for an alert annotation. Commented-out lines are skipped: # platform/monitoring/alerts.yaml carries a `# runbook_url points at a docs base` header comment. +SCOPE_DECL_RE = r"^\s*conformance\.home\.k3s/scope:\s*[\"\']?whole-repo" RUNBOOK_URL_RE = re.compile(r"""^\s*runbook_url:\s*["']?([^"'\s#]+)["']?\s*$""") rule_files = sorted( @@ -217,8 +219,10 @@ for name, sites in sorted(names.items()): f"`infra/devcontainer/dev.sh run bash -c '...'` and the runbook must say so") check("--context k3s-prod" in body, f"{md.relative_to(ROOT)} never passes --context k3s-prod — dev.sh merges both clusters " - f"and defaults to k3s-DEV, so a context-less command silently answers about the wrong " - f"cluster in the middle of an incident") + f"and picks a current-context PER INVOCATION — a default that has already changed once " + f"(k3s-dev until dev was decommissioned on 2026-09-13, k3s-prod now). A context-less command " + f"silently answers about whichever cluster the default happens to be, in the middle of an " + f"incident") # ADVISORY, not a failure: a runbook nothing links to yet. Writing the runbook BEFORE the alert is # the right order, and the repo's precedent for "authored but unreferenced" is advisory (see the @@ -353,13 +357,29 @@ check(re.search(rf"^\s*-\s*{re.escape(PREFIX)}\.yaml\b", read(BASE_KUST), re.M) # ============================================ (7) this check is load-bearing in the merge gate L0 = read(ROOT / "ci" / "l0.sh") GATE = read(ROOT / "ci" / "gate.sh") -check("alert-runbooks-resolvable/check.sh" in L0, - "ci/l0.sh does not run alert-runbooks-resolvable/check.sh. The requirement-scoped conformance " - "selection in Stage 3b would filter this check out of exactly the change that breaks it: " - "ci/touched-requirements.sh maps platform/monitoring/* to 'R-OBS-01 R-OBS-02' (not R-OBS-04) " - "and maps platform/kubevirt/* and runbooks/* to nothing, so adding an alert with a new " - "runbook_url attributes no requirement this test carries. It must be invoked directly and " - "unconditionally, like stages 3c/3d/3e") +# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism, in two halves. +# It is not enough that check.sh exists: the merge gate must SELECT it on every run. The repo used to +# get that with a hand-written "UNCONDITIONAL" stage per whole-repo check in ci/l0.sh, hardcoding the +# path; those stages are gone, because the next whole-repo invariant only got one if somebody +# REMEMBERED to write it. The property is now DECLARED by the test and HONOURED by the gate, so both +# halves are asserted — neither alone is sufficient. The mechanism itself (that the union really +# selects, and that a scope-declared SKIP exits 3 rather than passing green) is proven end-to-end and +# falsifiably by conformance/tests/l0-static/whole-repo-invariants-wired. +SELF_YAML = (ROOT / "conformance" / "tests" / "l0-static" / "alert-runbooks-resolvable" + / "chainsaw-test.yaml") +SELF = read(SELF_YAML) +check(re.search(SCOPE_DECL_RE, SELF, re.M) is not None, + "alert-runbooks-resolvable/chainsaw-test.yaml does not declare " + "conformance.home.k3s/scope: whole-repo, so the merge gate selects this check only when a " + "change happens to touch a requirement it is labelled with. ci/touched-requirements.sh maps " + "platform/monitoring/* to 'R-OBS-01 R-OBS-02' (NOT R-OBS-04) and maps platform/kubevirt/* and " + "runbooks/* to nothing at all — so adding an alert with a new runbook_url, or adding a runbook " + "without its configMapGenerator entry, would be filtered out of exactly the change that " + "introduces the dead link") +check("--include-scope" in L0, + "ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is a " + "purely requirement-scoped run again and the scope annotation above is inert decoration — this " + "check would then run only by coincidence") check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE), "ci/gate.sh does not set the merge block when the l0 stage fails") diff --git a/runbooks/README.md b/runbooks/README.md index 655066b..b580775 100644 --- a/runbooks/README.md +++ b/runbooks/README.md @@ -6,7 +6,7 @@ Every alert rule in this platform carries an annotation of the form runbook_url: "https://${BASE_DOMAIN}/runbooks/" -42 of them, over the 34 names below. Until this directory existed, that path returned **HTTP 404** on the +46 of them, over the 35 names below. Until this directory existed, that path returned **HTTP 404** on the live cluster: every alert that reached a human handed them a dead link at the exact moment they needed help. These pages are what makes the link resolve. @@ -18,10 +18,12 @@ is not the cluster being down. infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod`: `dev.sh` merges both clusters into one kubeconfig and defaults its -current-context to k3s-**dev**, so a command without `--context` silently answers about the wrong cluster. -`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is the equivalent form. Do not run two `dev.sh` -invocations at once — they share one container and interfere. +Always pass `--context k3s-prod`. `dev.sh` merges BOTH clusters into one kubeconfig and picks a +current-context per invocation, and **that default has already changed once** — k3s-dev until dev was +decommissioned on 2026-09-13, k3s-prod now. A command that names its context still means the same thing +next month; one that relies on the default does not, and the failure is silent. `CLUSTER=prod +infra/devcontainer/dev.sh run kubectl ...` selects it explicitly, and every invocation prints a banner +naming the cluster — read it. Do not run two `dev.sh` invocations at once; they share one container. **Use fully-qualified resource names, always.** Three CRDs on this cluster register the kind `Backup` — `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1`, `velero.io/v1` — and the bare short name resolves to @@ -92,6 +94,8 @@ Each affected runbook says so on its own page and gives the by-hand equivalent. - **`cnpg-backup-missed`** — no recent CNPG base backup. Velero does not cover these databases. - **`cnpg-wal-growth`** — `pg_wal` not being recycled. Never delete WAL by hand. - **`cnpg-exporter`** — the CNPG SQL exporter erroring: a blind spot that looks exactly like health. +- **`cnpg-primary-spread`** — two or more CNPG primaries on one node. Blast radius, not an outage, + and the rebalance costs ~9s of writes per database — one cluster at a time. ### Identity - **`keycloak-down`** — the IdP is down. Blast radius, and the break-glass path that still works. diff --git a/runbooks/backup-failure.md b/runbooks/backup-failure.md index ef0486f..b87a65c 100644 --- a/runbooks/backup-failure.md +++ b/runbooks/backup-failure.md @@ -44,10 +44,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/backup-missed.md b/runbooks/backup-missed.md index 0ba1ae5..a5b8958 100644 --- a/runbooks/backup-missed.md +++ b/runbooks/backup-missed.md @@ -43,10 +43,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/backup-partial.md b/runbooks/backup-partial.md index d652a0b..af522af 100644 --- a/runbooks/backup-partial.md +++ b/runbooks/backup-partial.md @@ -60,10 +60,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/cdi-import-stuck.md b/runbooks/cdi-import-stuck.md index f8bb8d9..204559e 100644 --- a/runbooks/cdi-import-stuck.md +++ b/runbooks/cdi-import-stuck.md @@ -37,10 +37,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/cert-expiry.md b/runbooks/cert-expiry.md index a8328d4..eceabc6 100644 --- a/runbooks/cert-expiry.md +++ b/runbooks/cert-expiry.md @@ -44,10 +44,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/cert-issuance.md b/runbooks/cert-issuance.md index ca19f42..b278f30 100644 --- a/runbooks/cert-issuance.md +++ b/runbooks/cert-issuance.md @@ -45,10 +45,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/cnpg-archiving.md b/runbooks/cnpg-archiving.md index 648f9b7..4505c6d 100644 --- a/runbooks/cnpg-archiving.md +++ b/runbooks/cnpg-archiving.md @@ -69,10 +69,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/cnpg-backup-missed.md b/runbooks/cnpg-backup-missed.md index 8b506bd..c274167 100644 --- a/runbooks/cnpg-backup-missed.md +++ b/runbooks/cnpg-backup-missed.md @@ -51,10 +51,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/cnpg-exporter.md b/runbooks/cnpg-exporter.md index 4edcea2..7714561 100644 --- a/runbooks/cnpg-exporter.md +++ b/runbooks/cnpg-exporter.md @@ -53,10 +53,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/cnpg-primary-spread.md b/runbooks/cnpg-primary-spread.md new file mode 100644 index 0000000..7287b68 --- /dev/null +++ b/runbooks/cnpg-primary-spread.md @@ -0,0 +1,187 @@ +# cnpg-primary-spread — Two or more CNPG primaries share one node + +**Alert:** `CNPGPrimariesCoLocated` (warning, 30m) +**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg` +**Expression:** `count by (node) () > 1` +**and** gated on every node being Ready: +`count(kube_node_status_condition{condition="Ready",status="true"} == 1) == count(kube_node_info)` + +## What this means here + +More than one CloudNativePG **primary** has been running on the same node for 30 minutes, while all +three nodes are Ready. There are exactly three CNPG Clusters on this platform — `keycloak/keycloak-db`, +`nextcloud-db/nextcloud-db` and `semaphore-db/semaphore-db` — so on a 3-node cluster the healthy shape is +one primary per node, and this alert says that shape has drifted. + +**This is BLAST RADIUS, not an outage.** Each Cluster has `instances: 3` with one pod per node +(`podAntiAffinityType: required`) and one synchronous standby (`method: any`, `number: 1`, +`dataDurability: preferred`), so every one of them survives losing the shared node — you just get two or +three simultaneous PostgreSQL failovers instead of one. Nothing is broken right now, which is why it is a +`warning` with a long `for`. + +**Why it matters anyway: this exact drift has already caused an outage.** All three Clusters used to run +`instances: 1` and all three primaries had drifted onto one node, and **on 2026-09-07 that node took SSO, +Nextcloud and Semaphore down together.** `instances: 3` means the same drift no longer causes an outage — +but if `keycloak-db`'s primary is co-located and that node dies, the SSO failover happens at the same +moment as two others, on nodes that are also absorbing Longhorn replica rebuilds. + +**Why the drift happens and cannot be declared away: CNPG has no declarative primary-placement field.** +The primary lands wherever the last promotion put it — a node reboot, a switchover, an operator upgrade. +So this is avoidable drift that must be *corrected*, not prevented, and the alert re-arms itself so it +will be reported again rather than silently persisting. + +**Why the node-readiness gate is in the expression.** During a real node outage, primaries legitimately +pile onto the survivors and there is nothing to do about it; alerting then would be noise at the worst +moment. The `and on ()` clause suppresses the rule unless every node is Ready, and re-arms when the node +returns. + +This rule was proven in both directions on live prod data (2026-09-13): the expression returned +`{node="k3s-prod-1"} => 3` before the rebalance and a **throwaway twin with `for: 1m`** was observed going +`pending` at +70s and `firing` at +130s; after two switchovers the same expression returned an **empty +vector while the inner join still resolved all three primaries** — quiet because the condition is false, +not because the query went blind. It is also unit-tested in `alerts.promtool-test.yaml` +(run `./run-alert-tests.sh`) with four cases, mutation-checked so that deleting the `cnpg_collector_up` +join, the readiness gate, or shortening `for:` makes a test FAIL. +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev +container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — +that hang is not the cluster being down. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. + +USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: +`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to +Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on +a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as +"the backups are gone" and acting on it turns a false negative into real data loss. Write +`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, +`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. + +Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for +agents here: every durable change goes through git (`git push origin main` **is** the deploy). +Where this runbook gives a mutating command it is for an operator, and it says so. + +## First commands + +Where the primaries are — the one command this alert is about: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get pods -A -l cnpg.io/instanceRole=primary \ + -o custom-columns=NS:.metadata.namespace,POD:.metadata.name,NODE:.spec.nodeName' +``` + +Are all three Clusters actually healthy enough to move one? (Fully-qualified: +`clusters.postgresql.cnpg.io` — `cluster` is an ambiguous short name.) + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,INSTANCES:.status.instances,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,PHASE:.status.phase' +``` + +Is every node Ready (the gate the rule itself applies)? + +``` +infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes -o wide' +``` + +Is there storage or I/O trouble that makes a switchover a bad idea right now? + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \ + -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy' +``` + +``` +ssh k3s-prod-1 'cat /proc/pressure/io' +``` + +And confirm DR is intact before you deliberately move a primary — the primary owns the backup +catalogue, and **fully-qualified names matter**: `kubectl get backup -A` prints "No resources found" on +this cluster even with 27 CNPG backups present, because three CRDs register the kind `Backup` and the +short name resolves to Longhorn's (zero objects): + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get backups.postgresql.cnpg.io -A | tail -10; echo; \ + kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint' +``` + +## Likely causes, most likely first + +### 1. A node rebooted and the primaries never moved back +The dominant cause. Promotions during the reboot put two primaries on a survivor, and CNPG has no reason +to move them back afterwards. Remediation: a deliberate switchover, below. + +### 2. A rolling operator or Postgres upgrade +`primaryUpdateMethod: switchover` is set deliberately (the CRD default `restart` is a +delete-and-recreate of the *primary*, i.e. tens of seconds with no Postgres — an SSO outage on +`keycloak-db`). Switchovers during an upgrade are correct behaviour and leave the primaries wherever +they landed. + +### 3. One node was cordoned or unschedulable for a while +Same mechanism. Note that `podAntiAffinityType: required` means an instance stays **Pending** until a +node returns rather than co-locating — that is the accepted cost of the 2026-09-07 fix, and it is why +the *instances* stay spread even when the *primaries* do not. + +### 4. A Longhorn or I/O event caused a health-check failover +A primary whose storage stalled gets demoted. Cross-read `/runbooks/node-io` and +`/runbooks/longhorn-degraded` — if that is why, fix the node before rebalancing. + +### 5. The alert is firing during a genuine node outage +It should not be able to: the readiness gate suppresses it. If it fires anyway with a node NotReady, the +gate is broken — that is a monitoring bug worth fixing, since a rule that cannot stop is fatigue. + +## Remediation — move one primary, deliberately + +The cnpg plugin is **NOT in the devcontainer image**; fetch the `kubectl-cnpg` release matching the +operator version (CNPG **1.30.0** here) before you start. + +``` +kubectl cnpg promote -n +``` + +**Do ONE CLUSTER AT A TIME and wait for `readyInstances: 3` before the next.** A switchover interrupts +writes to that database for about **9 seconds** — measured on both clusters moved on 2026-09-13. Nine +seconds of `keycloak-db` unavailability is a brief SSO interruption; doing two at once is an avoidable +compound one. + +Pick the target by reading the primaries table above and choosing an instance that already sits on a node +holding no other primary. Then re-run the table to confirm the new distribution, and let the alert clear +on its own. + +## When to escalate or stop + +- **Do not rebalance during any other incident.** A switchover is a small, deliberate write interruption, + and it is the wrong thing to add to a node that is already under I/O pressure or to a cluster with + degraded Longhorn volumes. This alert has a 30-minute `for` precisely because it is never urgent. +- **Never switch over a Cluster whose `readyInstances` is below 3.** You would be promoting into a + cluster that has no standby left to fall back on. +- **Never change `dataDurability` to `required`** while thinking about durability here: it is a strict + quorum, and one degraded standby would then **block writes** entirely on the SSO database. +- **Do not `kubectl delete pod` the primary to force a failover.** That is `restart` semantics — the + thing `primaryUpdateMethod: switchover` exists to avoid. +- **Check `ContinuousArchiving` and `firstRecoverabilityPoint` first** (`/runbooks/cnpg-archiving`). If + archiving is broken, that is a real countdown to a disk-full PANIC and it outranks blast-radius + tidying completely. +- Escalate to the operator for the switchover itself if the affected Cluster is `keycloak-db`: the brief + write interruption is user-visible across SSO, kubectl OIDC, Grafana, Headlamp and Gitea. + +--- +Source of truth: `runbooks/cnpg-primary-spread.md`. Index: `/runbooks/` +Records: `platform/cnpg/alerts.yaml` (the rule's own both-directions proof on live prod data and its +promtool tests); commits `f338e91` (instances 3), `e8d8601` (required anti-affinity + switchover), +`cac413e` (synchronous replication, `dataDurability: preferred`); the 2026-09-07 single-node SSO outage. diff --git a/runbooks/cnpg-wal-growth.md b/runbooks/cnpg-wal-growth.md index e3d9db2..4d11b76 100644 --- a/runbooks/cnpg-wal-growth.md +++ b/runbooks/cnpg-wal-growth.md @@ -43,10 +43,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/denied-flows.md b/runbooks/denied-flows.md index 5843b58..a44af99 100644 --- a/runbooks/denied-flows.md +++ b/runbooks/denied-flows.md @@ -44,10 +44,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/flux-reconcile.md b/runbooks/flux-reconcile.md index 52ec45d..70d12bb 100644 --- a/runbooks/flux-reconcile.md +++ b/runbooks/flux-reconcile.md @@ -43,10 +43,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/flux-suspended.md b/runbooks/flux-suspended.md index 44addf4..3a766bc 100644 --- a/runbooks/flux-suspended.md +++ b/runbooks/flux-suspended.md @@ -40,10 +40,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/image-automation.md b/runbooks/image-automation.md index 3d71e7b..596f1e5 100644 --- a/runbooks/image-automation.md +++ b/runbooks/image-automation.md @@ -35,10 +35,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/image-policy-stalled.md b/runbooks/image-policy-stalled.md index d61fd34..2edcf7d 100644 --- a/runbooks/image-policy-stalled.md +++ b/runbooks/image-policy-stalled.md @@ -35,10 +35,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/keycloak-client-secrets.md b/runbooks/keycloak-client-secrets.md index 2cc2d64..9c72463 100644 --- a/runbooks/keycloak-client-secrets.md +++ b/runbooks/keycloak-client-secrets.md @@ -69,10 +69,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/keycloak-db.md b/runbooks/keycloak-db.md index 6bfee83..0fd0d7a 100644 --- a/runbooks/keycloak-db.md +++ b/runbooks/keycloak-db.md @@ -49,10 +49,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/keycloak-down.md b/runbooks/keycloak-down.md index a758bb1..d14948d 100644 --- a/runbooks/keycloak-down.md +++ b/runbooks/keycloak-down.md @@ -41,10 +41,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/kubevirt-control-plane.md b/runbooks/kubevirt-control-plane.md index d4d55bb..c6c3cfb 100644 --- a/runbooks/kubevirt-control-plane.md +++ b/runbooks/kubevirt-control-plane.md @@ -39,10 +39,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/kubevirt-crashloop.md b/runbooks/kubevirt-crashloop.md index 97d7158..9d612cb 100644 --- a/runbooks/kubevirt-crashloop.md +++ b/runbooks/kubevirt-crashloop.md @@ -35,10 +35,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/kubevirt-launch-failed.md b/runbooks/kubevirt-launch-failed.md index 06e3c6b..6308be6 100644 --- a/runbooks/kubevirt-launch-failed.md +++ b/runbooks/kubevirt-launch-failed.md @@ -44,10 +44,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/kustomization.yaml b/runbooks/kustomization.yaml index 7554231..2682e79 100644 --- a/runbooks/kustomization.yaml +++ b/runbooks/kustomization.yaml @@ -58,6 +58,7 @@ configMapGenerator: - cnpg-archiving=cnpg-archiving.md - cnpg-backup-missed=cnpg-backup-missed.md - cnpg-exporter=cnpg-exporter.md + - cnpg-primary-spread=cnpg-primary-spread.md - cnpg-wal-growth=cnpg-wal-growth.md - denied-flows=denied-flows.md - flux-reconcile=flux-reconcile.md diff --git a/runbooks/longhorn-capacity.md b/runbooks/longhorn-capacity.md index 7b20a1c..9258f12 100644 --- a/runbooks/longhorn-capacity.md +++ b/runbooks/longhorn-capacity.md @@ -50,10 +50,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/longhorn-degraded.md b/runbooks/longhorn-degraded.md index bd410b3..803ddb1 100644 --- a/runbooks/longhorn-degraded.md +++ b/runbooks/longhorn-degraded.md @@ -49,10 +49,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/longhorn-faulted.md b/runbooks/longhorn-faulted.md index 3c475df..7a3a5d0 100644 --- a/runbooks/longhorn-faulted.md +++ b/runbooks/longhorn-faulted.md @@ -50,10 +50,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/monitoring-capacity.md b/runbooks/monitoring-capacity.md index 440063c..3a56dad 100644 --- a/runbooks/monitoring-capacity.md +++ b/runbooks/monitoring-capacity.md @@ -34,10 +34,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/node-cpu.md b/runbooks/node-cpu.md index 6fdfd6c..3a5e4a6 100644 --- a/runbooks/node-cpu.md +++ b/runbooks/node-cpu.md @@ -31,10 +31,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/node-disk.md b/runbooks/node-disk.md index d7419b5..9c4b06c 100644 --- a/runbooks/node-disk.md +++ b/runbooks/node-disk.md @@ -38,10 +38,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/node-down.md b/runbooks/node-down.md index 732684c..6974d70 100644 --- a/runbooks/node-down.md +++ b/runbooks/node-down.md @@ -38,10 +38,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/node-io.md b/runbooks/node-io.md index a696393..6a85e04 100644 --- a/runbooks/node-io.md +++ b/runbooks/node-io.md @@ -47,10 +47,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/node-memory.md b/runbooks/node-memory.md index c7bf90a..78c6784 100644 --- a/runbooks/node-memory.md +++ b/runbooks/node-memory.md @@ -34,10 +34,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/pvc-capacity.md b/runbooks/pvc-capacity.md index 3137fe9..241d109 100644 --- a/runbooks/pvc-capacity.md +++ b/runbooks/pvc-capacity.md @@ -40,10 +40,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/targets-down.md b/runbooks/targets-down.md index 4f05924..73643ca 100644 --- a/runbooks/targets-down.md +++ b/runbooks/targets-down.md @@ -42,10 +42,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/update-rollback.md b/runbooks/update-rollback.md index 78209fc..2f49e77 100644 --- a/runbooks/update-rollback.md +++ b/runbooks/update-rollback.md @@ -36,10 +36,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to diff --git a/runbooks/vpn-down.md b/runbooks/vpn-down.md index 1cfb6d2..caa7765 100644 --- a/runbooks/vpn-down.md +++ b/runbooks/vpn-down.md @@ -48,10 +48,14 @@ that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' -Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and -defaults its current-context to k3s-**dev**, so a command without `--context` silently answers -about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) -Do not run two `dev.sh` invocations at once — they share one container and interfere. +Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and +picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev +until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means +the same thing next month as it does today; one that relies on the default does not, and the failure is +silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run +kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about +to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and +interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to