jakman85andClaude Opus 5
8e017fad91
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
190 lines
11 KiB
Markdown
190 lines
11 KiB
Markdown
# cnpg-primary-spread — Two or more CNPG primaries share one node
|
|
|
|
**Alert:** `CNPGPrimariesCoLocated` (warning, 30m)
|
|
**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg`
|
|
**Expression:** `count by (node) (<ready *-rw endpoints joined to cnpg_collector_up>) > 1`
|
|
**and** gated on every node being Ready:
|
|
`count(kube_node_status_condition{condition="Ready",status="true"} == 1) == count(kube_node_info)`
|
|
|
|
## What this means here
|
|
|
|
More than one CloudNativePG **primary** has been running on the same node for 30 minutes, while all
|
|
three nodes are Ready. There are exactly three CNPG Clusters on this platform — `keycloak/keycloak-db`,
|
|
`nextcloud-db/nextcloud-db` and `semaphore-db/semaphore-db` — so on a 3-node cluster the healthy shape is
|
|
one primary per node, and this alert says that shape has drifted.
|
|
|
|
**This is BLAST RADIUS, not an outage.** Each Cluster has `instances: 3` with one pod per node
|
|
(`podAntiAffinityType: required`) and one synchronous standby (`method: any`, `number: 1`,
|
|
`dataDurability: preferred`), so every one of them survives losing the shared node — you just get two or
|
|
three simultaneous PostgreSQL failovers instead of one. Nothing is broken right now, which is why it is a
|
|
`warning` with a long `for`.
|
|
|
|
**Why it matters anyway: this exact drift has already caused an outage.** All three Clusters used to run
|
|
`instances: 1` and all three primaries had drifted onto one node, and **on 2026-09-07 that node took SSO,
|
|
Nextcloud and Semaphore down together.** `instances: 3` means the same drift no longer causes an outage —
|
|
but if `keycloak-db`'s primary is co-located and that node dies, the SSO failover happens at the same
|
|
moment as two others, on nodes that are also absorbing Longhorn replica rebuilds.
|
|
|
|
**Why the drift happens and cannot be declared away: CNPG has no declarative primary-placement field.**
|
|
The primary lands wherever the last promotion put it — a node reboot, a switchover, an operator upgrade.
|
|
So this is avoidable drift that must be *corrected*, not prevented, and the alert re-arms itself so it
|
|
will be reported again rather than silently persisting.
|
|
|
|
**Why the node-readiness gate is in the expression.** During a real node outage, primaries legitimately
|
|
pile onto the survivors and there is nothing to do about it; alerting then would be noise at the worst
|
|
moment. The `and on ()` clause suppresses the rule unless every node is Ready, and re-arms when the node
|
|
returns.
|
|
|
|
This rule was proven in both directions on live prod data (2026-09-13): the expression returned
|
|
`{node="k3s-prod-1"} => 3` before the rebalance and a **throwaway twin with `for: 1m`** was observed going
|
|
`pending` at +70s and `firing` at +130s; after two switchovers the same expression returned an **empty
|
|
vector while the inner join still resolved all three primaries** — quiet because the condition is false,
|
|
not because the query went blind. It is also unit-tested in `alerts.promtool-test.yaml`
|
|
(run `./run-alert-tests.sh`) with four cases, mutation-checked so that deleting the `cnpg_collector_up`
|
|
join, the readiness gate, or shortening `for:` makes a test FAIL.
|
|
## Running the commands in this runbook
|
|
|
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
|
that hang is not the cluster being down. Every command below runs as:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
|
|
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
|
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
|
|
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
|
|
the same thing next month as it does today; one that relies on the default does not, and the failure is
|
|
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
|
|
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
|
|
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
|
|
interfere.
|
|
|
|
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
|
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
|
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
|
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
|
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
|
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
|
|
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
|
and it says so.
|
|
|
|
## First commands
|
|
|
|
Where the primaries are — the one command this alert is about:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get pods -A -l cnpg.io/instanceRole=primary \
|
|
-o custom-columns=NS:.metadata.namespace,POD:.metadata.name,NODE:.spec.nodeName'
|
|
```
|
|
|
|
Are all three Clusters actually healthy enough to move one? (Fully-qualified:
|
|
`clusters.postgresql.cnpg.io` — `cluster` is an ambiguous short name.)
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
|
|
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,INSTANCES:.status.instances,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,PHASE:.status.phase'
|
|
```
|
|
|
|
Is every node Ready (the gate the rule itself applies)?
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes -o wide'
|
|
```
|
|
|
|
Is there storage or I/O trouble that makes a switchover a bad idea right now?
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
|
|
-o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy'
|
|
```
|
|
|
|
```
|
|
ssh k3s-prod-1 'cat /proc/pressure/io'
|
|
```
|
|
|
|
And confirm DR is intact before you deliberately move a primary — the primary owns the backup
|
|
catalogue, and **fully-qualified names matter**: `kubectl get backup -A` prints "No resources found" on
|
|
this cluster even with 27 CNPG backups present, because three CRDs register the kind `Backup` and the
|
|
short name resolves to Longhorn's (zero objects):
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get backups.postgresql.cnpg.io -A | tail -10; echo; \
|
|
kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
|
|
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint'
|
|
```
|
|
|
|
## Likely causes, most likely first
|
|
|
|
### 1. A node rebooted and the primaries never moved back
|
|
The dominant cause. Promotions during the reboot put two primaries on a survivor, and CNPG has no reason
|
|
to move them back afterwards. Remediation: a deliberate switchover, below.
|
|
|
|
### 2. A rolling operator or Postgres upgrade
|
|
`primaryUpdateMethod: switchover` is set deliberately (the CRD default `restart` is a
|
|
delete-and-recreate of the *primary*, i.e. tens of seconds with no Postgres — an SSO outage on
|
|
`keycloak-db`). Switchovers during an upgrade are correct behaviour and leave the primaries wherever
|
|
they landed.
|
|
|
|
### 3. One node was cordoned or unschedulable for a while
|
|
Same mechanism. Note that `podAntiAffinityType: required` means an instance stays **Pending** until a
|
|
node returns rather than co-locating — that is the accepted cost of the 2026-09-07 fix, and it is why
|
|
the *instances* stay spread even when the *primaries* do not.
|
|
|
|
### 4. A Longhorn or I/O event caused a health-check failover
|
|
A primary whose storage stalled gets demoted. Cross-read `/runbooks/node-io` and
|
|
`/runbooks/longhorn-degraded` — if that is why, fix the node before rebalancing.
|
|
|
|
### 5. The alert is firing during a genuine node outage
|
|
It should not be able to: the readiness gate suppresses it. If it fires anyway with a node NotReady, the
|
|
gate is broken — that is a monitoring bug worth fixing, since a rule that cannot stop is fatigue.
|
|
|
|
## Remediation — move one primary, deliberately
|
|
|
|
The cnpg plugin is **NOT in the devcontainer image**; fetch the `kubectl-cnpg` release matching the
|
|
operator version (CNPG **1.30.0** here) before you start.
|
|
|
|
```
|
|
kubectl cnpg promote -n <namespace> <cluster> <instance-on-an-idle-node>
|
|
```
|
|
|
|
**Do ONE CLUSTER AT A TIME and wait for `readyInstances: 3` before the next.** A switchover interrupts
|
|
writes to that database for about **9 seconds** — measured on both clusters moved on 2026-09-13. Nine
|
|
seconds of `keycloak-db` unavailability is a brief SSO interruption; doing two at once is an avoidable
|
|
compound one.
|
|
|
|
Pick the target by reading the primaries table above and choosing an instance that already sits on a node
|
|
holding no other primary. Then re-run the table to confirm the new distribution, and let the alert clear
|
|
on its own.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **Do not rebalance during any other incident.** A switchover is a small, deliberate write interruption,
|
|
and it is the wrong thing to add to a node that is already under I/O pressure or to a cluster with
|
|
degraded Longhorn volumes. This alert has a 30-minute `for` precisely because it is never urgent.
|
|
- **Never switch over a Cluster whose `readyInstances` is below 3.** You would be promoting into a
|
|
cluster that has no standby left to fall back on.
|
|
- **Never change `dataDurability` to `required`** while thinking about durability here: it is a strict
|
|
quorum, and one degraded standby would then **block writes** entirely on the SSO database.
|
|
- **Do not `kubectl delete pod` the primary to force a failover.** That is `restart` semantics — the
|
|
thing `primaryUpdateMethod: switchover` exists to avoid.
|
|
- **Check `ContinuousArchiving` and `firstRecoverabilityPoint` first** (`/runbooks/cnpg-archiving`). If
|
|
archiving is broken, that is a real countdown to a disk-full PANIC and it outranks blast-radius
|
|
tidying completely.
|
|
- Escalate to the operator for the switchover itself if the affected Cluster is `keycloak-db`: the brief
|
|
write interruption is user-visible across SSO, kubectl OIDC, Grafana, Headlamp and Gitea.
|
|
|
|
---
|
|
Source of truth: `runbooks/cnpg-primary-spread.md`. Index: `/runbooks/`
|
|
Records: `platform/cnpg/alerts.yaml` (the rule's own both-directions proof on live prod data and its
|
|
promtool tests); commits `f338e91` (instances 3), `e8d8601` (required anti-affinity + switchover),
|
|
`cac413e` (synchronous replication, `dataDurability: preferred`); the 2026-09-07 single-node SSO outage.
|