jakman85andClaude Opus 5
8e017fad91
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
153 lines
8.9 KiB
Markdown
153 lines
8.9 KiB
Markdown
# update-rollback — A HelmRelease failed its health gate and is rolling back
|
|
|
|
**Alert:** `UpdateHealthGateFailed` (critical, 10m)
|
|
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.automated-updates`
|
|
**Expression:** `max by (namespace, name) (gotk_reconcile_condition{kind="HelmRelease", type="Ready", status="False"}) == 1`
|
|
|
|
## What this means here
|
|
|
|
helm-controller holds a HelmRelease at Ready=False while a failed upgrade remediates and rolls back to
|
|
the last-good revision. This is the **R-UPD-03 safety rail firing** — the bad-update auto-rollback
|
|
working — so the alert is a report that the platform protected itself, not necessarily that something
|
|
is broken right now.
|
|
|
|
Read the expression honestly, though: it matches **any** HelmRelease at Ready=False, not only one that
|
|
is mid-rollback. The rule's own comment flags this as a `TODO(bring-up)`: tighten it with the `reason`
|
|
label (`UpgradeFailed` / `RolledBack`) once confirmed against the deployed helm-controller. So a
|
|
HelmRelease that simply cannot install — a bad value, a missing CRD, a crashlooping pod — lights this
|
|
`critical` too. Your first job is to tell those two cases apart, because they need opposite responses:
|
|
a rollback is self-healing and wants a git revert; a failed *install* is a stall and may be gating an
|
|
entire tier (see `/runbooks/flux-reconcile`).
|
|
|
|
Real precedent that rollback does work here: during the 2026-09-01 Cilium incident the Helm release
|
|
history shows `v8 e83bf695 → v9 eebd6c88 (the bad render) → v10 e83bf695 (identical to v8)`, and the
|
|
release secret for v9 reads `failed`. Helm's wait genuinely failed and the release was remediated — a
|
|
claim that remediation never engaged was **measured wrong**. That is why the cluster self-healed in
|
|
about six minutes.
|
|
**This alert is currently INERT.** `count(gotk_reconcile_condition)` returned **0 series** on prod on
|
|
2026-09-13 — there is no Flux-controller scrape job among Prometheus's 19 jobs, so every `gotk_*` rule
|
|
evaluates against nothing. Read the HelmRelease CRs directly with the commands below, and see
|
|
`/runbooks/targets-down` for the full list of inert rule groups.
|
|
## Running the commands in this runbook
|
|
|
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
|
that hang is not the cluster being down. Every command below runs as:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
|
|
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
|
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
|
|
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
|
|
the same thing next month as it does today; one that relies on the default does not, and the failure is
|
|
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
|
|
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
|
|
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
|
|
interfere.
|
|
|
|
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
|
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
|
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
|
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
|
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
|
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
|
|
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
|
and it says so.
|
|
|
|
## First commands
|
|
|
|
Which release, and what happened:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get helmreleases.helm.toolkit.fluxcd.io -A \
|
|
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,REASON:.status.conditions[?\(@.type==\"Ready\"\)].reason,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,LASTSUCCESS:.status.lastAppliedRevision'
|
|
```
|
|
|
|
The Helm revision history — this is what distinguishes "rolled back" from "never installed":
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n <namespace> get secrets -l owner=helm --sort-by=.metadata.creationTimestamp \
|
|
-o custom-columns=NAME:.metadata.name,STATUS:.metadata.labels.status,VERSION:.metadata.labels.version'
|
|
```
|
|
|
|
The controller's reasoning:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n flux-system logs deploy/helm-controller --tail=120 2>&1 | grep -iE "<release>|rollback|remediat|upgrade failed" | tail -30'
|
|
```
|
|
|
|
What the workload itself is doing:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n <namespace> get pods -o wide; \
|
|
kubectl --context k3s-prod -n <namespace> get events --sort-by=.lastTimestamp | tail -20'
|
|
```
|
|
|
|
If the release touches backups or databases, verify them with **fully-qualified names** before deciding
|
|
anything is fine — `kubectl get backup -A` prints "No resources found" on this cluster even with six
|
|
Velero and 27 CNPG backups present:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get backups.velero.io -A; echo; \
|
|
kubectl --context k3s-prod get backups.postgresql.cnpg.io -A; echo; \
|
|
kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A'
|
|
```
|
|
|
|
## Likely causes, most likely first
|
|
|
|
### 1. A chart upgrade shipped an incompatible image or value
|
|
The recorded instances: **kube-prometheus-stack 65 → 88 brought Grafana 13, which OOM-looped** and
|
|
became "the hidden blocker that stalled the whole kps upgrade and the platform tier" (fixed by raising
|
|
Grafana 256 → 512 → 768Mi and Prometheus 2 → 3Gi, commit `b2b1aa5`); and **headlamp-0.40.1** shipped a
|
|
`-session-ttl` chart/image desync that crashlooped. Remediation: pin the previous version in git
|
|
(`update.pin` is the documented escape hatch, docs/11) and push. The rollback has already restored
|
|
service; the git change stops it happening again next interval.
|
|
|
|
### 2. Not a rollback at all — a first install that cannot succeed
|
|
`REASON` will be an install/dependency failure and there will be no earlier `deployed` Helm secret. If
|
|
the release is inside the gating `platform` tier, everything downstream is frozen behind it — go to
|
|
`/runbooks/flux-reconcile`, which covers the "no matches for kind" / CRD-ordering class.
|
|
|
|
### 3. A timeout, not a failure
|
|
`platform` has `timeout: 5m` while Longhorn's HelmRelease has `timeout: 15m` — **the tier can time out
|
|
while a legitimate Longhorn upgrade is still running.** Give it another interval before declaring a
|
|
failure, and check whether pods are still progressing.
|
|
|
|
### 4. The rollback itself is stuck
|
|
Helm cannot roll back cleanly if the upgrade already mutated something immutable (a StatefulSet
|
|
selector, a PVC). Then Ready=False is permanent until an operator intervenes.
|
|
|
|
### 5. A post-upgrade health check is failing for an unrelated reason
|
|
A node under I/O pressure makes pods slow to become Ready and the upgrade's wait expires. Cross-read
|
|
`/runbooks/node-io`.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **Get the operator if the release is `cilium` or anything in the gating `platform` tier.** A CNI
|
|
rollback affects every node's datapath.
|
|
- **Do NOT enable `driftDetection.mode: enabled` on a CNI.** During the 2026-09-01 incident the Helm
|
|
storage manifest *was* the bad render, so drift correction would have actively re-asserted the broken
|
|
value. This is a recorded conclusion, not a preference.
|
|
- **Do not `helm rollback` by hand.** helm-controller owns the release; a manual rollback creates drift
|
|
it will fight. Use git.
|
|
- **Do not suspend the HelmRelease to silence the alert** — that trades a visible failure for
|
|
`/runbooks/flux-suspended` and silent drift.
|
|
- If the rolled-back release is CNPG or Keycloak, expect knock-on SSO effects and read
|
|
`/runbooks/keycloak-down`; note also that a CNPG update under `primaryUpdateMethod: switchover`
|
|
cannot change `imageName` and a postgresql parameter in the same reconcile — the operator rejects it.
|
|
|
|
---
|
|
Source of truth: `runbooks/update-rollback.md`. Index: `/runbooks/`
|
|
Records: `DECISIONS-LOG.md` "2026-09-01 — INCIDENT: merging PR #4 briefly broke Cilium" (the measured
|
|
Helm revision history and the drift-detection conclusion); the Grafana OOM and headlamp desync.
|