home-network-k3s-impl/runbooks/update-rollback.md

# update-rollback — A HelmRelease failed its health gate and is rolling back

**Alert:** `UpdateHealthGateFailed` (critical, 10m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.automated-updates`
**Expression:** `max by (namespace, name) (gotk_reconcile_condition{kind="HelmRelease", type="Ready", status="False"}) == 1`

## What this means here

helm-controller holds a HelmRelease at Ready=False while a failed upgrade remediates and rolls back to
the last-good revision. This is the **R-UPD-03 safety rail firing** — the bad-update auto-rollback
working — so the alert is a report that the platform protected itself, not necessarily that something
is broken right now.

Read the expression honestly, though: it matches **any** HelmRelease at Ready=False, not only one that
is mid-rollback. The rule's own comment flags this as a `TODO(bring-up)`: tighten it with the `reason`
label (`UpgradeFailed` / `RolledBack`) once confirmed against the deployed helm-controller. So a
HelmRelease that simply cannot install — a bad value, a missing CRD, a crashlooping pod — lights this
`critical` too. Your first job is to tell those two cases apart, because they need opposite responses:
a rollback is self-healing and wants a git revert; a failed *install* is a stall and may be gating an
entire tier (see `/runbooks/flux-reconcile`).

Real precedent that rollback does work here: during the 2026-09-01 Cilium incident the Helm release
history shows `v8 e83bf695 → v9 eebd6c88 (the bad render) → v10 e83bf695 (identical to v8)`, and the
release secret for v9 reads `failed`. Helm's wait genuinely failed and the release was remediated — a
claim that remediation never engaged was **measured wrong**. That is why the cluster self-healed in
about six minutes.
**This alert is currently INERT.** `count(gotk_reconcile_condition)` returned **0 series** on prod on
2026-09-13 — there is no Flux-controller scrape job among Prometheus's 19 jobs, so every `gotk_*` rule
evaluates against nothing. Read the HelmRelease CRs directly with the commands below, and see
`/runbooks/targets-down` for the full list of inert rule groups.
## Running the commands in this runbook

`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.

## First commands

Which release, and what happened:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get helmreleases.helm.toolkit.fluxcd.io -A \
     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,REASON:.status.conditions[?\(@.type==\"Ready\"\)].reason,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,LASTSUCCESS:.status.lastAppliedRevision'
```

The Helm revision history — this is what distinguishes "rolled back" from "never installed":

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n <namespace> get secrets -l owner=helm --sort-by=.metadata.creationTimestamp \
     -o custom-columns=NAME:.metadata.name,STATUS:.metadata.labels.status,VERSION:.metadata.labels.version'
```

The controller's reasoning:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n flux-system logs deploy/helm-controller --tail=120 2>&1 | grep -iE "<release>|rollback|remediat|upgrade failed" | tail -30'
```

What the workload itself is doing:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n <namespace> get pods -o wide; \
   kubectl --context k3s-prod -n <namespace> get events --sort-by=.lastTimestamp | tail -20'
```

If the release touches backups or databases, verify them with **fully-qualified names** before deciding
anything is fine — `kubectl get backup -A` prints "No resources found" on this cluster even with six
Velero and 27 CNPG backups present:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get backups.velero.io -A; echo; \
   kubectl --context k3s-prod get backups.postgresql.cnpg.io -A; echo; \
   kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A'
```

## Likely causes, most likely first

### 1. A chart upgrade shipped an incompatible image or value
The recorded instances: **kube-prometheus-stack 65 → 88 brought Grafana 13, which OOM-looped** and
became "the hidden blocker that stalled the whole kps upgrade and the platform tier" (fixed by raising
Grafana 256 → 512 → 768Mi and Prometheus 2 → 3Gi, commit `b2b1aa5`); and **headlamp-0.40.1** shipped a
`-session-ttl` chart/image desync that crashlooped. Remediation: pin the previous version in git
(`update.pin` is the documented escape hatch, docs/11) and push. The rollback has already restored
service; the git change stops it happening again next interval.

### 2. Not a rollback at all — a first install that cannot succeed
`REASON` will be an install/dependency failure and there will be no earlier `deployed` Helm secret. If
the release is inside the gating `platform` tier, everything downstream is frozen behind it — go to
`/runbooks/flux-reconcile`, which covers the "no matches for kind" / CRD-ordering class.

### 3. A timeout, not a failure
`platform` has `timeout: 5m` while Longhorn's HelmRelease has `timeout: 15m` — **the tier can time out
while a legitimate Longhorn upgrade is still running.** Give it another interval before declaring a
failure, and check whether pods are still progressing.

### 4. The rollback itself is stuck
Helm cannot roll back cleanly if the upgrade already mutated something immutable (a StatefulSet
selector, a PVC). Then Ready=False is permanent until an operator intervenes.

### 5. A post-upgrade health check is failing for an unrelated reason
A node under I/O pressure makes pods slow to become Ready and the upgrade's wait expires. Cross-read
`/runbooks/node-io`.

## When to escalate or stop

- **Get the operator if the release is `cilium` or anything in the gating `platform` tier.** A CNI
  rollback affects every node's datapath.
- **Do NOT enable `driftDetection.mode: enabled` on a CNI.** During the 2026-09-01 incident the Helm
  storage manifest *was* the bad render, so drift correction would have actively re-asserted the broken
  value. This is a recorded conclusion, not a preference.
- **Do not `helm rollback` by hand.** helm-controller owns the release; a manual rollback creates drift
  it will fight. Use git.
- **Do not suspend the HelmRelease to silence the alert** — that trades a visible failure for
  `/runbooks/flux-suspended` and silent drift.
- If the rolled-back release is CNPG or Keycloak, expect knock-on SSO effects and read
  `/runbooks/keycloak-down`; note also that a CNPG update under `primaryUpdateMethod: switchover`
  cannot change `imageName` and a postgresql parameter in the same reconcile — the operator rejects it.

---
Source of truth: `runbooks/update-rollback.md`. Index: `/runbooks/`
Records: `DECISIONS-LOG.md` "2026-09-01 — INCIDENT: merging PR #4 briefly broke Cilium" (the measured
Helm revision history and the drift-detection conclusion); the Grafana OOM and headlamp desync.