jakman85andClaude Opus 5
8e017fad91
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
169 lines
9.6 KiB
Markdown
169 lines
9.6 KiB
Markdown
# keycloak-down — The Keycloak IdP is down (SSO is unavailable)
|
|
|
|
**Alert:** `KeycloakDown` (critical, 5m)
|
|
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.identity`
|
|
**Expression:**
|
|
```
|
|
max(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+", condition="true"}) == 0
|
|
or absent(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+"})
|
|
```
|
|
|
|
## What this means here
|
|
|
|
No ready Keycloak **server** pod for five minutes. The regex is deliberately scoped to
|
|
`keycloak-[0-9]+` (the StatefulSet pods `keycloak-0`, `keycloak-1`) because the `keycloak` namespace also
|
|
holds the operator (`keycloak-operator-*`) and the database (`keycloak-db-*`); an unscoped `max()` would
|
|
be satisfied by those being ready and would never fire for the IdP itself. That scoping was a fix — the
|
|
original matched `.*postgres.*`.
|
|
|
|
## Blast radius — read this before triaging anything else
|
|
|
|
Keycloak is the root of trust. With SSO down, all of these are down or degraded **at once**:
|
|
|
|
- **kubectl OIDC login** (the public `kubernetes` client with PKCE, groups → RBAC)
|
|
- **Grafana** (OIDC), **Headlamp** (forwards the user's id_token to the apiserver), **Gitea**
|
|
- **forward-auth** via oauth2-proxy, and the five per-app oauth2-proxy sidecars (pihole, jackett,
|
|
qbittorrent, radarr, sonarr) — these **fail closed**, 302ing to a Keycloak that is not there
|
|
|
|
**You can still get in.** The certificate-based kubeconfig is untouched by the OIDC work and remains a
|
|
fully independent break-glass path — that is a deliberate design property, not an accident. From this
|
|
workstation the working non-interactive path is the dev-container kubeconfig; the host `k3s-oidc`
|
|
context will hang on a browser prompt, which is not the cluster being down.
|
|
|
|
**And you can still read this page.** The runbook route is deliberately **public, with no
|
|
forward-auth**, precisely so that `keycloak-down` is readable by an operator who cannot log in. If that
|
|
ever changes, this runbook becomes unreachable exactly when it is needed.
|
|
## Running the commands in this runbook
|
|
|
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
|
that hang is not the cluster being down. Every command below runs as:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
|
|
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
|
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
|
|
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
|
|
the same thing next month as it does today; one that relies on the default does not, and the failure is
|
|
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
|
|
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
|
|
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
|
|
interfere.
|
|
|
|
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
|
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
|
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
|
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
|
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
|
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
|
|
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
|
and it says so.
|
|
|
|
## First commands
|
|
|
|
Everything in the namespace, in one view — server, operator and database together:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n keycloak get pods -o wide; echo; \
|
|
kubectl --context k3s-prod -n keycloak get statefulsets,deployments'
|
|
```
|
|
|
|
Is the database the real cause? (It usually is. Fully-qualified: `clusters.postgresql.cnpg.io`.)
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -n keycloak \
|
|
-o custom-columns=NAME:.metadata.name,INSTANCES:.status.instances,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,PHASE:.status.phase'
|
|
```
|
|
|
|
The server's own complaint:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n keycloak logs keycloak-0 --tail=120 2>&1 | tail -50; echo ====; \
|
|
kubectl --context k3s-prod -n keycloak describe pod keycloak-0 | sed -n "/Events:/,\$p"'
|
|
```
|
|
|
|
The operator's view of the CR:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n keycloak get keycloaks.k8s.keycloak.org -o yaml | grep -A20 "status:"'
|
|
```
|
|
|
|
Is it reachable at all, from outside, with an explicit SNI (in-pod HTTPS tests to these hostnames are
|
|
misleading here):
|
|
|
|
```
|
|
curl -s -o /dev/null -w '%{http_code}\n' --resolve keycloak.home.keithjackson.dev:443:192.168.2.20 \
|
|
https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
|
|
```
|
|
|
|
## Likely causes, most likely first
|
|
|
|
### 1. The CNPG database is not available
|
|
Keycloak cannot start without Postgres. This is the highest-probability cause and it has its own runbook:
|
|
`/runbooks/keycloak-db`. Note that `keycloak-db` is a 3-instance CNPG Cluster with
|
|
`primaryUpdateMethod: switchover` and synchronous replication at `dataDurability: preferred` —
|
|
`preferred` specifically so that one degraded standby does **not** block writes, which on this database
|
|
would be a total SSO write outage.
|
|
|
|
### 2. A node that held the Keycloak pods went away
|
|
Both Keycloak pods and all three `keycloak-db` instances previously ran on **one node** — all three CNPG
|
|
primaries had drifted onto `k3s-prod-2` — and **on 2026-09-07 that node took SSO, Nextcloud and
|
|
Semaphore down together**. The fixes were `instances: 3` on every Cluster and
|
|
`podAntiAffinityType: required` (CNPG's default `preferred` is only a scheduler hint and co-locates on
|
|
the path that matters — recreating an instance after node loss). Check `-o wide` above: if everything is
|
|
on one node, that is a live re-occurrence of the same defect. Go to `/runbooks/node-down`.
|
|
|
|
### 3. A HelmRelease / operator upgrade went bad
|
|
Read `/runbooks/update-rollback`. Under `primaryUpdateMethod: switchover` the CNPG operator **rejects an
|
|
update that changes `imageName` and a postgresql parameter in the same reconcile** — a stuck reconcile
|
|
with that message is this.
|
|
|
|
### 4. TLS to the issuer is broken
|
|
Keycloak's clients validate the issuer's certificate. While the wildcard was LE-**staging**, in-cluster
|
|
clients rejected it with `x509: certificate signed by unknown authority`. If `/runbooks/cert-issuance` or
|
|
`/runbooks/cert-expiry` is also firing, fix that first.
|
|
|
|
### 5. Logins fail but Keycloak is UP
|
|
Then this is the **wrong alert and the wrong runbook**, and the distinction has cost this cluster dearly.
|
|
On 2026-09-06 every OIDC login on prod failed — *"Invalid client or Invalid client credentials"* — with
|
|
**all 12 clients holding the literal `(env:...)` placeholder as their secret**, while
|
|
`KeycloakDown`/`KeycloakDatabaseDown` were quiet, the import Job reported success, the CR read
|
|
`Done=True`, and the conformance test that asserts `Done=True` **passed throughout**. The only observable
|
|
was a human failing to log in. Go to `/runbooks/keycloak-client-secrets`.
|
|
Similarly, a `redirect_uri` mismatch after a hostname change is not an outage — same runbook.
|
|
And once, MFA broke every platform-admin login on its own: conditional-OTP used a non-existent config
|
|
key, giving `"Invalid role name: null"` → 403, fixed to `condUserRole: platform-admin`.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **Escalate immediately.** SSO down is a platform-wide outage; treat it as such.
|
|
- **Use the certificate-based kubeconfig** (the dev-container path) rather than trying to fix OIDC login
|
|
first. Do not burn the outage on getting `kubectl` to authenticate through the thing that is down.
|
|
- **Do not edit the realm to "reset" anything.** The realm import is a **ONE-SHOT that fails open**: the
|
|
operator logs *"Job already executed - not recreating"*, so a git change to
|
|
`keycloak-realm-import.yaml` does **not** re-import. The documented path is `kcadm.sh` inside
|
|
`keycloak-0` (admin credentials in the Secret `keycloak-admin-credentials`; `kcadm` is present,
|
|
`curl`/`python3` are **not**) **and** fixing git for reproducibility. Doing only one of the two is how
|
|
drift is created.
|
|
- **Do not delete the Keycloak CR or the StatefulSet.** Recreating the IdP is a restore operation, not a
|
|
restart.
|
|
- **Do not touch `keycloak-db` volumes.** If the database is the cause, read `/runbooks/keycloak-db` and
|
|
`/runbooks/cnpg-archiving`; only the Barman archive is a backup, standbys replicate deletions
|
|
faithfully.
|
|
- Break-glass local admins exist for gitea, nextcloud and semaphore. Nextcloud has **no native
|
|
group→admin mapping**, so platform-admins there are promoted by hand.
|
|
|
|
---
|
|
Source of truth: `runbooks/keycloak-down.md`. Index: `/runbooks/`
|
|
Records: `DECISIONS-LOG.md` (the 2026-09-06 client-secret lockout; the 2026-09-07 single-node SSO
|
|
outage); commits `f338e91`, `e8d8601`, `cac413e` (CNPG instances/anti-affinity/switchover/sync).
|