jakman85andClaude Opus 5
44983095e1
Every platform-health alert rule carries `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations over 34 distinct names, across platform/monitoring/alerts.yaml, platform/monitoring/pvc-capacity-alerts.yaml, platform/cnpg/alerts.yaml and platform/kubevirt/alerts.yaml. There was no runbooks/ directory and no HTTPRoute serving that path: an in-cluster probe of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404. Every alert that reached a human handed them a dead link at the exact moment the documentation is read, and the annotation made it look solved — the same green-surface-over-absent-mechanism class as the import Job that reported success while storing placeholder secrets. No alerts.yaml is touched and no runbook_url value changes. The existing URL scheme is made to resolve instead. WHAT LANDS - runbooks/<34>.md — one per name, each grounded in this cluster's real topology and, where one exists, its real incident history: the 2026-09-07 k3s-prod-1 etcd I/O starvation (measured PSI 11.36 vs 0.75/1.29, the exact journal lines, the four mechanisms proven inert), the Longhorn reboot tax and the freeze/thaw of concurrent-replica-rebuild-per-node-limit, Cilium's once-resolved egress interface cache (Ifindex 3 against a live 1780) and the absence of OSS failover, the CNPG 38-hour ContinuousArchiving=False with its measured 6.8 MiB/h disk-full clock and the 2026-09-12 aggregation-scope fix, the 2026-09-06 Keycloak lockout in which all 12 clients held literal placeholder secrets behind an entirely green cluster, the Velero R2 x-amz-tagging NotImplemented incompatibility and its v1.14.3-rc.2 pin, and swap re-enabling itself across reboots via systemd-gpt-auto-generator. Each page states what the alert means HERE, the first diagnostic commands, likely causes ordered by probability with a remediation each, and when to escalate or stop. - Every page uses FULLY-QUALIFIED resource names and says why: three CRDs register the kind `Backup` here, the bare short name resolves to Longhorn's (zero objects), so `kubectl get backup -A` prints "No resources found" on a cluster holding 6 Velero and 27 CNPG backups. Acting on that false negative is how a CLI quirk becomes data loss. - Every page states that kubectl only works via `infra/devcontainer/dev.sh run bash -c '...'` with an explicit `--context k3s-prod`, because dev.sh defaults its current-context to k3s-dev. - Pages also record which alert groups are INERT today (measured 2026-09-13: no Longhorn, cert-manager, Flux-controller or Hubble scrape job exists, so 13 rules evaluate against nothing) so nobody reads their silence as health. SERVING — runbooks/serving/ plus clusters/base/runbooks.yaml A two-replica nginx-unprivileged serving a ConfigMap generated from runbooks/*.md, at the apex `/runbooks` prefix, in its own non-gating post-platform Flux tier (dependsOn platform-config, wait:false) — the platform/homepage pattern, file for file. - The service catalog cannot express this route: ingressRoutes[].host is pattern-bound to a single DNS label plus the apex, so the bare apex is unrepresentable, and there is no path field. - Rejected: redirecting to the Gitea web UI. A runbook is read when the platform is broken, and Gitea is an app-tier workload whose own dependencies several of these runbooks document; Gateway API's RequestRedirect also has no "append .md" modifier, so it would need 34 hand-maintained ReplaceFullPath rules. - configMapGenerator maps each file to an EXTENSIONLESS key (node-down=node-down.md), so /runbooks/node-down is a plain static lookup with no rewrite rule. Served as text/plain: the namespace has DNS-only egress, so a CDN markdown renderer could not load anyway. - The tier declares NO postBuild.substituteFrom, deliberately: envsubst would run over the prose and silently empty every ${...} shell example (it fails open — the 2026-09-01 emptied Cilium k8sServiceHost). The HTTPRoute omits `hostnames:` and inherits the https-apex listener's, so nothing needs substituting. AUTH — PUBLIC, deliberately auth-mode=none. Gating these pages behind oauth2-proxy/Keycloak would make runbooks/keycloak-down unreadable when Keycloak is down and runbooks/keycloak-client-secrets unreadable during the exact 2026-09-06 failure mode. An operator who cannot log in is this route's primary reader. Content is operational procedure only, no secrets. The rationale is recorded in an annotation on the HTTPRoute so the route is left OUT of the Gateway ext-authz attachment when forward-auth enforcement lands. PROVEN — in-cluster probe through the Gateway on prod /runbooks/node-down HTTP 200 8613 bytes text/plain; charset=utf-8 /runbooks/keycloak-down HTTP 200 9378 bytes text/plain; charset=utf-8 /runbooks/cnpg-archiving HTTP 200 11729 bytes text/plain; charset=utf-8 /runbooks/longhorn-faulted HTTP 200 9135 bytes text/plain; charset=utf-8 /runbooks/vpn-down HTTP 200 11092 bytes text/plain; charset=utf-8 /runbooks/backup-failure HTTP 200 9604 bytes text/plain; charset=utf-8 HTTPRoute Accepted by io.cilium/gateway-controller on sectionName https-apex; the `/runbooks` prefix wins over homepage's apex `/` by Gateway API longest-path precedence. GUARD — conformance/tests/l0-static/alert-runbooks-resolvable (R-OBS-04) 363 offline assertions. It checks BOTH halves, because the regression is silent in both directions: an alert with a new runbook_url and no runbook links to a 404, AND a runbook committed without its configMapGenerator entry is in git, reviewed, merged, and still 404s in the cluster. Also asserts a quality floor so a stub cannot make the link look fixed, and that route/listener/nginx prefix agree with the annotations. Proven to fail, then reverted: a bogus runbook_url (2 FAIL), a runbook removed from the generator (1 FAIL), and a 31-byte stub (7 FAIL). Co-Authored-By: Claude Opus 5 <[email protected]>
163 lines
9.2 KiB
Markdown
163 lines
9.2 KiB
Markdown
# keycloak-down — The Keycloak IdP is down (SSO is unavailable)
|
|
|
|
**Alert:** `KeycloakDown` (critical, 5m)
|
|
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.identity`
|
|
**Expression:**
|
|
```
|
|
max(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+", condition="true"}) == 0
|
|
or absent(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+"})
|
|
```
|
|
|
|
## What this means here
|
|
|
|
No ready Keycloak **server** pod for five minutes. The regex is deliberately scoped to
|
|
`keycloak-[0-9]+` (the StatefulSet pods `keycloak-0`, `keycloak-1`) because the `keycloak` namespace also
|
|
holds the operator (`keycloak-operator-*`) and the database (`keycloak-db-*`); an unscoped `max()` would
|
|
be satisfied by those being ready and would never fire for the IdP itself. That scoping was a fix — the
|
|
original matched `.*postgres.*`.
|
|
|
|
## Blast radius — read this before triaging anything else
|
|
|
|
Keycloak is the root of trust. With SSO down, all of these are down or degraded **at once**:
|
|
|
|
- **kubectl OIDC login** (the public `kubernetes` client with PKCE, groups → RBAC)
|
|
- **Grafana** (OIDC), **Headlamp** (forwards the user's id_token to the apiserver), **Gitea**
|
|
- **forward-auth** via oauth2-proxy, and the five per-app oauth2-proxy sidecars (pihole, jackett,
|
|
qbittorrent, radarr, sonarr) — these **fail closed**, 302ing to a Keycloak that is not there
|
|
|
|
**You can still get in.** The certificate-based kubeconfig is untouched by the OIDC work and remains a
|
|
fully independent break-glass path — that is a deliberate design property, not an accident. From this
|
|
workstation the working non-interactive path is the dev-container kubeconfig; the host `k3s-oidc`
|
|
context will hang on a browser prompt, which is not the cluster being down.
|
|
|
|
**And you can still read this page.** The runbook route is deliberately **public, with no
|
|
forward-auth**, precisely so that `keycloak-down` is readable by an operator who cannot log in. If that
|
|
ever changes, this runbook becomes unreachable exactly when it is needed.
|
|
## Running the commands in this runbook
|
|
|
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
|
that hang is not the cluster being down. Every command below runs as:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
|
|
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
|
|
defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
|
|
about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
|
|
Do not run two `dev.sh` invocations at once — they share one container and interfere.
|
|
|
|
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
|
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
|
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
|
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
|
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
|
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
|
|
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
|
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
|
Where this runbook gives a mutating command it is for an operator, and it says so.
|
|
|
|
## First commands
|
|
|
|
Everything in the namespace, in one view — server, operator and database together:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n keycloak get pods -o wide; echo; \
|
|
kubectl --context k3s-prod -n keycloak get statefulsets,deployments'
|
|
```
|
|
|
|
Is the database the real cause? (It usually is. Fully-qualified: `clusters.postgresql.cnpg.io`.)
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -n keycloak \
|
|
-o custom-columns=NAME:.metadata.name,INSTANCES:.status.instances,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,PHASE:.status.phase'
|
|
```
|
|
|
|
The server's own complaint:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n keycloak logs keycloak-0 --tail=120 2>&1 | tail -50; echo ====; \
|
|
kubectl --context k3s-prod -n keycloak describe pod keycloak-0 | sed -n "/Events:/,\$p"'
|
|
```
|
|
|
|
The operator's view of the CR:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n keycloak get keycloaks.k8s.keycloak.org -o yaml | grep -A20 "status:"'
|
|
```
|
|
|
|
Is it reachable at all, from outside, with an explicit SNI (in-pod HTTPS tests to these hostnames are
|
|
misleading here):
|
|
|
|
```
|
|
curl -s -o /dev/null -w '%{http_code}\n' --resolve keycloak.home.keithjackson.dev:443:192.168.2.20 \
|
|
https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
|
|
```
|
|
|
|
## Likely causes, most likely first
|
|
|
|
### 1. The CNPG database is not available
|
|
Keycloak cannot start without Postgres. This is the highest-probability cause and it has its own runbook:
|
|
`/runbooks/keycloak-db`. Note that `keycloak-db` is a 3-instance CNPG Cluster with
|
|
`primaryUpdateMethod: switchover` and synchronous replication at `dataDurability: preferred` —
|
|
`preferred` specifically so that one degraded standby does **not** block writes, which on this database
|
|
would be a total SSO write outage.
|
|
|
|
### 2. A node that held the Keycloak pods went away
|
|
Both Keycloak pods and all three `keycloak-db` instances previously ran on **one node** — all three CNPG
|
|
primaries had drifted onto `k3s-prod-2` — and **on 2026-09-07 that node took SSO, Nextcloud and
|
|
Semaphore down together**. The fixes were `instances: 3` on every Cluster and
|
|
`podAntiAffinityType: required` (CNPG's default `preferred` is only a scheduler hint and co-locates on
|
|
the path that matters — recreating an instance after node loss). Check `-o wide` above: if everything is
|
|
on one node, that is a live re-occurrence of the same defect. Go to `/runbooks/node-down`.
|
|
|
|
### 3. A HelmRelease / operator upgrade went bad
|
|
Read `/runbooks/update-rollback`. Under `primaryUpdateMethod: switchover` the CNPG operator **rejects an
|
|
update that changes `imageName` and a postgresql parameter in the same reconcile** — a stuck reconcile
|
|
with that message is this.
|
|
|
|
### 4. TLS to the issuer is broken
|
|
Keycloak's clients validate the issuer's certificate. While the wildcard was LE-**staging**, in-cluster
|
|
clients rejected it with `x509: certificate signed by unknown authority`. If `/runbooks/cert-issuance` or
|
|
`/runbooks/cert-expiry` is also firing, fix that first.
|
|
|
|
### 5. Logins fail but Keycloak is UP
|
|
Then this is the **wrong alert and the wrong runbook**, and the distinction has cost this cluster dearly.
|
|
On 2026-09-06 every OIDC login on prod failed — *"Invalid client or Invalid client credentials"* — with
|
|
**all 12 clients holding the literal `(env:...)` placeholder as their secret**, while
|
|
`KeycloakDown`/`KeycloakDatabaseDown` were quiet, the import Job reported success, the CR read
|
|
`Done=True`, and the conformance test that asserts `Done=True` **passed throughout**. The only observable
|
|
was a human failing to log in. Go to `/runbooks/keycloak-client-secrets`.
|
|
Similarly, a `redirect_uri` mismatch after a hostname change is not an outage — same runbook.
|
|
And once, MFA broke every platform-admin login on its own: conditional-OTP used a non-existent config
|
|
key, giving `"Invalid role name: null"` → 403, fixed to `condUserRole: platform-admin`.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **Escalate immediately.** SSO down is a platform-wide outage; treat it as such.
|
|
- **Use the certificate-based kubeconfig** (the dev-container path) rather than trying to fix OIDC login
|
|
first. Do not burn the outage on getting `kubectl` to authenticate through the thing that is down.
|
|
- **Do not edit the realm to "reset" anything.** The realm import is a **ONE-SHOT that fails open**: the
|
|
operator logs *"Job already executed - not recreating"*, so a git change to
|
|
`keycloak-realm-import.yaml` does **not** re-import. The documented path is `kcadm.sh` inside
|
|
`keycloak-0` (admin credentials in the Secret `keycloak-admin-credentials`; `kcadm` is present,
|
|
`curl`/`python3` are **not**) **and** fixing git for reproducibility. Doing only one of the two is how
|
|
drift is created.
|
|
- **Do not delete the Keycloak CR or the StatefulSet.** Recreating the IdP is a restore operation, not a
|
|
restart.
|
|
- **Do not touch `keycloak-db` volumes.** If the database is the cause, read `/runbooks/keycloak-db` and
|
|
`/runbooks/cnpg-archiving`; only the Barman archive is a backup, standbys replicate deletions
|
|
faithfully.
|
|
- Break-glass local admins exist for gitea, nextcloud and semaphore. Nextcloud has **no native
|
|
group→admin mapping**, so platform-admins there are promoted by hand.
|
|
|
|
---
|
|
Source of truth: `runbooks/keycloak-down.md`. Index: `/runbooks/`
|
|
Records: `DECISIONS-LOG.md` (the 2026-09-06 client-secret lockout; the 2026-09-07 single-node SSO
|
|
outage); commits `f338e91`, `e8d8601`, `cac413e` (CNPG instances/anti-affinity/switchover/sync).
|