jakman85andClaude Opus 5
4fd75ccdaa
Plex cannot create libraries over the already-mounted 8.33T NFS bulk share while the server is UNCLAIMED. catalog/plex.yaml carried a long argument for leaving PLEX_CLAIM unwired; the operator has decided to override it. This implements the wiring in the way that respects the half of that argument which is still true. WHAT WAS TRUE AND STILL IS: a https://plex.tv/claim token has a ~4-minute TTL, far shorter than commit -> push -> Flux reconcile -> pod-roll on its own timers. So the committed value is a documented PLACEHOLDER and the live token is injected by the operator inside that window, with the reconcile FORCED rather than waited for. runbooks/plex-claim.md is that timed procedure. WHAT WAS NOT TRUE: the old comment claimed "a placeholder/absent claim Secret would crashloop the pod". Half of that is wrong, and the difference is the whole design. Established from plexinc/pms-docker's own first-run script (branch `master`, root/etc/cont-init.d/40-plex-first-run, s6-overlay v2), cross-checked byte-identical against the same file inside the running :latest image: * the claim exchange (line 106) is gated on PLEX_CLAIM being non-empty AND the PlexOnlineToken pref in /config being empty. Once the server is claimed the block is skipped entirely, so the env is a NO-OP on every later start whatever it holds ("If server is already logged in, this parameter is ignored" -- upstream README). * with an invalid/expired token on a still-unclaimed /config, plex.tv answers 403 with an empty body; curl's status is never checked, the token sed yields empty, the success branch at line 120 does not fire, there is no else branch and no non-zero exit, and the script (no `set -e`) exits 0. The pod comes up UNCLAIMED AND SERVING. It does not crashloop. * the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS (default 0 = continue even if a cont-init script failed), so even a non-zero exit would not halt startup. An ABSENT Secret is the real hard failure (secretKeyRef has no `optional`, so it is CreateContainerConfigError), which is why the Secret is committed, encrypted and listed in the wave kustomization rather than hand-created at claim time. NO SCHEMA CHANGE. `optional` on secretKeyRef would only be needed if the pod had to tolerate a missing Secret or could not start without a VALID token. Neither holds: the Secret always exists in the Flux graph, and an invalid token is non-fatal. schemas/servicedefinition.schema.json is untouched, and so are catalog/render.py's env emission and the renderer contract. Changes: * apps/wave-0/plex-claim-secret.sops.yaml -- new Opaque Secret plex-claim/plex, key PLEX_CLAIM, SOPS-encrypted to the committed age recipient. Its header carries NO dollar-brace expressions on purpose: kustomize preserves body comments into the stream envsubst reads (per conformance/tests/l0-static/flux-postbuild-substitution), so quoting the upstream shell conditions verbatim would have been a live substitution hazard on apps-wave-0, not prose. * catalog/plex.yaml -- the PLEX_CLAIM env via valueFrom.secretKeyRef, the "deliberately NOT set" block rewritten around the TTL + one-shot design, and the stale crashloop claim in the /config storage comment corrected. * apps/wave-0/plex.yaml -- re-rendered with the canonical command (`--all catalog -o apps --wave-layout --no-image-automation`). No L1 golden under conformance/tests/l1-rendering covers plex, so this is the only rendered surface that moves. * catalog/render.py -- plan_apps_waves places `<svc>-claim-secret.sops.yaml` in the app's OWN wave, like the bootstrap secret, plus a selftest assertion. Without this, the next `--emit-waves` would have silently dropped the Secret from apps/wave-0/kustomization.yaml and put the pod into CreateContainerConfigError. apps/wave-0/kustomization.yaml is the hand-pruned subset (vm-demo/keycloak are rendered but deliberately unlisted), so its one new line was applied without re-emitting the rest. * runbooks/plex-claim.md + its kustomization.yaml generator entry + README index -- the operator procedure that fits in 4 minutes, including the single forced-reconcile command and the mandatory pod roll (a changed Secret does NOT restart anything: PLEX_CLAIM is read only at container start and the pod template is byte-identical across this change). Touched requirements: R-SEC-01 (secrets encrypted at rest), R-GITOPS-02 (ServiceDefinition onboarding contract). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
153 lines
9.7 KiB
Markdown
153 lines
9.7 KiB
Markdown
# Alert runbooks — home-network-k3s prod
|
||
|
||
This is the index page served at `https://home.keithjackson.dev/runbooks/`.
|
||
|
||
Every alert rule in this platform carries an annotation of the form
|
||
|
||
runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"
|
||
|
||
46 of them, over 35 names. Until this directory existed, that path returned **HTTP 404** on the
|
||
live cluster: every alert that reached a human handed them a dead link at the exact moment they needed
|
||
help. These pages are what makes the link resolve.
|
||
|
||
## How to run the commands in these runbooks
|
||
|
||
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
||
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that hang
|
||
is not the cluster being down.
|
||
|
||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
||
|
||
Always pass `--context k3s-prod`. `dev.sh` merges BOTH clusters into one kubeconfig and picks a
|
||
current-context per invocation, and **that default has already changed once** — k3s-dev until dev was
|
||
decommissioned on 2026-09-13, k3s-prod now. A command that names its context still means the same thing
|
||
next month; one that relies on the default does not, and the failure is silent. `CLUSTER=prod
|
||
infra/devcontainer/dev.sh run kubectl ...` selects it explicitly, and every invocation prints a banner
|
||
naming the cluster — read it. Do not run two `dev.sh` invocations at once; they share one container.
|
||
|
||
**Use fully-qualified resource names, always.** Three CRDs on this cluster register the kind `Backup` —
|
||
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1`, `velero.io/v1` — and the bare short name resolves to
|
||
Longhorn's, which holds zero objects. `kubectl get backup -A` therefore prints **"No resources found"** on
|
||
a cluster holding 6 Velero and 27 CNPG backups (verified on prod 2026-09-13). Acting on that false
|
||
negative is how a CLI quirk becomes real data loss. Write `backups.velero.io`,
|
||
`backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`,
|
||
`volumes.longhorn.io`, `nodes.longhorn.io`.
|
||
|
||
`git push origin main` **is** the deploy; there is no other apply step. Durable changes go through git.
|
||
|
||
## Alerts you should not trust the silence of (measured 2026-09-13)
|
||
|
||
The Longhorn and cert-manager scrapes were ENABLED on 2026-09-13, so the Longhorn row below is no
|
||
longer inert — and re-keying the two robustness rules the same day fixed a SECOND inertness the
|
||
scrape had left in place (the metric is one-hot over a `state` label, not a 0–3 enum). Take the
|
||
Longhorn row as history; the rest of this table still holds, and Flux-controller and Hubble/Cilium
|
||
rule groups still evaluate against nothing and **cannot fire**:
|
||
|
||
| Missing scrape job | Inert alerts |
|
||
|---|---|
|
||
| Longhorn | `LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `LonghornDiskAllocationHigh`, `LonghornDiskUnschedulable` |
|
||
| cert-manager | `CertificateExpiringSoon`, `CertificateExpiryImminent`, `CertificateNotReady` |
|
||
| Flux (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` |
|
||
| Hubble (`hubble_drop_total`) | `HubbleDeniedFlowSpike` |
|
||
| etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all |
|
||
|
||
Each affected runbook says so on its own page and gives the by-hand equivalent. See `targets-down`.
|
||
|
||
## The runbooks
|
||
|
||
### Nodes
|
||
- **`node-down`** — a node's node-exporter is unscrapable; the node may be gone, or its kubelet may have
|
||
stopped posting status while the host stays up (the 2026-09-07 shape).
|
||
- **`node-io`** — disk I/O pressure/starvation. The etcd WAL shares a partition with Longhorn replicas;
|
||
this is the alert that would have called the 2026-09-07 outage and did not exist at the time.
|
||
- **`node-memory`** — memory pressure. Includes why `SwapTotal == 0` is not evidence swap is off.
|
||
- **`node-disk`** — a node filesystem filling up. `/data` is the one that matters.
|
||
- **`node-cpu`** — CPU PSI and run-queue depth, and how to tell CPU contention from I/O contention.
|
||
|
||
### Storage
|
||
- **`longhorn-degraded`** — a volume has lost redundancy. Normal after any reboot, and the rebuild is the
|
||
expensive part. Freeze/thaw commands here.
|
||
- **`longhorn-faulted`** — no usable replica. Data at risk; stop before anything destructive.
|
||
- **`longhorn-capacity`** — a node's Longhorn disk running out of room for replicas (allocation, not PVC fill; for PVC fill see `pvc-capacity` / `monitoring-capacity`).
|
||
- **`pvc-capacity`** — PVC filling up / almost full / predicted to fill. These rules work.
|
||
- **`monitoring-capacity`** — Prometheus or Loki running out of disk; don't let the monitor blind itself.
|
||
|
||
### GitOps and updates
|
||
- **`flux-reconcile`** — a Flux resource stuck at Ready=False. Which tier decides the blast radius.
|
||
- **`flux-suspended`** — reconciliation turned off and forgotten. Drift becomes silent.
|
||
- **`update-rollback`** — a HelmRelease failed its health gate and is rolling back.
|
||
- **`image-automation`** — the git write-back engine is failing (feature is disabled here).
|
||
- **`image-policy-stalled`** — an ImagePolicy/ImageRepository cannot resolve a tag.
|
||
|
||
### Certificates
|
||
- **`cert-issuance`** — ACME issuance failing. The DNS-01 self-check is the usual cause.
|
||
- **`cert-expiry`** — a certificate is expiring. Almost always a late symptom of `cert-issuance`.
|
||
|
||
### Networking
|
||
- **`vpn-down`** — the egress VPN tunnel. Explains what the kill-switch actually does, and the silent
|
||
egress-gateway staleness this alert cannot see — which now has its own alerts and page below.
|
||
- **`vpn-egress-stale`** — the tunnel is UP but `egress-class=vpn` pods have no egress: Cilium's cached
|
||
ifindex outlived a reconnect. Also covers the watchdog for that rule going inert.
|
||
- **`denied-flows`** — policy-denied flow spike. The Cilium hairpin/identity regression class.
|
||
|
||
### Backups and DR
|
||
- **`backup-failure`** — a Velero backup failed. The R2 object-tagging incompatibility is documented here.
|
||
- **`backup-partial`** — a Velero backup completed with partial failures, and which skips are deliberate.
|
||
- **`backup-missed`** — no successful Velero backup in 24h.
|
||
- **`cnpg-archiving`** — WAL archiving stuck / no recoverability point. **An availability incident, not
|
||
just a backup one.**
|
||
- **`cnpg-backup-missed`** — no recent CNPG base backup. Velero does not cover these databases.
|
||
- **`cnpg-wal-growth`** — `pg_wal` not being recycled. Never delete WAL by hand.
|
||
- **`cnpg-exporter`** — the CNPG SQL exporter erroring: a blind spot that looks exactly like health.
|
||
- **`cnpg-primary-spread`** — two or more CNPG primaries on one node. Blast radius, not an outage,
|
||
and the rebalance costs ~9s of writes per database — one cluster at a time.
|
||
|
||
### Identity
|
||
- **`keycloak-down`** — the IdP is down. Blast radius, and the break-glass path that still works.
|
||
- **`keycloak-db`** — Keycloak's CloudNativePG database is unavailable.
|
||
- **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
|
||
every green signal missed.
|
||
|
||
### Operator procedures (no alert links here)
|
||
- **`plex-claim`** — claiming the Plex Media Server. The plex.tv claim token lives ~4 minutes, so this
|
||
is a timed sequence: token → `sops` → push → **forced** `flux reconcile` → pod roll. Includes why a
|
||
changed Secret does not restart anything, and why an expired token is harmless.
|
||
|
||
### Virtualisation
|
||
- **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.
|
||
- **`kubevirt-launch-failed`** — a VMI reached the Failed phase. The three k3s-specific launch traps.
|
||
- **`kubevirt-crashloop`** — a virt-launcher pod restarting repeatedly.
|
||
- **`cdi-import-stuck`** — a DataVolume import not progressing. Check node I/O before restarting it.
|
||
|
||
## Editing a runbook
|
||
|
||
Each page is `runbooks/<name>.md` in the `home-network-k3s-impl` repo. Edit it, push to `main`, and Flux
|
||
re-renders the ConfigMap and rolls the two-replica server; kustomize's content hash on the generated
|
||
ConfigMap name is what triggers the roll.
|
||
|
||
Adding a **new** alert with a `runbook_url` requires the matching `runbooks/<name>.md` **and** its
|
||
`<name>=<name>.md` entry in `runbooks/kustomization.yaml`. Both are enforced by
|
||
`conformance/tests/l0-static/alert-runbooks-resolvable`, which runs unconditionally in the merge gate —
|
||
"the file exists in git" and "the URL resolves in the cluster" are different claims, and it checks both.
|
||
|
||
Every page must carry the four operator sections (*What this means here*, *First commands*, *Likely
|
||
causes, most likely first*, *When to escalate or stop*), mention `dev.sh`, and pass `--context k3s-prod`.
|
||
The check enforces that too, because a stub that makes the link resolve while helping nobody is worse than
|
||
the 404 it replaced — it looks fixed.
|
||
|
||
## Why this route has no login
|
||
|
||
Deliberately public. The UIs that declare forward-auth (pihole, jackett, qbittorrent, radarr, sonarr)
|
||
really are closed: enforcement is per-pod, each one's HTTPRoute targeting its oauth2-proxy sidecar on
|
||
:4180 rather than the app port. If
|
||
these pages did too, `keycloak-down` would be unreadable exactly when Keycloak is down,
|
||
`keycloak-db` unreadable when its database is down, and `keycloak-client-secrets` unreadable during the
|
||
precise failure that broke all 12 OIDC clients on 2026-09-06 behind an entirely green cluster. An
|
||
operator who cannot log in is this route's primary reader. The content is operational procedure — metric
|
||
names, thresholds, kubectl invocations, topology — and no secrets. See
|
||
`runbooks/serving/httproute.yaml` for the full reasoning, including how forward-auth is actually
|
||
enforced here and the instruction to leave this route **out** of any Gateway-level ext-authz attachment
|
||
if one is ever introduced.
|
||
|
||
The running narrative of real incidents on this cluster is `DECISIONS-LOG.md`.
|