--- # Runbooks tier — reconciled by its OWN non-gating post-platform Flux Kustomization # (clusters/base/runbooks.yaml, dependsOn platform-config, wait:false). Same placement and rationale as # platform/homepage: a leaf web server must never sit in the gating `platform` tier, where a UI's # unhealth once stalled the whole chain (DECISIONS-LOG: "Headlamp gating stall"). # # WHY THE MANIFESTS LIVE HERE, ALONGSIDE THE DOCS, RATHER THAN IN platform/runbooks/. # The markdown is the source of truth and the task is to serve it verbatim, which means the ConfigMap must # be GENERATED from these files — there must not be a second hand-maintained copy of 34 documents that can # drift from the originals. `configMapGenerator` does that, and its `files:` paths are resolved relative to # THIS directory. A `platform/runbooks/kustomization.yaml` would therefore need `../../runbooks/.md`, # which depends on kustomize load restrictions being relaxed — true for Flux's kustomize-controller, # refused by a plain `kustomize build`, and so a divergence between what the gate can verify and what the # cluster does. Making this directory the Flux path removes the question entirely: no `../`, identical # behaviour locally and in-cluster, one copy of every document. The serving manifests are tucked under # `serving/` so `runbooks/*.md` stays exactly the flat set of pages the URLs name. apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization namespace: runbooks resources: - serving/namespace.yaml - serving/nginx-config.yaml - serving/deployment.yaml - serving/service.yaml # default-deny baseline + Cilium gateway-ingress allow (DNS-only egress from the baseline). - serving/networkpolicy.yaml - serving/httproute.yaml # apex `/runbooks` prefix (binds the Gateway `https-apex` listener) configMapGenerator: # ONE ConfigMap holding every runbook, keyed by the EXACT name the alert annotations use. # # THE KEYS ARE EXTENSIONLESS ON PURPOSE. The annotations are `https://${BASE_DOMAIN}/runbooks/` # with NO `.md`, and this repo must not change them (they are spread across four alerts.yaml files that # other work is editing). `=` maps `node-down.md` on disk to the key `node-down`, the volume # mount turns each key into a file under the nginx root, and `/runbooks/node-down` becomes a plain static # lookup — no rewrite rule, nothing to get subtly wrong at 3am. Drop the `=` and every key would # carry `.md`, so every alert link would 404 while this file still looked correct. # # The generated name carries kustomize's content hash, so editing any runbook rolls the server # automatically and a stale projection is impossible. # # ADDING A RUNBOOK: add the file AND its line here. The L0 check # conformance/tests/l0-static/alert-runbooks-resolvable asserts both halves, because a file in git with # no generator entry is reviewed, merged, apparently done — and still a 404 in the cluster. - name: runbooks-content options: labels: app.kubernetes.io/name: runbooks files: # The index, served at /runbooks/ (nginx `try_files /runbooks/README`). - README=README.md - vuln-scanner=vuln-scanner.md - backup-failure=backup-failure.md - backup-missed=backup-missed.md - backup-partial=backup-partial.md # Operator procedure, not an alert page — nothing links here from an alert. Served because # the whole-cluster shutdown had NO written procedure until 2026-10-17 (infra/cluster.sh is # virsh-based and only ever applied to the decommissioned dev cluster), so the UPS move was # improvised live, and the one hazard it exposed -- a StatefulSet with # persistentVolumeClaimRetentionPolicy whenScaled: Delete loses its PVC when scaled to 0 -- # destroyed monitoring/loki's volume before anyone noticed. - cloudflare-tunnel=cloudflare-tunnel.md - cluster-maintenance-shutdown=cluster-maintenance-shutdown.md - cdi-import-stuck=cdi-import-stuck.md - cert-expiry=cert-expiry.md - cert-issuance=cert-issuance.md - cnpg-archiving=cnpg-archiving.md - cnpg-backup-missed=cnpg-backup-missed.md - cnpg-exporter=cnpg-exporter.md - cnpg-primary-spread=cnpg-primary-spread.md - cnpg-wal-growth=cnpg-wal-growth.md # Operator procedure, not alert-linked (Crafty's local login is a second factor behind # the Keycloak gate and its bootstrap password lives only on the PVC). - crafty-admin-credential=crafty-admin-credential.md - denied-flows=denied-flows.md - flux-reconcile=flux-reconcile.md - flux-rollout=flux-rollout.md - flux-suspended=flux-suspended.md - image-automation=image-automation.md - image-policy-stalled=image-policy-stalled.md - keycloak-client-secrets=keycloak-client-secrets.md - keycloak-passkey-reconciler=keycloak-passkey-reconciler.md - keycloak-db=keycloak-db.md - keycloak-down=keycloak-down.md - kubevirt-control-plane=kubevirt-control-plane.md - kubevirt-crashloop=kubevirt-crashloop.md - kubevirt-launch-failed=kubevirt-launch-failed.md - longhorn-capacity=longhorn-capacity.md - longhorn-degraded=longhorn-degraded.md - longhorn-faulted=longhorn-faulted.md - monitoring-capacity=monitoring-capacity.md - node-cpu=node-cpu.md - node-disk=node-disk.md - node-down=node-down.md - node-io=node-io.md - node-memory=node-memory.md - node-memory-swap=node-memory-swap.md # Covers all three platform-health.node-reboots rules (KuredRebootPending, # KuredNodeCoverageGap, KuredNotReporting) from platform/kured/alerts.yaml. - node-reboot-automation=node-reboot-automation.md # Linked from OAuth2ProxyGateUnobserved, and the diagnostic procedure for any gated app # that "is up but does not work" — the gate answers an unauthenticated XHR with a 302, # not a 401, so the failure arrives as an HTTP 200 carrying the wrong body. - oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md # Operator procedure, not alert-linked. Reading the per-app oauth2-proxy sidecar log to tell # Keycloak, the sidecar and the app's own login apart. Cookie collision between two sidecars # presents as random session loss in ONE app's log and is only legible with both side by side, # so the page has to be reachable at 3am without going looking for it in git. # SIBLING PAGE to oauth2-proxy-auth-failures above, and they are deliberately both here: this # one is the SESSION-LIFECYCLE procedure (why a session died), that one is the DIAGNOSTIC # TRIAGE (which of the three layers refused, and how to tell a 302-to-200 from a real success). - oauth2-proxy-session=oauth2-proxy-session.md # Operator procedure, not an alert page (no alert links to it — the # alert-runbooks-resolvable check treats that as advisory). It is served # because the plex.tv claim window is short and is not a thing to go looking # for in git: /runbooks/plex-claim has to resolve the moment it is needed. - plex-claim=plex-claim.md # Operator procedure, not an alert page, and NO alert is possible for it: plex renders no # metricsEndpoint, so there is no ServiceMonitor, no transcode-mode metric and nothing a rule # could evaluate. Served because the feature it completes is INERT until a human ticks a box in # the Plex UI — the one step GitOps cannot reach — and a page nobody can find is how a shipped # feature stays switched off for months. - plex-quicksync=plex-quicksync.md # Operator procedure, not alert-linked: how to STOP the PM agent service (three ways, # increasing severity), how to read why it is idle, how to tell a worker that was # correctly refused by the gate from one that broke, and the one-year expiry of its # Anthropic subscription token — which nothing warns about, because Claude Code's expiry # notice only fires for an interactive login, never for CLAUDE_CODE_OAUTH_TOKEN. - pm-agent=pm-agent.md - pvc-capacity=pvc-capacity.md - targets-down=targets-down.md - update-rollback=update-rollback.md - vpn-down=vpn-down.md - vpn-egress-stale=vpn-egress-stale.md # Operator procedure, not an alert page. Served because the component it completes is INERT # until three off-repo manual steps are done (kernel module, server key, WAN port-forward) — # and because the one check that actually proves it cannot be run from inside the house. # Peer-level alerting does not exist yet (it needs an exporter); this page is the substitute. - vpn-remote-access=vpn-remote-access.md