jakman85andClaude Opus 5.5
b85d1e3ed7
Hand-authored platform/vuln-scanner (non-gating Flux tier, digest-pinned Trivy 0.76.1 and python 3.14.9-alpine, read-only RBAC, per-pod oauth2-proxy, internal-only cve.<domain>), five alerts with promtool tests, R-VULN-01..09 registry entries, and vuln-* L0 checks with mutation self-tests. src/ and the bot workflow are integrated separately. Co-Authored-By: Claude Opus 5.5 <[email protected]>
140 lines
8.6 KiB
YAML
140 lines
8.6 KiB
YAML
---
|
|
# Runbooks tier — reconciled by its OWN non-gating post-platform Flux Kustomization
|
|
# (clusters/base/runbooks.yaml, dependsOn platform-config, wait:false). Same placement and rationale as
|
|
# platform/homepage: a leaf web server must never sit in the gating `platform` tier, where a UI's
|
|
# unhealth once stalled the whole chain (DECISIONS-LOG: "Headlamp gating stall").
|
|
#
|
|
# WHY THE MANIFESTS LIVE HERE, ALONGSIDE THE DOCS, RATHER THAN IN platform/runbooks/.
|
|
# The markdown is the source of truth and the task is to serve it verbatim, which means the ConfigMap must
|
|
# be GENERATED from these files — there must not be a second hand-maintained copy of 34 documents that can
|
|
# drift from the originals. `configMapGenerator` does that, and its `files:` paths are resolved relative to
|
|
# THIS directory. A `platform/runbooks/kustomization.yaml` would therefore need `../../runbooks/<name>.md`,
|
|
# which depends on kustomize load restrictions being relaxed — true for Flux's kustomize-controller,
|
|
# refused by a plain `kustomize build`, and so a divergence between what the gate can verify and what the
|
|
# cluster does. Making this directory the Flux path removes the question entirely: no `../`, identical
|
|
# behaviour locally and in-cluster, one copy of every document. The serving manifests are tucked under
|
|
# `serving/` so `runbooks/*.md` stays exactly the flat set of pages the URLs name.
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
namespace: runbooks
|
|
resources:
|
|
- serving/namespace.yaml
|
|
- serving/nginx-config.yaml
|
|
- serving/deployment.yaml
|
|
- serving/service.yaml
|
|
# default-deny baseline + Cilium gateway-ingress allow (DNS-only egress from the baseline).
|
|
- serving/networkpolicy.yaml
|
|
- serving/httproute.yaml # apex `/runbooks` prefix (binds the Gateway `https-apex` listener)
|
|
|
|
configMapGenerator:
|
|
# ONE ConfigMap holding every runbook, keyed by the EXACT name the alert annotations use.
|
|
#
|
|
# THE KEYS ARE EXTENSIONLESS ON PURPOSE. The annotations are `https://${BASE_DOMAIN}/runbooks/<name>`
|
|
# with NO `.md`, and this repo must not change them (they are spread across four alerts.yaml files that
|
|
# other work is editing). `<key>=<file>` maps `node-down.md` on disk to the key `node-down`, the volume
|
|
# mount turns each key into a file under the nginx root, and `/runbooks/node-down` becomes a plain static
|
|
# lookup — no rewrite rule, nothing to get subtly wrong at 3am. Drop the `<key>=` and every key would
|
|
# carry `.md`, so every alert link would 404 while this file still looked correct.
|
|
#
|
|
# The generated name carries kustomize's content hash, so editing any runbook rolls the server
|
|
# automatically and a stale projection is impossible.
|
|
#
|
|
# ADDING A RUNBOOK: add the file AND its line here. The L0 check
|
|
# conformance/tests/l0-static/alert-runbooks-resolvable asserts both halves, because a file in git with
|
|
# no generator entry is reviewed, merged, apparently done — and still a 404 in the cluster.
|
|
- name: runbooks-content
|
|
options:
|
|
labels:
|
|
app.kubernetes.io/name: runbooks
|
|
files:
|
|
# The index, served at /runbooks/ (nginx `try_files /runbooks/README`).
|
|
- README=README.md
|
|
- vuln-scanner=vuln-scanner.md
|
|
- backup-failure=backup-failure.md
|
|
- backup-missed=backup-missed.md
|
|
- backup-partial=backup-partial.md
|
|
# Operator procedure, not an alert page — nothing links here from an alert. Served because
|
|
# the whole-cluster shutdown had NO written procedure until 2026-10-17 (infra/cluster.sh is
|
|
# virsh-based and only ever applied to the decommissioned dev cluster), so the UPS move was
|
|
# improvised live, and the one hazard it exposed -- a StatefulSet with
|
|
# persistentVolumeClaimRetentionPolicy whenScaled: Delete loses its PVC when scaled to 0 --
|
|
# destroyed monitoring/loki's volume before anyone noticed.
|
|
- cloudflare-tunnel=cloudflare-tunnel.md
|
|
- cluster-maintenance-shutdown=cluster-maintenance-shutdown.md
|
|
- cdi-import-stuck=cdi-import-stuck.md
|
|
- cert-expiry=cert-expiry.md
|
|
- cert-issuance=cert-issuance.md
|
|
- cnpg-archiving=cnpg-archiving.md
|
|
- cnpg-backup-missed=cnpg-backup-missed.md
|
|
- cnpg-exporter=cnpg-exporter.md
|
|
- cnpg-primary-spread=cnpg-primary-spread.md
|
|
- cnpg-wal-growth=cnpg-wal-growth.md
|
|
# Operator procedure, not alert-linked (Crafty's local login is a second factor behind
|
|
# the Keycloak gate and its bootstrap password lives only on the PVC).
|
|
- crafty-admin-credential=crafty-admin-credential.md
|
|
- denied-flows=denied-flows.md
|
|
- flux-reconcile=flux-reconcile.md
|
|
- flux-rollout=flux-rollout.md
|
|
- flux-suspended=flux-suspended.md
|
|
- image-automation=image-automation.md
|
|
- image-policy-stalled=image-policy-stalled.md
|
|
- keycloak-client-secrets=keycloak-client-secrets.md
|
|
- keycloak-passkey-reconciler=keycloak-passkey-reconciler.md
|
|
- keycloak-db=keycloak-db.md
|
|
- keycloak-down=keycloak-down.md
|
|
- kubevirt-control-plane=kubevirt-control-plane.md
|
|
- kubevirt-crashloop=kubevirt-crashloop.md
|
|
- kubevirt-launch-failed=kubevirt-launch-failed.md
|
|
- longhorn-capacity=longhorn-capacity.md
|
|
- longhorn-degraded=longhorn-degraded.md
|
|
- longhorn-faulted=longhorn-faulted.md
|
|
- monitoring-capacity=monitoring-capacity.md
|
|
- node-cpu=node-cpu.md
|
|
- node-disk=node-disk.md
|
|
- node-down=node-down.md
|
|
- node-io=node-io.md
|
|
- node-memory=node-memory.md
|
|
- node-memory-swap=node-memory-swap.md
|
|
# Covers all three platform-health.node-reboots rules (KuredRebootPending,
|
|
# KuredNodeCoverageGap, KuredNotReporting) from platform/kured/alerts.yaml.
|
|
- node-reboot-automation=node-reboot-automation.md
|
|
# Linked from OAuth2ProxyGateUnobserved, and the diagnostic procedure for any gated app
|
|
# that "is up but does not work" — the gate answers an unauthenticated XHR with a 302,
|
|
# not a 401, so the failure arrives as an HTTP 200 carrying the wrong body.
|
|
- oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md
|
|
# Operator procedure, not alert-linked. Reading the per-app oauth2-proxy sidecar log to tell
|
|
# Keycloak, the sidecar and the app's own login apart. Cookie collision between two sidecars
|
|
# presents as random session loss in ONE app's log and is only legible with both side by side,
|
|
# so the page has to be reachable at 3am without going looking for it in git.
|
|
# SIBLING PAGE to oauth2-proxy-auth-failures above, and they are deliberately both here: this
|
|
# one is the SESSION-LIFECYCLE procedure (why a session died), that one is the DIAGNOSTIC
|
|
# TRIAGE (which of the three layers refused, and how to tell a 302-to-200 from a real success).
|
|
- oauth2-proxy-session=oauth2-proxy-session.md
|
|
# Operator procedure, not an alert page (no alert links to it — the
|
|
# alert-runbooks-resolvable check treats that as advisory). It is served
|
|
# because the plex.tv claim window is short and is not a thing to go looking
|
|
# for in git: /runbooks/plex-claim has to resolve the moment it is needed.
|
|
- plex-claim=plex-claim.md
|
|
# Operator procedure, not an alert page, and NO alert is possible for it: plex renders no
|
|
# metricsEndpoint, so there is no ServiceMonitor, no transcode-mode metric and nothing a rule
|
|
# could evaluate. Served because the feature it completes is INERT until a human ticks a box in
|
|
# the Plex UI — the one step GitOps cannot reach — and a page nobody can find is how a shipped
|
|
# feature stays switched off for months.
|
|
- plex-quicksync=plex-quicksync.md
|
|
# Operator procedure, not alert-linked: how to STOP the PM agent service (three ways,
|
|
# increasing severity), how to read why it is idle, how to tell a worker that was
|
|
# correctly refused by the gate from one that broke, and the one-year expiry of its
|
|
# Anthropic subscription token — which nothing warns about, because Claude Code's expiry
|
|
# notice only fires for an interactive login, never for CLAUDE_CODE_OAUTH_TOKEN.
|
|
- pm-agent=pm-agent.md
|
|
- pvc-capacity=pvc-capacity.md
|
|
- targets-down=targets-down.md
|
|
- update-rollback=update-rollback.md
|
|
- vpn-down=vpn-down.md
|
|
- vpn-egress-stale=vpn-egress-stale.md
|
|
# Operator procedure, not an alert page. Served because the component it completes is INERT
|
|
# until three off-repo manual steps are done (kernel module, server key, WAN port-forward) —
|
|
# and because the one check that actually proves it cannot be run from inside the house.
|
|
# Peer-level alerting does not exist yet (it needs an exporter); this page is the substitute.
|
|
- vpn-remote-access=vpn-remote-access.md
|