home-network-k3s-impl/runbooks/kustomization.yaml

---
# Runbooks tier — reconciled by its OWN non-gating post-platform Flux Kustomization
# (clusters/base/runbooks.yaml, dependsOn platform-config, wait:false). Same placement and rationale as
# platform/homepage: a leaf web server must never sit in the gating `platform` tier, where a UI's
# unhealth once stalled the whole chain (DECISIONS-LOG: "Headlamp gating stall").
#
# WHY THE MANIFESTS LIVE HERE, ALONGSIDE THE DOCS, RATHER THAN IN platform/runbooks/.
# The markdown is the source of truth and the task is to serve it verbatim, which means the ConfigMap must
# be GENERATED from these files — there must not be a second hand-maintained copy of 34 documents that can
# drift from the originals. `configMapGenerator` does that, and its `files:` paths are resolved relative to
# THIS directory. A `platform/runbooks/kustomization.yaml` would therefore need `../../runbooks/<name>.md`,
# which depends on kustomize load restrictions being relaxed — true for Flux's kustomize-controller,
# refused by a plain `kustomize build`, and so a divergence between what the gate can verify and what the
# cluster does. Making this directory the Flux path removes the question entirely: no `../`, identical
# behaviour locally and in-cluster, one copy of every document. The serving manifests are tucked under
# `serving/` so `runbooks/*.md` stays exactly the flat set of pages the URLs name.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: runbooks
resources:
  - serving/namespace.yaml
  - serving/nginx-config.yaml
  - serving/deployment.yaml
  - serving/service.yaml
  # default-deny baseline + Cilium gateway-ingress allow (DNS-only egress from the baseline).
  - serving/networkpolicy.yaml
  - serving/httproute.yaml   # apex `/runbooks` prefix (binds the Gateway `https-apex` listener)

configMapGenerator:
  # ONE ConfigMap holding every runbook, keyed by the EXACT name the alert annotations use.
  #
  # THE KEYS ARE EXTENSIONLESS ON PURPOSE. The annotations are `https://${BASE_DOMAIN}/runbooks/<name>`
  # with NO `.md`, and this repo must not change them (they are spread across four alerts.yaml files that
  # other work is editing). `<key>=<file>` maps `node-down.md` on disk to the key `node-down`, the volume
  # mount turns each key into a file under the nginx root, and `/runbooks/node-down` becomes a plain static
  # lookup — no rewrite rule, nothing to get subtly wrong at 3am. Drop the `<key>=` and every key would
  # carry `.md`, so every alert link would 404 while this file still looked correct.
  #
  # The generated name carries kustomize's content hash, so editing any runbook rolls the server
  # automatically and a stale projection is impossible.
  #
  # ADDING A RUNBOOK: add the file AND its line here. The L0 check
  # conformance/tests/l0-static/alert-runbooks-resolvable asserts both halves, because a file in git with
  # no generator entry is reviewed, merged, apparently done — and still a 404 in the cluster.
  - name: runbooks-content
    options:
      labels:
        app.kubernetes.io/name: runbooks
    files:
      # The index, served at /runbooks/ (nginx `try_files /runbooks/README`).
      - README=README.md
      - backup-failure=backup-failure.md
      - backup-missed=backup-missed.md
      - backup-partial=backup-partial.md
      - cdi-import-stuck=cdi-import-stuck.md
      - cert-expiry=cert-expiry.md
      - cert-issuance=cert-issuance.md
      - cnpg-archiving=cnpg-archiving.md
      - cnpg-backup-missed=cnpg-backup-missed.md
      - cnpg-exporter=cnpg-exporter.md
      - cnpg-primary-spread=cnpg-primary-spread.md
      - cnpg-wal-growth=cnpg-wal-growth.md
      # Operator procedure, not alert-linked (Crafty's local login is a second factor behind
      # the Keycloak gate and its bootstrap password lives only on the PVC).
      - crafty-admin-credential=crafty-admin-credential.md
      - denied-flows=denied-flows.md
      - flux-reconcile=flux-reconcile.md
      - flux-rollout=flux-rollout.md
      - flux-suspended=flux-suspended.md
      - image-automation=image-automation.md
      - image-policy-stalled=image-policy-stalled.md
      - keycloak-client-secrets=keycloak-client-secrets.md
      - keycloak-db=keycloak-db.md
      - keycloak-down=keycloak-down.md
      - kubevirt-control-plane=kubevirt-control-plane.md
      - kubevirt-crashloop=kubevirt-crashloop.md
      - kubevirt-launch-failed=kubevirt-launch-failed.md
      - longhorn-capacity=longhorn-capacity.md
      - longhorn-degraded=longhorn-degraded.md
      - longhorn-faulted=longhorn-faulted.md
      - monitoring-capacity=monitoring-capacity.md
      - node-cpu=node-cpu.md
      - node-disk=node-disk.md
      - node-down=node-down.md
      - node-io=node-io.md
      - node-memory=node-memory.md
      - node-memory-swap=node-memory-swap.md
      # Covers all three platform-health.node-reboots rules (KuredRebootPending,
      # KuredNodeCoverageGap, KuredNotReporting) from platform/kured/alerts.yaml.
      - node-reboot-automation=node-reboot-automation.md
      # Linked from OAuth2ProxyGateUnobserved, and the diagnostic procedure for any gated app
      # that "is up but does not work" — the gate answers an unauthenticated XHR with a 302,
      # not a 401, so the failure arrives as an HTTP 200 carrying the wrong body.
      - oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md
      # Operator procedure, not alert-linked. Reading the per-app oauth2-proxy sidecar log to tell
      # Keycloak, the sidecar and the app's own login apart. Cookie collision between two sidecars
      # presents as random session loss in ONE app's log and is only legible with both side by side,
      # so the page has to be reachable at 3am without going looking for it in git.
      # SIBLING PAGE to oauth2-proxy-auth-failures above, and they are deliberately both here: this
      # one is the SESSION-LIFECYCLE procedure (why a session died), that one is the DIAGNOSTIC
      # TRIAGE (which of the three layers refused, and how to tell a 302-to-200 from a real success).
      - oauth2-proxy-session=oauth2-proxy-session.md
      # Operator procedure, not an alert page (no alert links to it — the
      # alert-runbooks-resolvable check treats that as advisory). It is served
      # because the plex.tv claim window is short and is not a thing to go looking
      # for in git: /runbooks/plex-claim has to resolve the moment it is needed.
      - plex-claim=plex-claim.md
      # Operator procedure, not an alert page, and NO alert is possible for it: plex renders no
      # metricsEndpoint, so there is no ServiceMonitor, no transcode-mode metric and nothing a rule
      # could evaluate. Served because the feature it completes is INERT until a human ticks a box in
      # the Plex UI — the one step GitOps cannot reach — and a page nobody can find is how a shipped
      # feature stays switched off for months.
      - plex-quicksync=plex-quicksync.md
      # Operator procedure, not alert-linked: how to STOP the PM agent service (three ways,
      # increasing severity), how to read why it is idle, how to tell a worker that was
      # correctly refused by the gate from one that broke, and the one-year expiry of its
      # Anthropic subscription token — which nothing warns about, because Claude Code's expiry
      # notice only fires for an interactive login, never for CLAUDE_CODE_OAUTH_TOKEN.
      - pm-agent=pm-agent.md
      - pvc-capacity=pvc-capacity.md
      - targets-down=targets-down.md
      - update-rollback=update-rollback.md
      - vpn-down=vpn-down.md
      - vpn-egress-stale=vpn-egress-stale.md
      # Operator procedure, not an alert page. Served because the component it completes is INERT
      # until three off-repo manual steps are done (kernel module, server key, WAN port-forward) —
      # and because the one check that actually proves it cannot be run from inside the house.
      # Peer-level alerting does not exist yet (it needs an exporter); this page is the substitute.
      - vpn-remote-access=vpn-remote-access.md