jakman85andClaude Opus 5
f5353cbd30
A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it
holds itself: the announcer does not answer its own ARP request, so on the
lease holder the neighbour entry settles at FAILED and packets to that VIP die
before leaving the node. Measured on prod 2026-10-08, symmetric across 3 nodes
x 5 VIPs -- each node reached every VIP EXCEPT the ones it alone announced.
platform/vpn-ingress is uniquely exposed: it runs WireGuard on hostNetwork and
MASQUERADEs remote clients to the node's own address, so a VPN client takes
the node's LAN path and inherits its ARP failures. The router's udp/51820
forward pointed at k3s-prod-1, which held the shared Gateway lease, so all 16
Gateway hostnames were dead over the VPN -- while PiHole DNS kept answering
(its VIP has two Services and k3s-prod-2 held the other lease). The tunnel
handshook, names resolved, every app hung, and nothing reported it.
on k3s-prod-1: ip neigh show 192.168.2.20 -> FAILED (its own VIP)
on prod-2/3: ip neigh show 192.168.2.20 -> lladdr ... (prod-1), 443 OK
Mark the node and exclude it from announcing:
* node_baseline labels the forwarded node (node_baseline_vpn_ingress_node,
k3s-prod-1 on prod) homelab.keithjackson.dev/vpn-ingress=true, and removes
the label from any node that no longer matches so moving it MOVES the label.
* l2-announcement-policy.yaml excludes that label, so the node ARP-resolves
every VIP. Fixes all VIPs for VPN clients, not just the one that collided.
Cost: announcement candidacy drops from 3 nodes to 2.
* VpnIngressNodeAnnouncesL2VIP joins kube_lease_owner against kube_node_labels
(label added to the kube-state-metrics allowlist), with
VpnIngressNodeLabelMissing as its watchdog -- NotIn on an absent key matches,
so a missing label would silently re-arm the whole failure AND leave the
critical alert inert. Four promtool cases, including that inertness.
A node LABEL rather than a cluster-vars knob because the alert must read it
too, and extract_rules.py rightly refuses a ${...} placeholder in an alert
expr -- the rule unit-tested would not be the rule deployed. One source of
truth, and it touches no sensitive path.
Runbook/docs: step 3 of vpn-remote-access.md said "pick any node" and
"failover is this one line". Both false; the first caused this.
UNAPPLIED and unproven end to end; the Ansible tasks have never run. See the
handoff entry's third section.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
132 lines
8.1 KiB
YAML
132 lines
8.1 KiB
YAML
---
|
|
# Runbooks tier — reconciled by its OWN non-gating post-platform Flux Kustomization
|
|
# (clusters/base/runbooks.yaml, dependsOn platform-config, wait:false). Same placement and rationale as
|
|
# platform/homepage: a leaf web server must never sit in the gating `platform` tier, where a UI's
|
|
# unhealth once stalled the whole chain (DECISIONS-LOG: "Headlamp gating stall").
|
|
#
|
|
# WHY THE MANIFESTS LIVE HERE, ALONGSIDE THE DOCS, RATHER THAN IN platform/runbooks/.
|
|
# The markdown is the source of truth and the task is to serve it verbatim, which means the ConfigMap must
|
|
# be GENERATED from these files — there must not be a second hand-maintained copy of 34 documents that can
|
|
# drift from the originals. `configMapGenerator` does that, and its `files:` paths are resolved relative to
|
|
# THIS directory. A `platform/runbooks/kustomization.yaml` would therefore need `../../runbooks/<name>.md`,
|
|
# which depends on kustomize load restrictions being relaxed — true for Flux's kustomize-controller,
|
|
# refused by a plain `kustomize build`, and so a divergence between what the gate can verify and what the
|
|
# cluster does. Making this directory the Flux path removes the question entirely: no `../`, identical
|
|
# behaviour locally and in-cluster, one copy of every document. The serving manifests are tucked under
|
|
# `serving/` so `runbooks/*.md` stays exactly the flat set of pages the URLs name.
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
namespace: runbooks
|
|
resources:
|
|
- serving/namespace.yaml
|
|
- serving/nginx-config.yaml
|
|
- serving/deployment.yaml
|
|
- serving/service.yaml
|
|
# default-deny baseline + Cilium gateway-ingress allow (DNS-only egress from the baseline).
|
|
- serving/networkpolicy.yaml
|
|
- serving/httproute.yaml # apex `/runbooks` prefix (binds the Gateway `https-apex` listener)
|
|
|
|
configMapGenerator:
|
|
# ONE ConfigMap holding every runbook, keyed by the EXACT name the alert annotations use.
|
|
#
|
|
# THE KEYS ARE EXTENSIONLESS ON PURPOSE. The annotations are `https://${BASE_DOMAIN}/runbooks/<name>`
|
|
# with NO `.md`, and this repo must not change them (they are spread across four alerts.yaml files that
|
|
# other work is editing). `<key>=<file>` maps `node-down.md` on disk to the key `node-down`, the volume
|
|
# mount turns each key into a file under the nginx root, and `/runbooks/node-down` becomes a plain static
|
|
# lookup — no rewrite rule, nothing to get subtly wrong at 3am. Drop the `<key>=` and every key would
|
|
# carry `.md`, so every alert link would 404 while this file still looked correct.
|
|
#
|
|
# The generated name carries kustomize's content hash, so editing any runbook rolls the server
|
|
# automatically and a stale projection is impossible.
|
|
#
|
|
# ADDING A RUNBOOK: add the file AND its line here. The L0 check
|
|
# conformance/tests/l0-static/alert-runbooks-resolvable asserts both halves, because a file in git with
|
|
# no generator entry is reviewed, merged, apparently done — and still a 404 in the cluster.
|
|
- name: runbooks-content
|
|
options:
|
|
labels:
|
|
app.kubernetes.io/name: runbooks
|
|
files:
|
|
# The index, served at /runbooks/ (nginx `try_files /runbooks/README`).
|
|
- README=README.md
|
|
- backup-failure=backup-failure.md
|
|
- backup-missed=backup-missed.md
|
|
- backup-partial=backup-partial.md
|
|
- cdi-import-stuck=cdi-import-stuck.md
|
|
- cert-expiry=cert-expiry.md
|
|
- cert-issuance=cert-issuance.md
|
|
- cnpg-archiving=cnpg-archiving.md
|
|
- cnpg-backup-missed=cnpg-backup-missed.md
|
|
- cnpg-exporter=cnpg-exporter.md
|
|
- cnpg-primary-spread=cnpg-primary-spread.md
|
|
- cnpg-wal-growth=cnpg-wal-growth.md
|
|
# Operator procedure, not alert-linked (Crafty's local login is a second factor behind
|
|
# the Keycloak gate and its bootstrap password lives only on the PVC).
|
|
- crafty-admin-credential=crafty-admin-credential.md
|
|
- denied-flows=denied-flows.md
|
|
- flux-reconcile=flux-reconcile.md
|
|
- flux-rollout=flux-rollout.md
|
|
- flux-suspended=flux-suspended.md
|
|
- image-automation=image-automation.md
|
|
- image-policy-stalled=image-policy-stalled.md
|
|
- keycloak-client-secrets=keycloak-client-secrets.md
|
|
- keycloak-db=keycloak-db.md
|
|
- keycloak-down=keycloak-down.md
|
|
- kubevirt-control-plane=kubevirt-control-plane.md
|
|
- kubevirt-crashloop=kubevirt-crashloop.md
|
|
- kubevirt-launch-failed=kubevirt-launch-failed.md
|
|
- longhorn-capacity=longhorn-capacity.md
|
|
- longhorn-degraded=longhorn-degraded.md
|
|
- longhorn-faulted=longhorn-faulted.md
|
|
- monitoring-capacity=monitoring-capacity.md
|
|
- node-cpu=node-cpu.md
|
|
- node-disk=node-disk.md
|
|
- node-down=node-down.md
|
|
- node-io=node-io.md
|
|
- node-memory=node-memory.md
|
|
- node-memory-swap=node-memory-swap.md
|
|
# Linked from OAuth2ProxyGateUnobserved, and the diagnostic procedure for any gated app
|
|
# that "is up but does not work" — the gate answers an unauthenticated XHR with a 302,
|
|
# not a 401, so the failure arrives as an HTTP 200 carrying the wrong body.
|
|
- oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md
|
|
# Operator procedure, not alert-linked. Reading the per-app oauth2-proxy sidecar log to tell
|
|
# Keycloak, the sidecar and the app's own login apart. Cookie collision between two sidecars
|
|
# presents as random session loss in ONE app's log and is only legible with both side by side,
|
|
# so the page has to be reachable at 3am without going looking for it in git.
|
|
# SIBLING PAGE to oauth2-proxy-auth-failures above, and they are deliberately both here: this
|
|
# one is the SESSION-LIFECYCLE procedure (why a session died), that one is the DIAGNOSTIC
|
|
# TRIAGE (which of the three layers refused, and how to tell a 302-to-200 from a real success).
|
|
- oauth2-proxy-session=oauth2-proxy-session.md
|
|
# Operator procedure, not an alert page (no alert links to it — the
|
|
# alert-runbooks-resolvable check treats that as advisory). It is served
|
|
# because the plex.tv claim window is short and is not a thing to go looking
|
|
# for in git: /runbooks/plex-claim has to resolve the moment it is needed.
|
|
- plex-claim=plex-claim.md
|
|
# Operator procedure, not an alert page, and NO alert is possible for it: plex renders no
|
|
# metricsEndpoint, so there is no ServiceMonitor, no transcode-mode metric and nothing a rule
|
|
# could evaluate. Served because the feature it completes is INERT until a human ticks a box in
|
|
# the Plex UI — the one step GitOps cannot reach — and a page nobody can find is how a shipped
|
|
# feature stays switched off for months.
|
|
- plex-quicksync=plex-quicksync.md
|
|
# Operator procedure, not alert-linked: how to STOP the PM agent service (three ways,
|
|
# increasing severity), how to read why it is idle, how to tell a worker that was
|
|
# correctly refused by the gate from one that broke, and the one-year expiry of its
|
|
# Anthropic subscription token — which nothing warns about, because Claude Code's expiry
|
|
# notice only fires for an interactive login, never for CLAUDE_CODE_OAUTH_TOKEN.
|
|
- pm-agent=pm-agent.md
|
|
- pvc-capacity=pvc-capacity.md
|
|
- targets-down=targets-down.md
|
|
- update-rollback=update-rollback.md
|
|
- vpn-down=vpn-down.md
|
|
- vpn-egress-stale=vpn-egress-stale.md
|
|
# `VpnIngressNodeAnnouncesL2VIP` — the INBOUND remote-access VPN, not the egress tunnel the two
|
|
# pages above cover. A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease
|
|
# it holds itself, and vpn-ingress MASQUERADEs clients to the node's own address, so the
|
|
# forwarded node must announce nothing (measured 2026-10-08).
|
|
- vpn-ingress-l2-collision=vpn-ingress-l2-collision.md
|
|
# Operator procedure, not an alert page. Served because the component it completes is INERT
|
|
# until three off-repo manual steps are done (kernel module, server key, WAN port-forward) —
|
|
# and because the one check that actually proves it cannot be run from inside the house.
|
|
# Peer-level alerting does not exist yet (it needs an exporter); this page is the substitute.
|
|
- vpn-remote-access=vpn-remote-access.md
|