jakman85andClaude Opus 5
e82dab5df5
PR #69 claimed a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement
lease it holds, and excluded the VPN ingress node from announcing on that basis.
The claim makes a sharp prediction; today's power-cycle put ALL EIGHT
cilium-l2announce-* leases on k3s-prod-1, and the prediction failed.
Measured 2026-10-17 from the wireguard pod's host netns on each node: the holder
reaches 192.168.2.20/.22/.23/.24 normally (curl 200 in 18ms on .20) while its
ip neigh for every one of them sits at FAILED. The asserted cause is present
without the asserted effect, which falsifies it. .21 (unifi) fails from all three
nodes and is a separate pre-existing fault.
It is inert because no path resolves a VIP by ARP: cilium-dbg reports
KubeProxyReplacement on [eno1, wg0] and programs 192.168.2.20:443 =>
127.0.0.1:13606, so VIPs are DNAT'd in BPF before any neighbour lookup, on the
forwarded path as well as the host one.
Both probes behind the original diagnosis were bad. `ip neigh` FAILED is
permanent and harmless, and a ping artefact. `curl --interface wg0` — the control
used to rule out a host-netns artefact — fails on all three nodes, including ones
whose ARP resolves, so it discriminates nothing. The deeper error: a MASQUERADEd
client is FORWARDED while every pod-runnable probe is socket-originated. Sharing
a netns is not sharing a hook.
Reverted the policy nodeSelector, the Ansible node label, both alerts, their
promtool tests and the kube-state-metrics allowlist entry. Kept documentation
only: the measurements, which probes lie, and the fact that the forwarded path
remains unmeasured without a connected client.
Also recorded, for the reported "LAN and cluster work, internet does not":
new-peer.sh has emitted a FULL tunnel (0.0.0.0/0) by default since f92707f, so
there is no full-tunnel option to add and no code change is made. The server side
is capable — ip_forward=1, the 10.90.0.0/24 MASQUERADE present with an empty
CILIUM_POST_nat ahead of it, FORWARD accepts, default route intact, nodes reach
1.1.1.1. A full tunnel egresses the HOUSE WAN, not PIA: the egress policy selects
pods by egress-class=vpn, this pod is labelled internet, and tun0 exists on
k3s-prod-3 only. Root cause NOT found; no client has connected since the
power-cycle (0 forwarded packets, no handshake), so it is not reproducible here.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
127 lines
7.7 KiB
YAML
127 lines
7.7 KiB
YAML
---
|
|
# Runbooks tier — reconciled by its OWN non-gating post-platform Flux Kustomization
|
|
# (clusters/base/runbooks.yaml, dependsOn platform-config, wait:false). Same placement and rationale as
|
|
# platform/homepage: a leaf web server must never sit in the gating `platform` tier, where a UI's
|
|
# unhealth once stalled the whole chain (DECISIONS-LOG: "Headlamp gating stall").
|
|
#
|
|
# WHY THE MANIFESTS LIVE HERE, ALONGSIDE THE DOCS, RATHER THAN IN platform/runbooks/.
|
|
# The markdown is the source of truth and the task is to serve it verbatim, which means the ConfigMap must
|
|
# be GENERATED from these files — there must not be a second hand-maintained copy of 34 documents that can
|
|
# drift from the originals. `configMapGenerator` does that, and its `files:` paths are resolved relative to
|
|
# THIS directory. A `platform/runbooks/kustomization.yaml` would therefore need `../../runbooks/<name>.md`,
|
|
# which depends on kustomize load restrictions being relaxed — true for Flux's kustomize-controller,
|
|
# refused by a plain `kustomize build`, and so a divergence between what the gate can verify and what the
|
|
# cluster does. Making this directory the Flux path removes the question entirely: no `../`, identical
|
|
# behaviour locally and in-cluster, one copy of every document. The serving manifests are tucked under
|
|
# `serving/` so `runbooks/*.md` stays exactly the flat set of pages the URLs name.
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
namespace: runbooks
|
|
resources:
|
|
- serving/namespace.yaml
|
|
- serving/nginx-config.yaml
|
|
- serving/deployment.yaml
|
|
- serving/service.yaml
|
|
# default-deny baseline + Cilium gateway-ingress allow (DNS-only egress from the baseline).
|
|
- serving/networkpolicy.yaml
|
|
- serving/httproute.yaml # apex `/runbooks` prefix (binds the Gateway `https-apex` listener)
|
|
|
|
configMapGenerator:
|
|
# ONE ConfigMap holding every runbook, keyed by the EXACT name the alert annotations use.
|
|
#
|
|
# THE KEYS ARE EXTENSIONLESS ON PURPOSE. The annotations are `https://${BASE_DOMAIN}/runbooks/<name>`
|
|
# with NO `.md`, and this repo must not change them (they are spread across four alerts.yaml files that
|
|
# other work is editing). `<key>=<file>` maps `node-down.md` on disk to the key `node-down`, the volume
|
|
# mount turns each key into a file under the nginx root, and `/runbooks/node-down` becomes a plain static
|
|
# lookup — no rewrite rule, nothing to get subtly wrong at 3am. Drop the `<key>=` and every key would
|
|
# carry `.md`, so every alert link would 404 while this file still looked correct.
|
|
#
|
|
# The generated name carries kustomize's content hash, so editing any runbook rolls the server
|
|
# automatically and a stale projection is impossible.
|
|
#
|
|
# ADDING A RUNBOOK: add the file AND its line here. The L0 check
|
|
# conformance/tests/l0-static/alert-runbooks-resolvable asserts both halves, because a file in git with
|
|
# no generator entry is reviewed, merged, apparently done — and still a 404 in the cluster.
|
|
- name: runbooks-content
|
|
options:
|
|
labels:
|
|
app.kubernetes.io/name: runbooks
|
|
files:
|
|
# The index, served at /runbooks/ (nginx `try_files /runbooks/README`).
|
|
- README=README.md
|
|
- backup-failure=backup-failure.md
|
|
- backup-missed=backup-missed.md
|
|
- backup-partial=backup-partial.md
|
|
- cdi-import-stuck=cdi-import-stuck.md
|
|
- cert-expiry=cert-expiry.md
|
|
- cert-issuance=cert-issuance.md
|
|
- cnpg-archiving=cnpg-archiving.md
|
|
- cnpg-backup-missed=cnpg-backup-missed.md
|
|
- cnpg-exporter=cnpg-exporter.md
|
|
- cnpg-primary-spread=cnpg-primary-spread.md
|
|
- cnpg-wal-growth=cnpg-wal-growth.md
|
|
# Operator procedure, not alert-linked (Crafty's local login is a second factor behind
|
|
# the Keycloak gate and its bootstrap password lives only on the PVC).
|
|
- crafty-admin-credential=crafty-admin-credential.md
|
|
- denied-flows=denied-flows.md
|
|
- flux-reconcile=flux-reconcile.md
|
|
- flux-rollout=flux-rollout.md
|
|
- flux-suspended=flux-suspended.md
|
|
- image-automation=image-automation.md
|
|
- image-policy-stalled=image-policy-stalled.md
|
|
- keycloak-client-secrets=keycloak-client-secrets.md
|
|
- keycloak-db=keycloak-db.md
|
|
- keycloak-down=keycloak-down.md
|
|
- kubevirt-control-plane=kubevirt-control-plane.md
|
|
- kubevirt-crashloop=kubevirt-crashloop.md
|
|
- kubevirt-launch-failed=kubevirt-launch-failed.md
|
|
- longhorn-capacity=longhorn-capacity.md
|
|
- longhorn-degraded=longhorn-degraded.md
|
|
- longhorn-faulted=longhorn-faulted.md
|
|
- monitoring-capacity=monitoring-capacity.md
|
|
- node-cpu=node-cpu.md
|
|
- node-disk=node-disk.md
|
|
- node-down=node-down.md
|
|
- node-io=node-io.md
|
|
- node-memory=node-memory.md
|
|
- node-memory-swap=node-memory-swap.md
|
|
# Linked from OAuth2ProxyGateUnobserved, and the diagnostic procedure for any gated app
|
|
# that "is up but does not work" — the gate answers an unauthenticated XHR with a 302,
|
|
# not a 401, so the failure arrives as an HTTP 200 carrying the wrong body.
|
|
- oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md
|
|
# Operator procedure, not alert-linked. Reading the per-app oauth2-proxy sidecar log to tell
|
|
# Keycloak, the sidecar and the app's own login apart. Cookie collision between two sidecars
|
|
# presents as random session loss in ONE app's log and is only legible with both side by side,
|
|
# so the page has to be reachable at 3am without going looking for it in git.
|
|
# SIBLING PAGE to oauth2-proxy-auth-failures above, and they are deliberately both here: this
|
|
# one is the SESSION-LIFECYCLE procedure (why a session died), that one is the DIAGNOSTIC
|
|
# TRIAGE (which of the three layers refused, and how to tell a 302-to-200 from a real success).
|
|
- oauth2-proxy-session=oauth2-proxy-session.md
|
|
# Operator procedure, not an alert page (no alert links to it — the
|
|
# alert-runbooks-resolvable check treats that as advisory). It is served
|
|
# because the plex.tv claim window is short and is not a thing to go looking
|
|
# for in git: /runbooks/plex-claim has to resolve the moment it is needed.
|
|
- plex-claim=plex-claim.md
|
|
# Operator procedure, not an alert page, and NO alert is possible for it: plex renders no
|
|
# metricsEndpoint, so there is no ServiceMonitor, no transcode-mode metric and nothing a rule
|
|
# could evaluate. Served because the feature it completes is INERT until a human ticks a box in
|
|
# the Plex UI — the one step GitOps cannot reach — and a page nobody can find is how a shipped
|
|
# feature stays switched off for months.
|
|
- plex-quicksync=plex-quicksync.md
|
|
# Operator procedure, not alert-linked: how to STOP the PM agent service (three ways,
|
|
# increasing severity), how to read why it is idle, how to tell a worker that was
|
|
# correctly refused by the gate from one that broke, and the one-year expiry of its
|
|
# Anthropic subscription token — which nothing warns about, because Claude Code's expiry
|
|
# notice only fires for an interactive login, never for CLAUDE_CODE_OAUTH_TOKEN.
|
|
- pm-agent=pm-agent.md
|
|
- pvc-capacity=pvc-capacity.md
|
|
- targets-down=targets-down.md
|
|
- update-rollback=update-rollback.md
|
|
- vpn-down=vpn-down.md
|
|
- vpn-egress-stale=vpn-egress-stale.md
|
|
# Operator procedure, not an alert page. Served because the component it completes is INERT
|
|
# until three off-repo manual steps are done (kernel module, server key, WAN port-forward) —
|
|
# and because the one check that actually proves it cannot be run from inside the house.
|
|
# Peer-level alerting does not exist yet (it needs an exporter); this page is the substitute.
|
|
- vpn-remote-access=vpn-remote-access.md
|