0 Fork 0

Files

T

jakman85andClaude Opus 5 a3f4ddb971 merge: feat/egress-gateway-staleness-alert — finished so it can actually pass the gate

Brings the two alerts for the silent egress-gateway failure: VPNEgressSteeringStale
(critical) and VPNEgressUplinkLabelMissing (warning), plus the kube-state-metrics
allowlist entry that makes them non-inert.

WHY THE BRANCH SAT IN THE BACKLOG: it could not pass the merge gate. Three
whole-repo checks failed on it, and each one was a real gap rather than a nuisance:

  * alert-runbooks-resolvable — both alerts link to /runbooks/vpn-egress-stale,
    which did not exist. An alert firing now would hand the operator a 404.
  * alert-rules-unit-tested — alerts.yaml is already a covered file, so new rules
    in it must be tested or a "tested" file quietly grows untested rules.
  * alert-metrics-have-scrape-targets — see the guard bug below.

WHAT THE ALERTS CATCH. egress-class=vpn pods leave through one gateway node's tun0;
Cilium resolves it to an ifindex and caches it. An OpenVPN reconnect destroys and
re-creates tun0 with a NEW ifindex, and the node label
homelab.keithjackson.dev/egress-uplink=<addr>-<ifindex> is what forces Cilium to
re-resolve. If the label does not follow the tunnel, the cache points at a destroyed
interface and every vpn-class pod is silently dropped — while tun0 stays up and
VPNTunnelDown says nothing. That ran ~80 minutes undetected on the dev gateway.

Verified this is live on prod, not inert: k3s-prod-3 carries egress-gateway=true and
egress-uplink=10.150.192.86-139, and vpn-egress-steering has been in place 14 days.
The label was exported on 0 series before this change, so the allowlist entry is
what switches the alerts on.

runbooks/vpn-egress-stale.md — written and wired into the configMapGenerator, with
the four required operator sections. It leads with proving egress from a pod the
policy selects, because the symptom is "outbound traffic silently disappears", and
it says plainly not to "fix" this by deleting the policy: that does not restore
egress, it sends vpn-class traffic out of the real ISP connection, which is the leak
the class exists to prevent.

platform/monitoring/tests/vpn-egress.test.yaml — four promtool fixtures. 1 and 2 are
the SAME tunnel reconnect with and without the label following it, which isolates the
mechanism the rule actually keys on: a label VALUE change is not a changing sample,
it retires one series and starts another. 3 is the inert case (gateway node, no uplink
label) where the critical rule CANNOT fire, which is why the warning exists.

Two fixes the branch needed that were defects in the checks themselves:

1. alert-metrics-have-scrape-targets asserted the KSM allowlist matched
   `nodes=[...egress-gateway]` with a literal closing bracket — i.e. that the list
   contained EXACTLY that one label. Adding a second label correctly (egress-uplink,
   which these alerts need) therefore FAILED the check while the property it guards
   was perfectly satisfied. Loosened to `[,\]]`: in the list, not the whole list. A
   guard that breaks when you correctly extend the thing it guards teaches people to
   edit the guard, which is how it stops guarding.

2. UNTESTED_ALERTS listed VPNTunnelDown, and the check flagged it as stale because
   "asserted on" is a bare word match over the test file — my header COMMENT
   mentioning VPNTunnelDown was enough to count as coverage. Rather than delete the
   register entry and claim coverage that did not exist, the fixtures now assert
   VPNTunnelDown QUIET in all four cases. That required making them faithful: they
   had no node_network_receive_bytes_total{device="tun0"} at all, so VPNTunnelDown
   would have been firing throughout unnoticed, because promtool only checks the
   alertnames you name. A healthy tun0 passes traffic, and that silence IS the claim
   these rules exist for.

NOTE FOR LATER, not fixed here: that word-match means any alert named in prose counts
as tested. Tightening it to `alertname:` positions could reveal other prose-only
"coverage", and the register is shrink-only, so the cascade wants doing deliberately
rather than at 01:00 unattended.

Conflicts resolved: kept HEAD's Velero custom-resource-state block and took only the
branch's one-line allowlist change (14 lines down to 1); inserted the branch's
2026-09-12 DECISIONS-LOG entry in date order rather than appending it out of sequence.

Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips,
0 unwired. The new unit tests pass under the pinned promtool 3.14.0.

Co-Authored-By: Claude Opus 5 <[email protected]>

2026-09-19 00:52:06 -05:00

9.4 KiB

Raw Blame History

Alert runbooks — home-network-k3s prod

This is the index page served at https://home.keithjackson.dev/runbooks/.

Every alert rule in this platform carries an annotation of the form

runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"

46 of them, over the 35 names below. Until this directory existed, that path returned HTTP 404 on the live cluster: every alert that reached a human handed them a dead link at the exact moment they needed help. These pages are what makes the link resolve.

How to run the commands in these runbooks

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down.

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — k3s-dev until dev was decommissioned on 2026-09-13, k3s-prod now. A command that names its context still means the same thing next month; one that relies on the default does not, and the failure is silent. CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster — read it. Do not run two dev.sh invocations at once; they share one container.

Use fully-qualified resource names, always. Three CRDs on this cluster register the kind Backup — longhorn.io/v1beta2, postgresql.cnpg.io/v1, velero.io/v1 — and the bare short name resolves to Longhorn's, which holds zero objects. kubectl get backup -A therefore prints "No resources found" on a cluster holding 6 Velero and 27 CNPG backups (verified on prod 2026-09-13). Acting on that false negative is how a CLI quirk becomes real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

git push origin main is the deploy; there is no other apply step. Durable changes go through git.

Alerts you should not trust the silence of (measured 2026-09-13)

The Longhorn and cert-manager scrapes were ENABLED on 2026-09-13, so the Longhorn row below is no longer inert — and re-keying the two robustness rules the same day fixed a SECOND inertness the scrape had left in place (the metric is one-hot over a state label, not a 0–3 enum). Take the Longhorn row as history; the rest of this table still holds, and Flux-controller and Hubble/Cilium rule groups still evaluate against nothing and cannot fire:

Missing scrape job	Inert alerts
Longhorn	`LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `LonghornDiskAllocationHigh`, `LonghornDiskUnschedulable`
cert-manager	`CertificateExpiringSoon`, `CertificateExpiryImminent`, `CertificateNotReady`
Flux (`gotk_*`)	`FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed`
Hubble (`hubble_drop_total`)	`HubbleDeniedFlowSpike`
etcd (`kubeEtcd` disabled for k3s, deliberately)	no etcd fsync alerting exists at all

Each affected runbook says so on its own page and gives the by-hand equivalent. See targets-down.

The runbooks

Nodes

node-down — a node's node-exporter is unscrapable; the node may be gone, or its kubelet may have stopped posting status while the host stays up (the 2026-09-07 shape).
node-io — disk I/O pressure/starvation. The etcd WAL shares a partition with Longhorn replicas; this is the alert that would have called the 2026-09-07 outage and did not exist at the time.
node-memory — memory pressure. Includes why SwapTotal == 0 is not evidence swap is off.
node-disk — a node filesystem filling up. /data is the one that matters.
node-cpu — CPU PSI and run-queue depth, and how to tell CPU contention from I/O contention.

Storage

longhorn-degraded — a volume has lost redundancy. Normal after any reboot, and the rebuild is the expensive part. Freeze/thaw commands here.
longhorn-faulted — no usable replica. Data at risk; stop before anything destructive.
longhorn-capacity — a node's Longhorn disk running out of room for replicas (allocation, not PVC fill; for PVC fill see pvc-capacity / monitoring-capacity).
pvc-capacity — PVC filling up / almost full / predicted to fill. These rules work.
monitoring-capacity — Prometheus or Loki running out of disk; don't let the monitor blind itself.

GitOps and updates

flux-reconcile — a Flux resource stuck at Ready=False. Which tier decides the blast radius.
flux-suspended — reconciliation turned off and forgotten. Drift becomes silent.
update-rollback — a HelmRelease failed its health gate and is rolling back.
image-automation — the git write-back engine is failing (feature is disabled here).
image-policy-stalled — an ImagePolicy/ImageRepository cannot resolve a tag.

Certificates

cert-issuance — ACME issuance failing. The DNS-01 self-check is the usual cause.
cert-expiry — a certificate is expiring. Almost always a late symptom of cert-issuance.

Networking

vpn-down — the egress VPN tunnel. Explains what the kill-switch actually does, and the silent egress-gateway staleness this alert cannot see — which now has its own alerts and page below.
vpn-egress-stale — the tunnel is UP but egress-class=vpn pods have no egress: Cilium's cached ifindex outlived a reconnect. Also covers the watchdog for that rule going inert.
denied-flows — policy-denied flow spike. The Cilium hairpin/identity regression class.

Backups and DR

backup-failure — a Velero backup failed. The R2 object-tagging incompatibility is documented here.
backup-partial — a Velero backup completed with partial failures, and which skips are deliberate.
backup-missed — no successful Velero backup in 24h.
cnpg-archiving — WAL archiving stuck / no recoverability point. An availability incident, not just a backup one.
cnpg-backup-missed — no recent CNPG base backup. Velero does not cover these databases.
cnpg-wal-growth — pg_wal not being recycled. Never delete WAL by hand.
cnpg-exporter — the CNPG SQL exporter erroring: a blind spot that looks exactly like health.
cnpg-primary-spread — two or more CNPG primaries on one node. Blast radius, not an outage, and the rebalance costs ~9s of writes per database — one cluster at a time.

Identity

keycloak-down — the IdP is down. Blast radius, and the break-glass path that still works.
keycloak-db — Keycloak's CloudNativePG database is unavailable.
keycloak-client-secrets — the client/URL reconciler failing or silent. The 2026-09-06 lockout that every green signal missed.

Virtualisation

kubevirt-control-plane — VM lifecycle operations will fail; running VMs keep running.
kubevirt-launch-failed — a VMI reached the Failed phase. The three k3s-specific launch traps.
kubevirt-crashloop — a virt-launcher pod restarting repeatedly.
cdi-import-stuck — a DataVolume import not progressing. Check node I/O before restarting it.

Editing a runbook

Each page is runbooks/<name>.md in the home-network-k3s-impl repo. Edit it, push to main, and Flux re-renders the ConfigMap and rolls the two-replica server; kustomize's content hash on the generated ConfigMap name is what triggers the roll.

Adding a new alert with a runbook_url requires the matching runbooks/<name>.md and its <name>=<name>.md entry in runbooks/kustomization.yaml. Both are enforced by conformance/tests/l0-static/alert-runbooks-resolvable, which runs unconditionally in the merge gate — "the file exists in git" and "the URL resolves in the cluster" are different claims, and it checks both.

Every page must carry the four operator sections (What this means here, First commands, Likely causes, most likely first, When to escalate or stop), mention dev.sh, and pass --context k3s-prod. The check enforces that too, because a stub that makes the link resolve while helping nobody is worse than the 404 it replaced — it looks fixed.

Deliberately public. The UIs that declare forward-auth (pihole, jackett, qbittorrent, radarr, sonarr) really are closed: enforcement is per-pod, each one's HTTPRoute targeting its oauth2-proxy sidecar on :4180 rather than the app port. If these pages did too, keycloak-down would be unreadable exactly when Keycloak is down, keycloak-db unreadable when its database is down, and keycloak-client-secrets unreadable during the precise failure that broke all 12 OIDC clients on 2026-09-06 behind an entirely green cluster. An operator who cannot log in is this route's primary reader. The content is operational procedure — metric names, thresholds, kubectl invocations, topology — and no secrets. See runbooks/serving/httproute.yaml for the full reasoning, including how forward-auth is actually enforced here and the instruction to leave this route out of any Gateway-level ext-authz attachment if one is ever introduced.

The running narrative of real incidents on this cluster is DECISIONS-LOG.md.

9.4 KiB Raw Blame History Unescape Escape