0 Fork 0

Files

T

jakman85andClaude Opus 5 b7341b329b fix(runbooks): declare whole-repo scope, drop the bespoke l0 stage, +cnpg-primary-spread

Three corrections after rebasing onto main.

1. THE GUARD CAUGHT A REAL ONE ON ITS FIRST CONTACT WITH main. main had gained
   `CNPGPrimariesCoLocated` (platform/cnpg/alerts.yaml) pointing at
   /runbooks/cnpg-primary-spread, with no runbook. The check failed the rebase exactly as
   designed — 46 annotations, 35 names, 34 present — naming both the missing file AND the
   missing configMapGenerator entry. runbooks/cnpg-primary-spread.md now exists: what
   co-located primaries actually cost here (blast radius, not an outage, because
   instances: 3 + required anti-affinity), why the drift is uncorrectable-by-declaration
   (CNPG has no primary-placement field), the readiness gate that keeps the rule quiet
   during a real node outage, and the ~9s-per-database switchover rebalance, one cluster
   at a time. The 2026-09-07 outage in which all three primaries sat on one node is the
   reason the alert exists at all.

2. WHOLE-REPO SCOPE, NOT A HAND-WRITTEN STAGE. main's refactor replaced ci/l0.sh's
   per-check "UNCONDITIONAL" stages with a declaration:
   `conformance.home.k3s/scope: "whole-repo"`, unioned into every gate run by Stage 3b's
   `--include-scope`. My +38-line Stage 3f is removed (ci/l0.sh is now byte-identical to
   main) and the annotation is declared instead. Assertion (7) of the check was rewritten
   to match: it no longer asserts its own path appears in ci/l0.sh — it asserts the test
   DECLARES the scope and that ci/l0.sh still passes --include-scope, since neither half
   alone makes the check load-bearing. That matters here because
   ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02", not
   R-OBS-04, and maps platform/kubevirt/* and runbooks/* to nothing at all.

3. A FACTUAL CORRECTION IN ALL 35 RUNBOOKS. Every page claimed `dev.sh` defaults its
   current-context to k3s-dev. main's `9b4744d fix(devcontainer): dev.sh defaults to
   prod` made that false, and a runbook that misstates which cluster a command talks to is
   worse than one that says nothing. The pages now say what is durably true: the default
   has already changed once, so naming `--context k3s-prod` is what makes a command mean
   the same thing next month, and the per-invocation banner is there to be read. The
   check's own failure message was corrected the same way.

Re-verified after all three: check green at 374 assertions (46 annotations, 35/35 runbooks
present, 35/35 projected), re-applied to prod, and re-probed in-cluster through the
Gateway — node-down, keycloak-down, keycloak-client-secrets, cnpg-archiving,
longhorn-faulted, vpn-down, node-io and cnpg-primary-spread all HTTP 200 with
text/plain; charset=utf-8 and 8.9-12.1 KB of real body; /runbooks/ index 200; an unknown
name 404s with a pointer rather than a blank page; and homepage still answers the apex `/`,
confirming Gateway API longest-path precedence.

Full merge gate: PASS (spec-immutability, lint, l0, render-golden, l1 all green;
prom-rules SKIPPED for want of promtool in the image; L2/L3 SKIPPED, no cluster).

Co-Authored-By: Claude Opus 5 <[email protected]>

2026-09-13 00:00:24 -05:00

11 KiB

Raw Blame History

vpn-down — The egress VPN tunnel is down

Alert: VPNTunnelDown (critical, 5m) File: platform/monitoring/alerts.yaml, group platform-health.networking Expression:

absent(node_network_receive_bytes_total{device="tun0"})
and on() (
    (count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
  or (count(count_over_time(node_network_receive_bytes_total{device="tun0"}[1h])) > 0)
)

What this means here

There is no in-cluster egress-gateway pod on this platform (ADR-0015). The tunnel is a host systemd service — vpn-egress-tunnel, OpenVPN to PIA — running on one node, and steering is Cilium eBPF via a CiliumEgressGatewayPolicy (vpn-egress-steering). OpenVPN creates and destroys tun0 with the connection, so the tun0 byte-counter series exists if and only if the tunnel is up. The rule keys on that presence rather than node_network_up, because tun0 reports operstate=unknown and node_network_up is therefore 0 even when the tunnel is fine.

Live state, measured 2026-09-13: the tunnel is up (node_network_receive_bytes_total{device="tun0"} = 1 series) and the gateway node is labelled (kube_node_labels{...egress_gateway="true"} = 1 series), so this alert is armed and quiet. The gateway is k3s-prod-3 (192.168.2.12, eno1).

The two facts that decide your response:

Cilium OSS has no egress-gateway failover. Gateway-node loss is an outage, not a degradation. The ADR-0016 supersede drafted on 2026-09-06 is still open. Worse, Cilium OSS picks "the first node in lexical ordering" among nodeSelector matches — so labelling k3s-prod-1 would silently steal the gateway role and the public egress IP from k3s-prod-3. [vpn_egress_gateway_prod] is a separate inventory group holding only k3s-prod-3 for this reason; run the role as -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3, with --list-hosts first, every time.
The absent() was originally evaluated cluster-wide and became a permanent false alarm. On a cluster with no gateway node it could never clear, and from 2026-09-05T21:23Z it fired continuously into email-critical — unbounded noise on the one channel that must stay trustworthy. The two guards OR'd above are the fix: the alert arms itself the moment a node is labelled, and stays armed through a kube-state-metrics outage. Before that, the very first version keyed on absent(gluetun_vpn_up), a metric gluetun's control API never exported at all — a pre-existing perma-false-alarm. If this alert is firing, verify the tunnel by hand before believing it.

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two dev.sh invocations at once; they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git (git push origin main is the deploy). Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

Confirm on the host, not from Prometheus:

ssh k3s-prod-3 'systemctl status vpn-egress-tunnel --no-pager | head -20; echo; ip -br addr show tun0; echo; ip link show tun0 | head -2'

Which node is the gateway, and is it labelled?

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get nodes -L homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink; echo; \
   kubectl --context k3s-prod get ciliumegressgatewaypolicies.cilium.io -o yaml | grep -A6 "nodeSelector\|egressIP\|interface"'

The check that catches the silent failure (see cause 1 below) — does Cilium's datapath point at the interface that actually exists?

ssh k3s-prod-3 'ip -o link | awk -F: "{print \$1, \$2}" | grep tun0'
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg bpf egress list'

If the Egress Ifindex in that map is not the live tun0 ifindex, egress is broken even though this alert is quiet.

Is traffic actually leaving through the tunnel?

ssh k3s-prod-3 'curl -s --interface tun0 --max-time 10 https://ipinfo.io/ip; echo; nft list table inet vpn_killswitch 2>/dev/null | grep -E "counter|killswitch_" | head'

Likely causes, most likely first

1. The tunnel is UP but Cilium is steering at a dead interface (the silent one)

This is the richest failure on the platform and it does not set off this alert. Cilium resolves a CiliumEgressGatewayPolicy's interface: tun0 exactly once, caches (egress IP, ifindex), and never re-resolves. On k3s-dev-3, cilium-dbg bpf egress list showed Egress Ifindex 3 — there is no ifindex 3 on that node; the live tun0 was 1780. Egress was dead for about 80 minutes with nothing alerting and the conformance ledger still reading green. egress-gateway-reconciliation-trigger-interval: 1s does not cover it (netlink link events are not Kubernetes object events).

Measured behaviour, so you can pick a lever that works:

action	result
annotate the CEGP	map → 1780 in 8s, PIA IP returns
`systemctl restart vpn-egress-tunnel`	tun0 1780→1985, map stays 1780, still broken
the exact `vpn-killswitch` stop/30s/start sequence	still broken 20s+ later
set any unrelated label on the gateway node	recovers within 10s

The durable fix is in roles/vpn_egress_gateway: the OpenVPN up-script publishes a node label homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex> on every connect (via systemctl start --no-block, so it can never block the tunnel), plus a 2-minute convergence timer for boot. A node label, not a CEGP annotation — a node writing to GitOps state is a drift fight. Nothing monitors this coupling yet; that gap is exactly why dev sat dead for 80 minutes.

2. PIA handed two gateways the same tunnel address

Standing rule: one PIA server per gateway. The vaulted .ovpn pins one server, so when prod came up green, dev went down: both gateways dialled 158.173.241.112:1198 and PIA gave both the tunnel address 10.72.192.7. Each reconnect steals it — a measured ~2-minute mutual-eviction cycle (dev restarting at 22:57:05, 22:59:03, 23:01:05, 23:03:05, 23:05:06 UTC). Fix: the role var vpn_egress_gateway_pia_remote ("<host> <port>") rewrites only the .ovpn remote line; prod is pinned to 158.173.241.135 (pool 158.173.241.{34,55,112,135,239}). "A single green probe is not a stability check."

3. PIA "unreachable" — suspect a stale server list first

The 2026-08-22/23 chain went port collision → UDP/53 → 8080 → "MTU" (Cilium VXLAN pod MTU 1450 is real but irrelevant to a 14-byte handshake) → "the network blackholes PIA" (wrong) before the actual cause: gluetun's stale bundled server list, dialling retired 46.246.x/212.112.x addresses when live SE Stockholm is 158.173.241.x. Lesson recorded verbatim: when a client "can't reach" the provider, suspect the stale server list first.

4. The gateway node is down

absent() also fires when the gateway node or its node-exporter is gone, so this alert covers gateway-node-down by design. Go to /runbooks/node-down. Remember: there is no failover — while k3s-prod-3 is down, VPN-classed egress is an outage.

5. Nothing is wrong and the alert is lying

Given the two false-alarm post-mortems above, always confirm ip -br addr show tun0 on the host before acting.

What the kill-switch does and does not do

Proven with counters on k3s-prod-3, because the intuitive reading is wrong: with bpf.masquerade: true (which Egress Gateway requires) SNAT and egress happen in eBPF and bpf_redirect() skips netfilter POSTROUTING. Measured: forward saddr=pod-CIDR 0, postrouting oif WAN saddr=pod-CIDR 0. What actually fails closed is Cilium's own drop — tcpdump during tunnel stop and restart showed 0 packets leaving either device. The nft table is kept as defence-in-depth for node-originated tunnel-sourced traffic and was proven both ways: ping -I 10.72.192.7 with the ip-rule → 3/3 at 144 ms with killswitch_accept advancing; rule removed → killswitch_drop 0→3 packets / 252 bytes, 100% loss. Every rule carries a counter, so: if killswitch_drop ever counts in normal operation, the datapath has changed. A reported "leak window" was measured as no WAN leak — a stale egress IP follows the cached ifindex, not a routing lookup. Note the .nft file uses table + flush table, which clears rules but leaves named-set elements intact.

When to escalate or stop

Escalate immediately if the gateway node is down. No failover exists; this is an outage with no in-cluster remedy.
Never label a second node egress-gateway=true. Lexical ordering means k3s-prod-1 would silently take the role and the public egress IP.
Do not restart the tunnel expecting Cilium to follow — measured above, it does not. If the map is stale, touching a node label is the lever that works (operator action).
Do not remove or "simplify" the nft kill-switch because the counters read zero. Zero on the pod path is the expected, measured result of eBPF masquerade; the table guards a different path.
A leak concern is a security matter (docs/07) — escalate rather than experimenting.

Source of truth: runbooks/vpn-down.md. Index: /runbooks/ Records: DECISIONS-LOG.md "2026-09-12 — VPN egress on PROD (k3s-prod-3), and three things the repo believed that are not true"; the 2026-08-22/23 gluetun chain; commit 155df8e (R-NET-06 re-proven on prod, recovery ≤3s).

11 KiB Raw Blame History