jakman85andClaude Opus 5
b7341b329b
Three corrections after rebasing onto main. 1. THE GUARD CAUGHT A REAL ONE ON ITS FIRST CONTACT WITH main. main had gained `CNPGPrimariesCoLocated` (platform/cnpg/alerts.yaml) pointing at /runbooks/cnpg-primary-spread, with no runbook. The check failed the rebase exactly as designed — 46 annotations, 35 names, 34 present — naming both the missing file AND the missing configMapGenerator entry. runbooks/cnpg-primary-spread.md now exists: what co-located primaries actually cost here (blast radius, not an outage, because instances: 3 + required anti-affinity), why the drift is uncorrectable-by-declaration (CNPG has no primary-placement field), the readiness gate that keeps the rule quiet during a real node outage, and the ~9s-per-database switchover rebalance, one cluster at a time. The 2026-09-07 outage in which all three primaries sat on one node is the reason the alert exists at all. 2. WHOLE-REPO SCOPE, NOT A HAND-WRITTEN STAGE. main's refactor replaced ci/l0.sh's per-check "UNCONDITIONAL" stages with a declaration: `conformance.home.k3s/scope: "whole-repo"`, unioned into every gate run by Stage 3b's `--include-scope`. My +38-line Stage 3f is removed (ci/l0.sh is now byte-identical to main) and the annotation is declared instead. Assertion (7) of the check was rewritten to match: it no longer asserts its own path appears in ci/l0.sh — it asserts the test DECLARES the scope and that ci/l0.sh still passes --include-scope, since neither half alone makes the check load-bearing. That matters here because ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02", not R-OBS-04, and maps platform/kubevirt/* and runbooks/* to nothing at all. 3. A FACTUAL CORRECTION IN ALL 35 RUNBOOKS. Every page claimed `dev.sh` defaults its current-context to k3s-dev. main's `9b4744d fix(devcontainer): dev.sh defaults to prod` made that false, and a runbook that misstates which cluster a command talks to is worse than one that says nothing. The pages now say what is durably true: the default has already changed once, so naming `--context k3s-prod` is what makes a command mean the same thing next month, and the per-invocation banner is there to be read. The check's own failure message was corrected the same way. Re-verified after all three: check green at 374 assertions (46 annotations, 35/35 runbooks present, 35/35 projected), re-applied to prod, and re-probed in-cluster through the Gateway — node-down, keycloak-down, keycloak-client-secrets, cnpg-archiving, longhorn-faulted, vpn-down, node-io and cnpg-primary-spread all HTTP 200 with text/plain; charset=utf-8 and 8.9-12.1 KB of real body; /runbooks/ index 200; an unknown name 404s with a pointer rather than a blank page; and homepage still answers the apex `/`, confirming Gateway API longest-path precedence. Full merge gate: PASS (spec-immutability, lint, l0, render-golden, l1 all green; prom-rules SKIPPED for want of promtool in the image; L2/L3 SKIPPED, no cluster). Co-Authored-By: Claude Opus 5 <[email protected]>
188 lines
11 KiB
Markdown
188 lines
11 KiB
Markdown
# vpn-down — The egress VPN tunnel is down
|
|
|
|
**Alert:** `VPNTunnelDown` (critical, 5m)
|
|
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.networking`
|
|
**Expression:**
|
|
```
|
|
absent(node_network_receive_bytes_total{device="tun0"})
|
|
and on() (
|
|
(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
|
|
or (count(count_over_time(node_network_receive_bytes_total{device="tun0"}[1h])) > 0)
|
|
)
|
|
```
|
|
|
|
## What this means here
|
|
|
|
There is **no in-cluster egress-gateway pod** on this platform (ADR-0015). The tunnel is a **host
|
|
systemd service** — `vpn-egress-tunnel`, OpenVPN to PIA — running on one node, and steering is Cilium
|
|
eBPF via a `CiliumEgressGatewayPolicy` (`vpn-egress-steering`). OpenVPN creates and destroys `tun0` with
|
|
the connection, so the `tun0` byte-counter series exists **if and only if** the tunnel is up. The rule
|
|
keys on that presence rather than `node_network_up`, because `tun0` reports `operstate=unknown` and
|
|
`node_network_up` is therefore 0 even when the tunnel is fine.
|
|
|
|
Live state, measured 2026-09-13: **the tunnel is up** (`node_network_receive_bytes_total{device="tun0"}`
|
|
= 1 series) and the gateway node is labelled (`kube_node_labels{...egress_gateway="true"}` = 1 series),
|
|
so this alert is armed and quiet. The gateway is **`k3s-prod-3`** (192.168.2.12, `eno1`).
|
|
|
|
**The two facts that decide your response:**
|
|
|
|
1. **Cilium OSS has no egress-gateway failover.** Gateway-node loss is an outage, not a degradation.
|
|
The ADR-0016 supersede drafted on 2026-09-06 is still open. Worse, Cilium OSS picks *"the first node
|
|
in lexical ordering"* among nodeSelector matches — so labelling `k3s-prod-1` would **silently steal**
|
|
the gateway role and the public egress IP from `k3s-prod-3`. `[vpn_egress_gateway_prod]` is a
|
|
separate inventory group holding only `k3s-prod-3` for this reason; run the role as
|
|
`-e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3`, with `--list-hosts` first,
|
|
every time.
|
|
2. **The `absent()` was originally evaluated cluster-wide and became a permanent false alarm.** On a
|
|
cluster with no gateway node it could never clear, and from 2026-09-05T21:23Z it fired continuously
|
|
into `email-critical` — unbounded noise on the one channel that must stay trustworthy. The two
|
|
guards OR'd above are the fix: the alert arms itself the moment a node is labelled, and stays armed
|
|
through a kube-state-metrics outage. Before that, the very first version keyed on
|
|
`absent(gluetun_vpn_up)`, a metric gluetun's control API never exported at all — a pre-existing
|
|
perma-false-alarm. **If this alert is firing, verify the tunnel by hand before believing it.**
|
|
## Running the commands in this runbook
|
|
|
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
|
that hang is not the cluster being down. Every command below runs as:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
|
|
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
|
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
|
|
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
|
|
the same thing next month as it does today; one that relies on the default does not, and the failure is
|
|
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
|
|
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
|
|
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
|
|
interfere.
|
|
|
|
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
|
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
|
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
|
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
|
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
|
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
|
|
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
|
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
|
Where this runbook gives a mutating command it is for an operator, and it says so.
|
|
|
|
## First commands
|
|
|
|
Confirm on the host, not from Prometheus:
|
|
|
|
```
|
|
ssh k3s-prod-3 'systemctl status vpn-egress-tunnel --no-pager | head -20; echo; ip -br addr show tun0; echo; ip link show tun0 | head -2'
|
|
```
|
|
|
|
Which node is the gateway, and is it labelled?
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get nodes -L homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink; echo; \
|
|
kubectl --context k3s-prod get ciliumegressgatewaypolicies.cilium.io -o yaml | grep -A6 "nodeSelector\|egressIP\|interface"'
|
|
```
|
|
|
|
**The check that catches the silent failure** (see cause 1 below) — does Cilium's datapath point at the
|
|
interface that actually exists?
|
|
|
|
```
|
|
ssh k3s-prod-3 'ip -o link | awk -F: "{print \$1, \$2}" | grep tun0'
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg bpf egress list'
|
|
```
|
|
|
|
If the `Egress Ifindex` in that map is not the live `tun0` ifindex, egress is broken **even though this
|
|
alert is quiet**.
|
|
|
|
Is traffic actually leaving through the tunnel?
|
|
|
|
```
|
|
ssh k3s-prod-3 'curl -s --interface tun0 --max-time 10 https://ipinfo.io/ip; echo; nft list table inet vpn_killswitch 2>/dev/null | grep -E "counter|killswitch_" | head'
|
|
```
|
|
|
|
## Likely causes, most likely first
|
|
|
|
### 1. The tunnel is UP but Cilium is steering at a dead interface (the silent one)
|
|
This is the richest failure on the platform and it does **not** set off this alert.
|
|
**Cilium resolves a `CiliumEgressGatewayPolicy`'s `interface: tun0` exactly once**, caches
|
|
(egress IP, ifindex), and never re-resolves. On `k3s-dev-3`, `cilium-dbg bpf egress list` showed
|
|
`Egress Ifindex 3` — **there is no ifindex 3 on that node**; the live `tun0` was **1780**. Egress was
|
|
dead for about 80 minutes with nothing alerting and the conformance ledger still reading green.
|
|
`egress-gateway-reconciliation-trigger-interval: 1s` does **not** cover it (netlink link events are not
|
|
Kubernetes object events).
|
|
|
|
Measured behaviour, so you can pick a lever that works:
|
|
| action | result |
|
|
|---|---|
|
|
| annotate the CEGP | map → 1780 in 8s, PIA IP returns |
|
|
| `systemctl restart vpn-egress-tunnel` | tun0 1780→1985, **map stays 1780**, still broken |
|
|
| the exact `vpn-killswitch` stop/30s/start sequence | still broken 20s+ later |
|
|
| set **any** unrelated label on the gateway **node** | recovers within 10s |
|
|
|
|
The durable fix is in `roles/vpn_egress_gateway`: the OpenVPN up-script publishes a node label
|
|
`homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on every connect (via
|
|
`systemctl start --no-block`, so it can never block the tunnel), plus a 2-minute convergence timer for
|
|
boot. A node label, not a CEGP annotation — a node writing to GitOps state is a drift fight.
|
|
**Nothing monitors this coupling yet**; that gap is exactly why dev sat dead for 80 minutes.
|
|
|
|
### 2. PIA handed two gateways the same tunnel address
|
|
Standing rule: **one PIA server per gateway.** The vaulted `.ovpn` pins one server, so when prod came up
|
|
green, dev went down: both gateways dialled `158.173.241.112:1198` and PIA gave both the tunnel address
|
|
`10.72.192.7`. Each reconnect steals it — a measured ~2-minute mutual-eviction cycle (dev restarting at
|
|
22:57:05, 22:59:03, 23:01:05, 23:03:05, 23:05:06 UTC). Fix: the role var
|
|
`vpn_egress_gateway_pia_remote` ("`<host> <port>`") rewrites only the `.ovpn` `remote` line; prod is
|
|
pinned to **`158.173.241.135`** (pool `158.173.241.{34,55,112,135,239}`). *"A single green probe is not
|
|
a stability check."*
|
|
|
|
### 3. PIA "unreachable" — suspect a stale server list first
|
|
The 2026-08-22/23 chain went port collision → UDP/53 → 8080 → "MTU" (Cilium VXLAN pod MTU 1450 is real
|
|
but irrelevant to a **14-byte** handshake) → "the network blackholes PIA" (**wrong**) before the actual
|
|
cause: **gluetun's stale bundled server list**, dialling retired `46.246.x`/`212.112.x` addresses when
|
|
live SE Stockholm is `158.173.241.x`. Lesson recorded verbatim: when a client "can't reach" the
|
|
provider, suspect the stale server list **first**.
|
|
|
|
### 4. The gateway node is down
|
|
`absent()` also fires when the gateway node or its node-exporter is gone, so this alert covers
|
|
gateway-node-down by design. Go to `/runbooks/node-down`. Remember: **there is no failover** — while
|
|
`k3s-prod-3` is down, VPN-classed egress is an outage.
|
|
|
|
### 5. Nothing is wrong and the alert is lying
|
|
Given the two false-alarm post-mortems above, always confirm `ip -br addr show tun0` on the host before
|
|
acting.
|
|
|
|
## What the kill-switch does and does not do
|
|
|
|
Proven with counters on `k3s-prod-3`, because the intuitive reading is wrong: with
|
|
**`bpf.masquerade: true`** (which Egress Gateway requires) SNAT and egress happen in eBPF and
|
|
`bpf_redirect()` **skips netfilter POSTROUTING**. Measured: `forward` saddr=pod-CIDR **0**,
|
|
`postrouting` oif WAN saddr=pod-CIDR **0**. **What actually fails closed is Cilium's own drop** —
|
|
tcpdump during tunnel stop and restart showed **0 packets leaving either device**. The nft table is
|
|
kept as defence-in-depth for node-originated tunnel-sourced traffic and was proven both ways:
|
|
`ping -I 10.72.192.7` with the ip-rule → 3/3 at 144 ms with `killswitch_accept` advancing; rule removed
|
|
→ `killswitch_drop` 0→3 packets / 252 bytes, 100% loss. Every rule carries a `counter`, so:
|
|
**if `killswitch_drop` ever counts in normal operation, the datapath has changed.** A reported "leak
|
|
window" was **measured as no WAN leak** — a stale egress IP follows the cached ifindex, not a routing
|
|
lookup. Note the `.nft` file uses `table` + `flush table`, which clears rules but leaves named-set
|
|
elements intact.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **Escalate immediately if the gateway node is down.** No failover exists; this is an outage with no
|
|
in-cluster remedy.
|
|
- **Never label a second node `egress-gateway=true`.** Lexical ordering means `k3s-prod-1` would
|
|
silently take the role and the public egress IP.
|
|
- **Do not restart the tunnel expecting Cilium to follow** — measured above, it does not. If the map is
|
|
stale, touching a node label is the lever that works (operator action).
|
|
- **Do not remove or "simplify" the nft kill-switch** because the counters read zero. Zero on the pod
|
|
path is the expected, measured result of eBPF masquerade; the table guards a different path.
|
|
- A leak concern is a security matter (docs/07) — escalate rather than experimenting.
|
|
|
|
---
|
|
Source of truth: `runbooks/vpn-down.md`. Index: `/runbooks/`
|
|
Records: `DECISIONS-LOG.md` "2026-09-12 — VPN egress on PROD (k3s-prod-3), and three things the repo
|
|
believed that are not true"; the 2026-08-22/23 gluetun chain; commit `155df8e` (R-NET-06 re-proven on
|
|
prod, recovery ≤3s).
|