jakman85andClaude Opus 5
8e017fad91
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
190 lines
11 KiB
Markdown
190 lines
11 KiB
Markdown
# vpn-down — The egress VPN tunnel is down
|
|
|
|
**Alert:** `VPNTunnelDown` (critical, 5m)
|
|
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.networking`
|
|
**Expression:**
|
|
```
|
|
absent(node_network_receive_bytes_total{device="tun0"})
|
|
and on() (
|
|
(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
|
|
or (count(count_over_time(node_network_receive_bytes_total{device="tun0"}[1h])) > 0)
|
|
)
|
|
```
|
|
|
|
## What this means here
|
|
|
|
There is **no in-cluster egress-gateway pod** on this platform (ADR-0015). The tunnel is a **host
|
|
systemd service** — `vpn-egress-tunnel`, OpenVPN to PIA — running on one node, and steering is Cilium
|
|
eBPF via a `CiliumEgressGatewayPolicy` (`vpn-egress-steering`). OpenVPN creates and destroys `tun0` with
|
|
the connection, so the `tun0` byte-counter series exists **if and only if** the tunnel is up. The rule
|
|
keys on that presence rather than `node_network_up`, because `tun0` reports `operstate=unknown` and
|
|
`node_network_up` is therefore 0 even when the tunnel is fine.
|
|
|
|
Live state, measured 2026-09-13: **the tunnel is up** (`node_network_receive_bytes_total{device="tun0"}`
|
|
= 1 series) and the gateway node is labelled (`kube_node_labels{...egress_gateway="true"}` = 1 series),
|
|
so this alert is armed and quiet. The gateway is **`k3s-prod-3`** (192.168.2.12, `eno1`).
|
|
|
|
**The two facts that decide your response:**
|
|
|
|
1. **Cilium OSS has no egress-gateway failover.** Gateway-node loss is an outage, not a degradation.
|
|
The ADR-0016 supersede drafted on 2026-09-06 is still open. Worse, Cilium OSS picks *"the first node
|
|
in lexical ordering"* among nodeSelector matches — so labelling `k3s-prod-1` would **silently steal**
|
|
the gateway role and the public egress IP from `k3s-prod-3`. `[vpn_egress_gateway_prod]` is a
|
|
separate inventory group holding only `k3s-prod-3` for this reason; run the role as
|
|
`-e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3`, with `--list-hosts` first,
|
|
every time.
|
|
2. **The `absent()` was originally evaluated cluster-wide and became a permanent false alarm.** On a
|
|
cluster with no gateway node it could never clear, and from 2026-09-05T21:23Z it fired continuously
|
|
into `email-critical` — unbounded noise on the one channel that must stay trustworthy. The two
|
|
guards OR'd above are the fix: the alert arms itself the moment a node is labelled, and stays armed
|
|
through a kube-state-metrics outage. Before that, the very first version keyed on
|
|
`absent(gluetun_vpn_up)`, a metric gluetun's control API never exported at all — a pre-existing
|
|
perma-false-alarm. **If this alert is firing, verify the tunnel by hand before believing it.**
|
|
## Running the commands in this runbook
|
|
|
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
|
that hang is not the cluster being down. Every command below runs as:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
|
|
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
|
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
|
|
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
|
|
the same thing next month as it does today; one that relies on the default does not, and the failure is
|
|
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
|
|
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
|
|
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
|
|
interfere.
|
|
|
|
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
|
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
|
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
|
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
|
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
|
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
|
|
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
|
and it says so.
|
|
|
|
## First commands
|
|
|
|
Confirm on the host, not from Prometheus:
|
|
|
|
```
|
|
ssh k3s-prod-3 'systemctl status vpn-egress-tunnel --no-pager | head -20; echo; ip -br addr show tun0; echo; ip link show tun0 | head -2'
|
|
```
|
|
|
|
Which node is the gateway, and is it labelled?
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get nodes -L homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink; echo; \
|
|
kubectl --context k3s-prod get ciliumegressgatewaypolicies.cilium.io -o yaml | grep -A6 "nodeSelector\|egressIP\|interface"'
|
|
```
|
|
|
|
**The check that catches the silent failure** (see cause 1 below) — does Cilium's datapath point at the
|
|
interface that actually exists?
|
|
|
|
```
|
|
ssh k3s-prod-3 'ip -o link | awk -F: "{print \$1, \$2}" | grep tun0'
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg bpf egress list'
|
|
```
|
|
|
|
If the `Egress Ifindex` in that map is not the live `tun0` ifindex, egress is broken **even though this
|
|
alert is quiet**.
|
|
|
|
Is traffic actually leaving through the tunnel?
|
|
|
|
```
|
|
ssh k3s-prod-3 'curl -s --interface tun0 --max-time 10 https://ipinfo.io/ip; echo; nft list table inet vpn_killswitch 2>/dev/null | grep -E "counter|killswitch_" | head'
|
|
```
|
|
|
|
## Likely causes, most likely first
|
|
|
|
### 1. The tunnel is UP but Cilium is steering at a dead interface (the silent one)
|
|
This is the richest failure on the platform and it does **not** set off this alert.
|
|
**Cilium resolves a `CiliumEgressGatewayPolicy`'s `interface: tun0` exactly once**, caches
|
|
(egress IP, ifindex), and never re-resolves. On `k3s-dev-3`, `cilium-dbg bpf egress list` showed
|
|
`Egress Ifindex 3` — **there is no ifindex 3 on that node**; the live `tun0` was **1780**. Egress was
|
|
dead for about 80 minutes with nothing alerting and the conformance ledger still reading green.
|
|
`egress-gateway-reconciliation-trigger-interval: 1s` does **not** cover it (netlink link events are not
|
|
Kubernetes object events).
|
|
|
|
Measured behaviour, so you can pick a lever that works:
|
|
| action | result |
|
|
|---|---|
|
|
| annotate the CEGP | map → 1780 in 8s, PIA IP returns |
|
|
| `systemctl restart vpn-egress-tunnel` | tun0 1780→1985, **map stays 1780**, still broken |
|
|
| the exact `vpn-killswitch` stop/30s/start sequence | still broken 20s+ later |
|
|
| set **any** unrelated label on the gateway **node** | recovers within 10s |
|
|
|
|
The durable fix is in `roles/vpn_egress_gateway`: the OpenVPN up-script publishes a node label
|
|
`homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on every connect (via
|
|
`systemctl start --no-block`, so it can never block the tunnel), plus a 2-minute convergence timer for
|
|
boot. A node label, not a CEGP annotation — a node writing to GitOps state is a drift fight.
|
|
**Nothing monitors this coupling yet**; that gap is exactly why dev sat dead for 80 minutes.
|
|
|
|
### 2. PIA handed two gateways the same tunnel address
|
|
Standing rule: **one PIA server per gateway.** The vaulted `.ovpn` pins one server, so when prod came up
|
|
green, dev went down: both gateways dialled `158.173.241.112:1198` and PIA gave both the tunnel address
|
|
`10.72.192.7`. Each reconnect steals it — a measured ~2-minute mutual-eviction cycle (dev restarting at
|
|
22:57:05, 22:59:03, 23:01:05, 23:03:05, 23:05:06 UTC). Fix: the role var
|
|
`vpn_egress_gateway_pia_remote` ("`<host> <port>`") rewrites only the `.ovpn` `remote` line; prod is
|
|
pinned to **`158.173.241.135`** (pool `158.173.241.{34,55,112,135,239}`). *"A single green probe is not
|
|
a stability check."*
|
|
|
|
### 3. PIA "unreachable" — suspect a stale server list first
|
|
The 2026-08-22/23 chain went port collision → UDP/53 → 8080 → "MTU" (Cilium VXLAN pod MTU 1450 is real
|
|
but irrelevant to a **14-byte** handshake) → "the network blackholes PIA" (**wrong**) before the actual
|
|
cause: **gluetun's stale bundled server list**, dialling retired `46.246.x`/`212.112.x` addresses when
|
|
live SE Stockholm is `158.173.241.x`. Lesson recorded verbatim: when a client "can't reach" the
|
|
provider, suspect the stale server list **first**.
|
|
|
|
### 4. The gateway node is down
|
|
`absent()` also fires when the gateway node or its node-exporter is gone, so this alert covers
|
|
gateway-node-down by design. Go to `/runbooks/node-down`. Remember: **there is no failover** — while
|
|
`k3s-prod-3` is down, VPN-classed egress is an outage.
|
|
|
|
### 5. Nothing is wrong and the alert is lying
|
|
Given the two false-alarm post-mortems above, always confirm `ip -br addr show tun0` on the host before
|
|
acting.
|
|
|
|
## What the kill-switch does and does not do
|
|
|
|
Proven with counters on `k3s-prod-3`, because the intuitive reading is wrong: with
|
|
**`bpf.masquerade: true`** (which Egress Gateway requires) SNAT and egress happen in eBPF and
|
|
`bpf_redirect()` **skips netfilter POSTROUTING**. Measured: `forward` saddr=pod-CIDR **0**,
|
|
`postrouting` oif WAN saddr=pod-CIDR **0**. **What actually fails closed is Cilium's own drop** —
|
|
tcpdump during tunnel stop and restart showed **0 packets leaving either device**. The nft table is
|
|
kept as defence-in-depth for node-originated tunnel-sourced traffic and was proven both ways:
|
|
`ping -I 10.72.192.7` with the ip-rule → 3/3 at 144 ms with `killswitch_accept` advancing; rule removed
|
|
→ `killswitch_drop` 0→3 packets / 252 bytes, 100% loss. Every rule carries a `counter`, so:
|
|
**if `killswitch_drop` ever counts in normal operation, the datapath has changed.** A reported "leak
|
|
window" was **measured as no WAN leak** — a stale egress IP follows the cached ifindex, not a routing
|
|
lookup. Note the `.nft` file uses `table` + `flush table`, which clears rules but leaves named-set
|
|
elements intact.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **Escalate immediately if the gateway node is down.** No failover exists; this is an outage with no
|
|
in-cluster remedy.
|
|
- **Never label a second node `egress-gateway=true`.** Lexical ordering means `k3s-prod-1` would
|
|
silently take the role and the public egress IP.
|
|
- **Do not restart the tunnel expecting Cilium to follow** — measured above, it does not. If the map is
|
|
stale, touching a node label is the lever that works (operator action).
|
|
- **Do not remove or "simplify" the nft kill-switch** because the counters read zero. Zero on the pod
|
|
path is the expected, measured result of eBPF masquerade; the table guards a different path.
|
|
- A leak concern is a security matter (docs/07) — escalate rather than experimenting.
|
|
|
|
---
|
|
Source of truth: `runbooks/vpn-down.md`. Index: `/runbooks/`
|
|
Records: `DECISIONS-LOG.md` "2026-09-12 — VPN egress on PROD (k3s-prod-3), and three things the repo
|
|
believed that are not true"; the 2026-08-22/23 gluetun chain; commit `155df8e` (R-NET-06 re-proven on
|
|
prod, recovery ≤3s).
|