merge: feat/egress-gateway-staleness-alert — finished so it can actually pass the gate
Brings the two alerts for the silent egress-gateway failure: VPNEgressSteeringStale
(critical) and VPNEgressUplinkLabelMissing (warning), plus the kube-state-metrics
allowlist entry that makes them non-inert.
WHY THE BRANCH SAT IN THE BACKLOG: it could not pass the merge gate. Three
whole-repo checks failed on it, and each one was a real gap rather than a nuisance:
* alert-runbooks-resolvable — both alerts link to /runbooks/vpn-egress-stale,
which did not exist. An alert firing now would hand the operator a 404.
* alert-rules-unit-tested — alerts.yaml is already a covered file, so new rules
in it must be tested or a "tested" file quietly grows untested rules.
* alert-metrics-have-scrape-targets — see the guard bug below.
WHAT THE ALERTS CATCH. egress-class=vpn pods leave through one gateway node's tun0;
Cilium resolves it to an ifindex and caches it. An OpenVPN reconnect destroys and
re-creates tun0 with a NEW ifindex, and the node label
homelab.keithjackson.dev/egress-uplink=<addr>-<ifindex> is what forces Cilium to
re-resolve. If the label does not follow the tunnel, the cache points at a destroyed
interface and every vpn-class pod is silently dropped — while tun0 stays up and
VPNTunnelDown says nothing. That ran ~80 minutes undetected on the dev gateway.
Verified this is live on prod, not inert: k3s-prod-3 carries egress-gateway=true and
egress-uplink=10.150.192.86-139, and vpn-egress-steering has been in place 14 days.
The label was exported on 0 series before this change, so the allowlist entry is
what switches the alerts on.
runbooks/vpn-egress-stale.md — written and wired into the configMapGenerator, with
the four required operator sections. It leads with proving egress from a pod the
policy selects, because the symptom is "outbound traffic silently disappears", and
it says plainly not to "fix" this by deleting the policy: that does not restore
egress, it sends vpn-class traffic out of the real ISP connection, which is the leak
the class exists to prevent.
platform/monitoring/tests/vpn-egress.test.yaml — four promtool fixtures. 1 and 2 are
the SAME tunnel reconnect with and without the label following it, which isolates the
mechanism the rule actually keys on: a label VALUE change is not a changing sample,
it retires one series and starts another. 3 is the inert case (gateway node, no uplink
label) where the critical rule CANNOT fire, which is why the warning exists.
Two fixes the branch needed that were defects in the checks themselves:
1. alert-metrics-have-scrape-targets asserted the KSM allowlist matched
`nodes=[...egress-gateway]` with a literal closing bracket — i.e. that the list
contained EXACTLY that one label. Adding a second label correctly (egress-uplink,
which these alerts need) therefore FAILED the check while the property it guards
was perfectly satisfied. Loosened to `[,\]]`: in the list, not the whole list. A
guard that breaks when you correctly extend the thing it guards teaches people to
edit the guard, which is how it stops guarding.
2. UNTESTED_ALERTS listed VPNTunnelDown, and the check flagged it as stale because
"asserted on" is a bare word match over the test file — my header COMMENT
mentioning VPNTunnelDown was enough to count as coverage. Rather than delete the
register entry and claim coverage that did not exist, the fixtures now assert
VPNTunnelDown QUIET in all four cases. That required making them faithful: they
had no node_network_receive_bytes_total{device="tun0"} at all, so VPNTunnelDown
would have been firing throughout unnoticed, because promtool only checks the
alertnames you name. A healthy tun0 passes traffic, and that silence IS the claim
these rules exist for.
NOTE FOR LATER, not fixed here: that word-match means any alert named in prose counts
as tested. Tightening it to `alertname:` positions could reveal other prose-only
"coverage", and the register is shrink-only, so the cascade wants doing deliberately
rather than at 01:00 unattended.
Conflicts resolved: kept HEAD's Velero custom-resource-state block and took only the
branch's one-line allowlist change (14 lines down to 1); inserted the branch's
2026-09-12 DECISIONS-LOG entry in date order rather than appending it out of sequence.
Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips,
0 unwired. The new unit tests pass under the pinned promtool 3.14.0.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
commit
a3f4ddb971
14 files changed
+864
-5
No files matched your search
@@ -2424,6 +2424,148 @@ container dies with `PermissionError` on every run; my older base had `0400` and
|
||||
- The apex flip itself is `migration/dns-apex-inversion`'s job. This change only guarantees the realm
|
||||
follows git once the knob moves; it does not move it.
|
||||
|
||||
## 2026-09-12 — The silent-gateway outage now has an alert, and the direct signal turned out not to exist (branch `feat/egress-gateway-staleness-alert`)
|
||||
|
||||
Hand-off item 4 from the section above: *"A silent gateway outage still has no alert… nothing
|
||||
watches the invariant `egress-uplink` label == `cilium-dbg bpf egress list`'s Egress Ifindex."*
|
||||
Closed, as far as Prometheus can close it — and the part that cannot be closed is recorded here
|
||||
rather than papered over with a plausible-looking metric name.
|
||||
|
||||
### The direct signal is not available, verified rather than assumed
|
||||
|
||||
The invariant wants the BPF egress map's contents. They are not in Prometheus, for two independent
|
||||
reasons, both checked read-only against live prod:
|
||||
|
||||
| check | result |
|
||||
|---|---|
|
||||
| `{__name__=~"cilium.*egress.*"}`, `cilium_drop.*`, `hubble_drop.*` | **empty vector** — no `cilium_*`/`hubble_*` series exist at all |
|
||||
| container ports on the cilium DaemonSet | `health=9879`, `peer-service=4244` — **no metrics port**; only the operator has `operator-prometheus-serve-addr :9963` |
|
||||
| `kubectl get servicemonitors -A \| grep -iE 'cilium\|hubble'` | none — Hubble/Cilium metrics export is still the open TODO in `platform/cilium/helmrelease.yaml` |
|
||||
|
||||
So `HubbleDeniedFlowSpike` cannot fire today either; that is a separate, pre-existing gap, noted
|
||||
and left alone. And even with agent metrics turned on, **Cilium OSS publishes no series carrying
|
||||
egress-map contents** — the map is readable only through `cilium-dbg bpf egress list`, a CLI over
|
||||
the BPF map. Nothing was invented to fill the hole.
|
||||
|
||||
What node-exporter *does* export is the live tun0 **ifindex as a metric value**:
|
||||
`node_network_iface_id{device="tun0"}` = `129` on `k3s-prod-3`. Read-only on the live cluster, all
|
||||
three agree right now:
|
||||
|
||||
```
|
||||
node label homelab.keithjackson.dev/egress-uplink = 10.150.192.86-129
|
||||
node-exporter node_network_iface_id{device="tun0"} = 129
|
||||
bpf map Egress IP 10.150.192.86 / Egress Ifindex 129 (all 16 rows, 4 vpn-class pods)
|
||||
```
|
||||
|
||||
(The map on the two NON-gateway agents reads `Egress IP 0.0.0.0 / Ifindex 0` with
|
||||
`Gateway IP 192.168.2.12` — only the gateway node resolves the interface. Worth knowing before
|
||||
debugging the wrong agent.)
|
||||
|
||||
### PromQL cannot express the equality test — so the invariant is counted instead
|
||||
|
||||
The label's value is `<addr>-<ifindex>`; the live ifindex is a metric VALUE. PromQL has no
|
||||
label-to-number conversion, so `label == value` is **not expressible**, and there is no
|
||||
level-triggered form of this comparison. What is expressible is the same invariant counted over a
|
||||
window: **the published label must change at least as often as tun0 is re-created.**
|
||||
|
||||
```promql
|
||||
(
|
||||
changes(max(node_network_iface_id{device="tun0"})[1h:1m])
|
||||
>
|
||||
(count(count_over_time(max by (label_homelab_keithjackson_dev_egress_uplink) (
|
||||
kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""})[1h:1m])) - 1)
|
||||
)
|
||||
and on()
|
||||
(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
|
||||
```
|
||||
|
||||
`count(count_over_time(max by (<label>) (…)[1h:1m]))` is the number of DISTINCT label values in the
|
||||
window. The `max by (<label>)` wrapper is load-bearing: it aggregates away `pod`/`instance` so a
|
||||
kube-state-metrics restart is not read as a label change. Both sides reduce to a single unlabelled
|
||||
series, which is why `and on()` joins — the same reason `VPNTunnelDown` needs `on()`.
|
||||
|
||||
### Proof it evaluates — executed against prod's Prometheus, port-forward + `/api/v1/query`
|
||||
|
||||
There is no promtool in this repo, so query execution IS the verification. The **rendered** expr was
|
||||
extracted from `kubectl kustomize platform-config` output and run verbatim:
|
||||
|
||||
| state | query | result |
|
||||
|---|---|---|
|
||||
| healthy, now | rendered `VPNEgressSteeringStale` | `[]` — not firing |
|
||||
| **no gateway node** (`@ 1789245000`, before prod's stand-up: no tun0 series, no labels) | same expr | `[]` — cannot fire |
|
||||
| **stale**, constructed from real data | same expr with the already-exported `egress-gateway` label substituted for `egress-uplink`, window 3h | **`1` — FIRES** |
|
||||
| same expr, window 1h | | `[]` |
|
||||
|
||||
The stale case is honest and cost prod nothing: prod's tun0 really did go **128 → 129** ~2.5h
|
||||
earlier when it moved to PIA `.135`, and the `egress-gateway` label really did *not* follow it
|
||||
(`changes(...[3h:1m])` = 1, distinct label values = 1 → `1 > 0`). That is bit-for-bit the shape of
|
||||
a label that missed a flap. The healthy counterfactual is the one thing left as reasoning rather
|
||||
than a query: over that same 3h the REAL `egress-uplink` label held two values
|
||||
(`10.72.192.7-128`, then `10.150.192.86-129`), giving `1 > 1` = false. It could not be queried
|
||||
because the label was not exported yet — which is the next point.
|
||||
|
||||
Supporting queries, also live: the distinct-value idiom returns **1** for a label that held still
|
||||
over the window and **2** for one whose value changed inside it (node 3's `egress-gateway` label
|
||||
over 12h, which went `""` → `"true"` at stand-up); the gateway guard returns 1; subquery +
|
||||
`changes()` returns 0/1/1/1 over 15m/3h/6h/12h as the flap history requires.
|
||||
|
||||
### Behaviour at the edges
|
||||
|
||||
- **No gateway node:** three independent reasons it cannot fire — no tun0 series, no uplink label,
|
||||
and the `kube_node_labels` guard `VPNTunnelDown` already established. Verified at a past
|
||||
timestamp when prod genuinely had no gateway. This matters: `VPNTunnelDown` fired permanently on
|
||||
prod for exactly this reason and had to be retrofitted.
|
||||
- **Convergence:** the up-script rewrites the label within seconds of a connect, worst case ~5m
|
||||
(its 30 × 10s kube-API retry budget on a cold boot, when the API is not up yet). The condition is
|
||||
true during that gap, so `for: 15m` — comfortably past the 2-minute timer and the 5-minute retry
|
||||
budget — is what keeps a normal reconnect quiet.
|
||||
- **Bounded, deliberately:** a missed flap alerts from +15m to +1h, then RESOLVES even if egress is
|
||||
still dead. An unbounded critical on `email-critical` is the worse failure for this repo, and 15
|
||||
minutes still beats 80. A level-triggered version needs the map's ifindex as a real metric — the
|
||||
node-exporter textfile collector route, still open.
|
||||
|
||||
### The second rule exists because the first one fails OPEN
|
||||
|
||||
`VPNEgressSteeringStale` needs `egress-uplink` in kube-state-metrics. Without it the comparison
|
||||
degrades to an empty vector and alerts nothing — so `VPNEgressUplinkLabelMissing` (warning,
|
||||
`for: 30m`) watches the precondition: a node labelled `egress-gateway=true` with no `egress-uplink`
|
||||
label visible. Live right now it returns **1**, correctly: the KSM allowlist extension is in this
|
||||
same branch and not yet deployed. It self-clears on the first reconcile, and `for: 30m` covers the
|
||||
kube-state-metrics roll. The rule and the allowlist are in the same Flux kustomization on purpose.
|
||||
|
||||
`kube-state-metrics.metricLabelsAllowlist` now reads
|
||||
`nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]` — a label
|
||||
DIMENSION on the same 3 `kube_node_labels` series, not new series, on exactly one node.
|
||||
|
||||
### Rejected, with reasons
|
||||
|
||||
- **A black-box egress probe** is the most honest signal — it is the only one that answers "does a
|
||||
vpn-class pod's traffic actually leave via the tunnel?" and the only one that catches "the label
|
||||
was written and Cilium ignored it anyway". Rejected **for now**: there is no blackbox-exporter and
|
||||
no probe CronJob in this repo, so it means a new always-on component (itself vpn-class, itself an
|
||||
alerting surface, itself dependent on a third-party endpoint) — and it could not have been
|
||||
verified on prod today. It remains the right next step.
|
||||
- **A node-exporter textfile metric from the re-steer script** (hand-off item 4's own suggestion) is
|
||||
the only route to a true level-triggered label-vs-MAP comparison. It needs a textfile directory
|
||||
mounted into the node-exporter DaemonSet plus an Ansible change applied on a prod node — neither
|
||||
of which this branch could execute or verify. Left open, now with the reason recorded.
|
||||
- **`absent(node_network_receive_bytes_total{device="tun0"})`-style presence checks** — that IS
|
||||
`VPNTunnelDown`, and it is exactly the blind spot: green while steering points at a dead ifindex.
|
||||
|
||||
### Left for the operator / not settled here
|
||||
|
||||
1. **The rules are unproven in the firing sense.** The expressions evaluate against live prod data
|
||||
and the stale branch was exercised with a real flap, but no alert has actually reached
|
||||
Alertmanager or the inbox. `conformance/tests/l3-fault-injection/vpn-killswitch` stops and
|
||||
restarts the tunnel and is the natural place to assert `VPNEgressSteeringStale` stays quiet
|
||||
through a healthy reconnect; the alerting L3 test with the SMTP sink is where a fired-and-emailed
|
||||
proof would come from. Neither was fired at prod unasked.
|
||||
2. **`HubbleDeniedFlowSpike` cannot fire** — no cilium/hubble metrics are scraped on either cluster.
|
||||
Pre-existing, orthogonal, and the same TODO in `platform/cilium/helmrelease.yaml` blocks it.
|
||||
3. **Window vs. stickiness** is a judgement call, not a measurement: 1h chosen so a missed flap
|
||||
alerts for ~45 minutes. If the operator would rather have a sticky alert that needs manual
|
||||
acknowledgement, that is the textfile-collector design, not a tweak to this one.
|
||||
|
||||
## 2026-09-13 — **PROD IS THE APEX.** Dev decommissioned; and a GitOps deadlock found the hard way
|
||||
|
||||
Prod serves `home.keithjackson.dev`. Dev's three VMs (Proxmox 101/102/105) are stopped with
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
[1m[run.sh][0m Coverage report first (always)
|
||||
[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
|
||||
[coverage] wrote /home/keith/source/home-network-k3s-impl/conformance/_artifacts/coverage/coverage.md and /home/keith/source/home-network-k3s-impl/conformance/_artifacts/coverage/coverage.json
|
||||
[1m[run.sh][0m No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
|
||||
ERROR: cluster reachable but 'chainsaw' (v0.2.12) not on PATH.
|
||||
Install the pinned binary or run via the in-cluster Job (incluster/).
|
||||
@@ -0,0 +1,4 @@
|
||||
<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/node-loss" tests="1" failures="0" errors="0" id="0" time="">
|
||||
<testcase name="node-loss" classname="" time="115.855688" />
|
||||
</testsuite>
|
||||
</testsuites>
|
||||
@@ -0,0 +1,232 @@
|
||||
[1m[run.sh][0m Coverage report first (always)
|
||||
[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
|
||||
[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
|
||||
[1m[run.sh][0m No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
|
||||
[1m[run.sh][0m LIGHT pass: no tests selected.
|
||||
[1m[run.sh][0m HEAVY pass: 1 test(s) SERIAL (parallel=1) — node-loss
|
||||
Version: 0.2.12
|
||||
Loading config (/work/conformance/config/chainsaw-config.yaml)...
|
||||
- Using test file: chainsaw-test
|
||||
- TestDirs [/work/conformance/tests/l3-fault-injection/node-loss]
|
||||
- SkipDelete false
|
||||
- FailFast false
|
||||
- ReportFormat 'JUNIT-TEST'
|
||||
- ReportName 'chainsaw-report-heavy'
|
||||
- ReportPath '/work/conformance/_artifacts'
|
||||
- Namespace ''
|
||||
- FullName false
|
||||
- IncludeTestRegex ''
|
||||
- ExcludeTestRegex ''
|
||||
- ApplyTimeout 1m30s
|
||||
- AssertTimeout 3m0s
|
||||
- CleanupTimeout 2m0s
|
||||
- DeleteTimeout 2m0s
|
||||
- ErrorTimeout 1m0s
|
||||
- ExecTimeout 3m0s
|
||||
- DeletionPropagationPolicy Background
|
||||
- Parallel 1
|
||||
- Template true
|
||||
- NoCluster false
|
||||
- PauseOnFailure false
|
||||
Loading tests...
|
||||
- node-loss (/work/conformance/tests/l3-fault-injection/node-loss)
|
||||
Loading values...
|
||||
Running tests...
|
||||
=== RUN chainsaw
|
||||
=== PAUSE chainsaw
|
||||
=== CONT chainsaw
|
||||
=== RUN chainsaw/node-loss
|
||||
=== PAUSE chainsaw/node-loss
|
||||
=== CONT chainsaw/node-loss
|
||||
l.go:52: | 05:22:26 | [32;1mnode-loss[0;22m | [32;1m@chainsaw [0;22m | [32;1mCREATE[0;22m | [32;1mOK[0;22m | v1/Namespace @ chainsaw-advanced-boxer
|
||||
l.go:52: | 05:22:26 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mTRY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 05:22:26 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -euo pipefail
|
||||
NS="$NAMESPACE"
|
||||
|
||||
# -- Gate 1: skip cleanly (green) if the longhorn tier is not present yet. --
|
||||
if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
|
||||
echo "SKIP: 'longhorn' StorageClass absent — node-loss not applicable on this cluster yet"
|
||||
echo NODE_LOSS_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# -- Gate 2: a 3-replica volume + single-node-loss headroom needs >= 3 Ready nodes. --
|
||||
READY="$(kubectl get nodes \
|
||||
-o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
|
||||
| grep -c '^True$' || true)"
|
||||
echo "ready nodes: ${READY}"
|
||||
if [ "${READY:-0}" -lt 3 ]; then
|
||||
echo "SKIP: only ${READY:-0} Ready node(s) (<3) — node-loss resilience not meaningful yet"
|
||||
echo NODE_LOSS_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
|
||||
CANARY="node-loss-canary-$(date +%s%N)"
|
||||
|
||||
# -- GIVEN: a Longhorn PVC (platform default SC = 3 replicas across nodes) + a
|
||||
# writer pod that writes the canary and flushes it to the replicated volume. --
|
||||
kubectl -n "$NS" apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: survivor
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: longhorn
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
EOF
|
||||
|
||||
apply_pod() {
|
||||
kubectl -n "$NS" apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: $1
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: app
|
||||
image: busybox:1.36.1
|
||||
command: ["sh", "-c", "sleep 3600"]
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: survivor
|
||||
EOF
|
||||
}
|
||||
|
||||
apply_pod writer
|
||||
kubectl -n "$NS" wait --for=condition=Ready pod/writer --timeout=240s
|
||||
printf '%s' "$CANARY" | kubectl -n "$NS" exec -i writer -- sh -c 'cat > /data/canary && sync'
|
||||
echo "wrote canary: $(kubectl -n "$NS" exec writer -- cat /data/canary)"
|
||||
|
||||
# The victim node is the one the attached volume currently lives on.
|
||||
VICTIM="$(kubectl -n "$NS" get pod writer -o jsonpath='{.spec.nodeName}')"
|
||||
[ -n "$VICTIM" ] || { echo "FAIL: could not determine writer's node" >&2; exit 1; }
|
||||
echo "victim node (currently hosts the attached volume): $VICTIM"
|
||||
|
||||
# Best-effort precondition log: if the Longhorn Volume CR is readable, show the
|
||||
# volume is genuinely 3-replica/healthy before we take a node. Non-fatal (the
|
||||
# runner is least-privilege and may not have longhorn.io read rights) — the
|
||||
# data-availability assertion below is the authoritative proof, not this.
|
||||
PVNAME="$(kubectl -n "$NS" get pvc survivor -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
|
||||
if [ -n "$PVNAME" ]; then
|
||||
kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
|
||||
-o jsonpath='pre-fault robustness={.status.robustness} replicas={.spec.numberOfReplicas}{"\n"}' \
|
||||
2>/dev/null || echo "(Longhorn Volume CR not readable — continuing; data read is the proof)"
|
||||
fi
|
||||
|
||||
# -- WHEN: FAULT INJECTION — remove the victim node FROM SERVICE. --
|
||||
# Label it with THIS test's unique namespace so `finally` can always find and
|
||||
# uncordon exactly this node; then cordon it; then evict ONLY our own writer
|
||||
# pod, forcing the volume to detach from the victim. (No `drain` — unrelated
|
||||
# pods on the node are never touched.)
|
||||
kubectl label node "$VICTIM" "conformance.home.k3s/cordoned-by=$NS" --overwrite
|
||||
kubectl cordon "$VICTIM"
|
||||
echo "cordoned + labelled victim node: $VICTIM"
|
||||
kubectl -n "$NS" delete pod writer --wait=true --timeout=240s
|
||||
echo "evicted writer off the victim node; volume detached from $VICTIM"
|
||||
|
||||
# -- THEN: a fresh workload re-binds the SAME volume. The victim is unschedulable,
|
||||
# so the scheduler MUST place the reader on a SURVIVING node; Longhorn re-attaches
|
||||
# the volume there from its replicas on the other nodes. --
|
||||
apply_pod reader
|
||||
kubectl -n "$NS" wait --for=condition=Ready pod/reader --timeout=360s
|
||||
READER_NODE="$(kubectl -n "$NS" get pod reader -o jsonpath='{.spec.nodeName}')"
|
||||
echo "reader scheduled on surviving node: $READER_NODE (victim was $VICTIM)"
|
||||
if [ "$READER_NODE" = "$VICTIM" ]; then
|
||||
echo "FAIL: reader landed back on the cordoned victim node — node was not removed from service" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
GOT="$(kubectl -n "$NS" exec reader -- cat /data/canary)"
|
||||
echo "recovered='$GOT' expected='$CANARY'"
|
||||
[ "$GOT" = "$CANARY" ] || { echo "FAIL: data not available after node loss" >&2; exit 1; }
|
||||
echo "volume available + data intact from a surviving node"
|
||||
|
||||
# Best-effort reconvergence log: if the Longhorn Volume CR is readable, wait
|
||||
# (bounded) for robustness to settle back to available (healthy|degraded — both
|
||||
# serve I/O; a healthy 3-replica volume rebuilds a replica once headroom exists).
|
||||
# Non-fatal for the same least-privilege reason as the pre-fault log.
|
||||
if [ -n "$PVNAME" ]; then
|
||||
for i in $(seq 1 30); do
|
||||
ROB="$(kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
|
||||
-o jsonpath='{.status.robustness}' 2>/dev/null || true)"
|
||||
[ -z "$ROB" ] && { echo "(Longhorn Volume CR not readable — skipping reconvergence log)"; break; }
|
||||
echo "post-fault robustness[$i]=$ROB"
|
||||
case "$ROB" in healthy|degraded) break ;; esac
|
||||
sleep 10
|
||||
done
|
||||
fi
|
||||
|
||||
echo NODE_LOSS_VOLUME_AVAILABLE_OK
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
ready nodes: 3
|
||||
persistentvolumeclaim/survivor created
|
||||
pod/writer created
|
||||
pod/writer condition met
|
||||
wrote canary: node-loss-canary-1789795346371286043
|
||||
victim node (currently hosts the attached volume): k3s-prod-2
|
||||
pre-fault robustness=healthy replicas=3
|
||||
node/k3s-prod-2 labeled
|
||||
node/k3s-prod-2 cordoned
|
||||
cordoned + labelled victim node: k3s-prod-2
|
||||
pod "writer" deleted from chainsaw-advanced-boxer namespace
|
||||
evicted writer off the victim node; volume detached from k3s-prod-2
|
||||
pod/reader created
|
||||
pod/reader condition met
|
||||
reader scheduled on surviving node: k3s-prod-3 (victim was k3s-prod-2)
|
||||
recovered='node-loss-canary-1789795346371286043' expected='node-loss-canary-1789795346371286043'
|
||||
volume available + data intact from a surviving node
|
||||
post-fault robustness[1]=healthy
|
||||
NODE_LOSS_VOLUME_AVAILABLE_OK
|
||||
l.go:52: | 05:23:37 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mTRY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mFINALLY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -uo pipefail
|
||||
for n in $(kubectl get nodes -l "conformance.home.k3s/cordoned-by=$NAMESPACE" -o name 2>/dev/null); do
|
||||
kubectl uncordon "${n#node/}" || true
|
||||
kubectl label "$n" "conformance.home.k3s/cordoned-by-" || true
|
||||
echo "restored node to service: ${n#node/}"
|
||||
done
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
node/k3s-prod-2 uncordoned
|
||||
node/k3s-prod-2 unlabeled
|
||||
restored node to service: k3s-prod-2
|
||||
l.go:52: | 05:23:37 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c kubectl -n "$NAMESPACE" delete pod writer reader --ignore-not-found --wait=false || true
|
||||
kubectl -n "$NAMESPACE" delete pvc survivor --ignore-not-found --wait=false || true
|
||||
echo "cleanup best-effort; chainsaw namespace teardown finalizes"
|
||||
l.go:52: | 05:23:38 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
pod "reader" deleted from chainsaw-advanced-boxer namespace
|
||||
persistentvolumeclaim "survivor" deleted from chainsaw-advanced-boxer namespace
|
||||
cleanup best-effort; chainsaw namespace teardown finalizes
|
||||
l.go:52: | 05:23:38 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 05:23:38 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mFINALLY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 05:23:38 | [36;1mnode-loss[0;22m | [36;1m@chainsaw [0;22m | [36;1mCLEANUP[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 05:23:38 | [32;1mnode-loss[0;22m | [32;1m@chainsaw [0;22m | [32;1mDELETE[0;22m | [32;1mOK[0;22m | v1/Namespace @ chainsaw-advanced-boxer
|
||||
l.go:52: | 05:24:21 | [36;1mnode-loss[0;22m | [36;1m@chainsaw [0;22m | [36;1mCLEANUP[0;22m | [36;1mEND[0;22m |
|
||||
--- PASS: chainsaw (0.00s)
|
||||
--- PASS: chainsaw/node-loss (115.86s)
|
||||
PASS
|
||||
Tests Summary...
|
||||
- Passed tests 1
|
||||
- Failed tests 0
|
||||
- Skipped tests 0
|
||||
Done.
|
||||
merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
|
||||
[1m[run.sh][0m Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md
|
||||
@@ -5,3 +5,4 @@
|
||||
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "9d0814ae61e6a0ba64d3789a71f1cd9c985efbf0", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NS_PRIVILEGED_OK, TUNNEL_HEALTHY_OK, NO_LEAK_OK, INTERNET_UNAFFECTED_OK, TUNNEL_SELF_HEAL_OK, TUNNEL_RECOVERED_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-NET-06"], "run_id": "20260913T014923Z-9d0814a", "sentinel": "TUNNEL_RECOVERED_OK", "test": "vpn-killswitch", "ts_utc": "2026-09-13T01:49:23Z"}
|
||||
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "24b8111d0c39e9417e9f4d782f333f683540cfa7", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK, DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-03"], "run_id": "20260914T021505Z-24b8111", "sentinel": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK", "test": "dr-restore", "ts_utc": "2026-09-14T02:15:05Z"}
|
||||
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "bd17b65db9fa906f336528e90b429ec1e2377963", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L0", "note": "L0 offline check exited 0; run_l1_offline.py renders its skips as JUnit <skipped/>, so a green JUnit is truthful without a sentinel", "proof": "proven", "requirements": ["R-ENV-02", "R-STATE-03"], "run_id": "20260914T021902Z-bd17b65", "sentinel": null, "test": "placeholder-guard-coupling", "ts_utc": "2026-09-14T02:19:02Z"}
|
||||
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "9339d5a79e2301d399e4e20722b164583c53c55d", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NODE_LOSS_VOLUME_AVAILABLE_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-02"], "run_id": "20260919T052225Z-9339d5a", "sentinel": "NODE_LOSS_VOLUME_AVAILABLE_OK", "test": "node-loss", "ts_utc": "2026-09-19T05:22:25Z"}
|
||||
@@ -256,7 +256,13 @@ SCRAPE_OWNERS: list[dict] = [
|
||||
"file": KPS,
|
||||
"require": [
|
||||
r"metricLabelsAllowlist:",
|
||||
r"nodes=\[homelab\.keithjackson\.dev/egress-gateway\]",
|
||||
# The label must be IN the allowlist, not BE the whole allowlist. This pattern used to
|
||||
# end in a literal `]`, asserting the list held exactly this one label — so adding a
|
||||
# second one (`egress-uplink`, for the steering-staleness alerts) made this check FAIL
|
||||
# while the property it guards stayed perfectly satisfied. A guard that breaks when you
|
||||
# correctly extend the thing it guards teaches people to edit the guard, which is how it
|
||||
# stops guarding. `[,\\]]` accepts either the end of the list or a further entry.
|
||||
r"nodes=\[homelab\.keithjackson\.dev/egress-gateway[,\]]",
|
||||
],
|
||||
"forbid": [],
|
||||
"why": "KSM exposes NO custom labels by default. Without the allowlist entry "
|
||||
|
||||
@@ -179,6 +179,10 @@ UNTESTED_FILES = {
|
||||
# and NodeIOPressureCritical quiet on the same measured state — the whole point of that fixture is
|
||||
# that the two PSI detectors and the eight cause rules must agree about what "healthy" is, so the
|
||||
# detectors are no longer unasserted.
|
||||
# It SHRANK BY ONE on 2026-09-19: platform/monitoring/tests/vpn-egress.test.yaml pins the two new
|
||||
# egress-steering rules, and asserts VPNTunnelDown QUIET in all four fixtures — that silence IS the
|
||||
# claim those rules exist for (tun0 up and passing traffic while vpn-class egress is dropped), so it
|
||||
# had to become an assertion rather than a sentence in a comment.
|
||||
UNTESTED_ALERTS = {
|
||||
("platform/cnpg/alerts.yaml", "CNPGWALArchiveBacklog"),
|
||||
("platform/cnpg/alerts.yaml", "CNPGNoRecoverabilityPoint"),
|
||||
@@ -195,7 +199,6 @@ UNTESTED_ALERTS = {
|
||||
("platform/monitoring/alerts.yaml", "CertificateExpiringSoon"),
|
||||
("platform/monitoring/alerts.yaml", "CertificateExpiryImminent"),
|
||||
("platform/monitoring/alerts.yaml", "CertificateNotReady"),
|
||||
("platform/monitoring/alerts.yaml", "VPNTunnelDown"),
|
||||
("platform/monitoring/alerts.yaml", "HubbleDeniedFlowSpike"),
|
||||
("platform/monitoring/alerts.yaml", "KeycloakDown"),
|
||||
("platform/monitoring/alerts.yaml", "KeycloakDatabaseDown"),
|
||||
|
||||
@@ -904,6 +904,132 @@ spec:
|
||||
summary: "Egress VPN tunnel down"
|
||||
description: "The egress-gateway VPN tunnel is down (or its metric is absent) for 5m — outbound traffic could leak or stall (leak risk, docs/07)."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-down"
|
||||
# -----------------------------------------------------------------------------------------
|
||||
# THE BLIND SPOT `VPNTunnelDown` LEAVES (added 2026-09-12) — R-OBS-04, R-NET-06
|
||||
#
|
||||
# `VPNTunnelDown` goes green the instant a tun0 exists. That is not the same as vpn-class pods
|
||||
# having egress, because Cilium resolves a CiliumEgressGatewayPolicy's `interface: tun0`
|
||||
# exactly ONCE and caches an (egress IP, egress ifindex) pair. Every OpenVPN reconnect
|
||||
# re-creates tun0 with a NEW ifindex, so after the first reconnect the BPF egress map points
|
||||
# at a DESTROYED interface: `bpf_redirect()` fails and the packets are dropped — 0 out tun0,
|
||||
# 0 out the WAN (measured; there is no leak, there is simply no egress). tun0 is present and
|
||||
# carrying bytes the whole time, so `VPNTunnelDown` stays green. The dev gateway sat in
|
||||
# exactly that state for ~80 minutes on 2026-09-12 with nothing alerting.
|
||||
#
|
||||
# The datapath now self-heals: `roles/vpn_egress_gateway` publishes
|
||||
# `homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on the gateway node from the
|
||||
# OpenVPN up-script on every connect (plus a 2-minute convergence timer for the boot case),
|
||||
# and that node write is what makes Cilium re-resolve. These two rules watch whether it
|
||||
# ACTUALLY converged — if the re-steer mechanism itself fails we are back to a silent outage.
|
||||
#
|
||||
# WHY THIS SIGNAL AND NOT THE MAP ITSELF. The direct invariant is "the gateway node's
|
||||
# `egress-uplink` label == the Egress Ifindex in Cilium's BPF egress map". The map side is NOT
|
||||
# observable in Prometheus, verified on prod 2026-09-12 rather than assumed:
|
||||
# * the cilium-agent DaemonSet exposes no metrics port at all (`health=9879`,
|
||||
# `peer-service=4244`; only the OPERATOR has `operator-prometheus-serve-addr :9963`), and
|
||||
# there is no ServiceMonitor for cilium or hubble on either cluster — Hubble/Cilium
|
||||
# metrics export is still the open TODO in platform/cilium/helmrelease.yaml. Every
|
||||
# `cilium_*` and `hubble_*` query returns an empty vector today (which also means
|
||||
# `HubbleDeniedFlowSpike` above cannot fire yet).
|
||||
# * and even with agent metrics enabled, Cilium OSS publishes NO series carrying egress-map
|
||||
# CONTENTS. The map is readable only via `cilium-dbg bpf egress list` — a CLI over the BPF
|
||||
# map, not an exporter. No metric name is invented here to paper over that.
|
||||
# So the observable is the label: the input to the re-steer, compared against the live tun0
|
||||
# identity that node-exporter does export (`node_network_iface_id{device="tun0"}` — the real
|
||||
# ifindex, 129 on k3s-prod-3 right now, matching both the label `10.150.192.86-129` and
|
||||
# `cilium-dbg bpf egress list`'s `Egress Ifindex 129`, checked by hand on the live cluster).
|
||||
#
|
||||
# WHAT THIS DOES AND DOES NOT CATCH. It catches the whole input half: the up-script not
|
||||
# firing, the label write failing or never being deployed, the convergence timer being dead —
|
||||
# i.e. every way the tun0 identity can move without Cilium being told. It does NOT catch
|
||||
# "the label was written and Cilium still did not re-resolve"; nothing in Prometheus can,
|
||||
# until either the re-steer script exports the map's ifindex through a node-exporter textfile
|
||||
# collector (DECISIONS-LOG 2026-09-12, hand-off item 4) or a vpn-class black-box probe exists.
|
||||
# A probe was considered and rejected for now: there is no blackbox-exporter and no probe
|
||||
# CronJob in this repo, so it means a NEW always-on component whose own failures become a new
|
||||
# alerting surface — worth doing, but not as the thing that closes a hole today.
|
||||
#
|
||||
# WHY A `changes()` COMPARISON RATHER THAN AN EQUALITY TEST. PromQL cannot compare a metric
|
||||
# VALUE (the live ifindex) against a number embedded in a LABEL VALUE (`...-129`): there is no
|
||||
# label-to-number conversion, so the level-triggered equality test is not expressible. What IS
|
||||
# expressible is the same invariant counted over a window: the published label must change at
|
||||
# least as often as tun0 is re-created. `count(count_over_time(max by (<label>) (...)[1h:1m]))`
|
||||
# is the number of DISTINCT label values seen in the last hour (verified live: 1 for a label
|
||||
# that held still, 2 for one that changed value inside the window); the `max by (<label>)`
|
||||
# wrapper is load-bearing — it aggregates away `pod`/`instance`, so a kube-state-metrics
|
||||
# restart is not mistaken for a label change. Both sides reduce to a single unlabelled series,
|
||||
# which is why `and on()` joins cleanly, exactly as `VPNTunnelDown` does.
|
||||
#
|
||||
# Consequence, stated plainly: this is bounded by its window. A missed flap is alerted from
|
||||
# +15m to +1h and then the alert RESOLVES even if egress is still dead. That is a deliberate
|
||||
# trade against the repo's alert-fatigue history (an unbounded critical on the one channel
|
||||
# that must stay trustworthy is worse), and it still turns an 80-minute silent outage into an
|
||||
# email within 15 minutes. The textfile-collector gauge is what would make it level-triggered.
|
||||
- alert: VPNEgressSteeringStale
|
||||
# FIRES when the live tun0 was re-created more times in the last hour than the gateway
|
||||
# node's `egress-uplink` label changed — i.e. Cilium was not told about at least one
|
||||
# tunnel reconnect, so the BPF egress map is pointing at an ifindex that no longer exists
|
||||
# and vpn-class pods have no egress. Convergence is NOT this: the label is rewritten within
|
||||
# seconds of a connect (worst case ~5m, the up-script's 30 x 10s kube-API retry budget on a
|
||||
# cold boot), so `for: 15m` cannot fire on a healthy reconnect.
|
||||
expr: |
|
||||
(
|
||||
changes(max(node_network_iface_id{device="tun0"})[1h:1m])
|
||||
>
|
||||
(
|
||||
count(
|
||||
count_over_time(
|
||||
max by (label_homelab_keithjackson_dev_egress_uplink) (
|
||||
kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""}
|
||||
)[1h:1m]
|
||||
)
|
||||
) - 1
|
||||
)
|
||||
)
|
||||
and on()
|
||||
(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
|
||||
for: 15m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "VPN egress steering is stale — vpn-class pods have no egress"
|
||||
description: >-
|
||||
The gateway node's tun0 was re-created without the `egress-uplink` label following it, so
|
||||
Cilium's cached egress ifindex points at a destroyed interface: egress-class=vpn pods are
|
||||
being dropped, silently, while tun0 still looks healthy. Compare
|
||||
`kubectl get nodes -L homelab.keithjackson.dev/egress-uplink` against
|
||||
`cilium-dbg bpf egress list` on the gateway node; re-applying the CiliumEgressGatewayPolicy
|
||||
(or setting any node label) forces a fresh resolution.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
|
||||
- alert: VPNEgressUplinkLabelMissing
|
||||
# The watchdog for the rule above: `VPNEgressSteeringStale` needs the `egress-uplink` label
|
||||
# in kube-state-metrics, and if that label is missing its comparison degrades to an empty
|
||||
# vector — it fails OPEN, alerting nothing. So alert on the precondition itself. A gateway
|
||||
# node with no published uplink means the re-steer mechanism is not in place at all
|
||||
# (roles/vpn_egress_gateway not applied on this node, the up-script never ran, or the
|
||||
# kube-state-metrics label allowlist in kube-prometheus-stack.yaml lost the entry), which is
|
||||
# the configuration that produced the 80-minute dev outage. Warning, not critical: it is a
|
||||
# rollout/config gap to fix in working hours, not a page. `for: 30m` so that the Helm upgrade
|
||||
# that rolls kube-state-metrics — both changes ship in this same kustomization — cannot page
|
||||
# on the way in, and so an Ansible run in progress does not either.
|
||||
expr: |
|
||||
(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
|
||||
and on()
|
||||
absent(kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""})
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Egress gateway node publishes no egress-uplink label"
|
||||
description: >-
|
||||
A node carries `homelab.keithjackson.dev/egress-gateway=true` but no
|
||||
`homelab.keithjackson.dev/egress-uplink` label is visible in kube-state-metrics. Without it
|
||||
nothing can tell whether Cilium's egress steering followed the last tunnel reconnect, and
|
||||
`VPNEgressSteeringStale` is inert. Check the label on the node, then the
|
||||
`vpn-egress-resteer` service/timer on the gateway node, then the kube-state-metrics
|
||||
metricLabelsAllowlist.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
|
||||
# -----------------------------------------------------------------------------------------
|
||||
# (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun
|
||||
# retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd
|
||||
# service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`
|
||||
|
||||
@@ -231,7 +231,7 @@ spec:
|
||||
# 3 extra series, no cardinality concern. Label name is sanitized by KSM to
|
||||
# `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`).
|
||||
metricLabelsAllowlist:
|
||||
- "nodes=[homelab.keithjackson.dev/egress-gateway]"
|
||||
- "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]"
|
||||
# --- VELERO SCHEDULE OBJECTS AS METRICS (added 2026-09-13) ---------------------------------
|
||||
# Velero exports NOTHING that enumerates Schedule objects and nothing that exposes
|
||||
# `spec.paused`. Verified on prod against velero v1.18.1: all 35 velero_* metric names that
|
||||
|
||||
@@ -0,0 +1,188 @@
|
||||
# promtool rule-unit tests for VPNEgressSteeringStale + VPNEgressUplinkLabelMissing
|
||||
# (platform/monitoring/alerts.yaml)
|
||||
#
|
||||
# Run with: platform/monitoring/tests/run.sh
|
||||
# (the merge gate runs the same harness via
|
||||
# conformance/tests/l0-static/alert-rules-unit-tested)
|
||||
#
|
||||
# WHAT THESE RULES CLAIM, AND WHY IT NEEDS PINNING IN BOTH DIRECTIONS
|
||||
# `egress-class=vpn` pods leave through one gateway node's tun0. Cilium resolves that interface to an
|
||||
# IFINDEX and caches it; an OpenVPN reconnect destroys and re-creates tun0 with a NEW ifindex, and the
|
||||
# gateway node's up-script publishes `homelab.keithjackson.dev/egress-uplink=<addr>-<ifindex>` so that
|
||||
# node write forces Cilium to re-resolve. If the label does not follow the tunnel, the cache points at a
|
||||
# destroyed interface and every vpn-class pod is dropped — while tun0 itself stays up, so VPNTunnelDown
|
||||
# says nothing. That shape ran ~80 minutes undetected on the dev gateway.
|
||||
#
|
||||
# VPNEgressSteeringStale detects it by comparing HOW MANY TIMES tun0's ifindex changed in the last hour
|
||||
# against HOW MANY DISTINCT `egress-uplink` label values appeared in the same hour. The subtlety worth
|
||||
# testing is that a label VALUE change is not a sample change: it RETIRES one series and STARTS another,
|
||||
# so "the label followed the tunnel" shows up as two overlapping series in the window, not as a
|
||||
# changing value. Tests 1 and 2 are the same tunnel reconnect with and without that second series, so
|
||||
# they isolate exactly that mechanism — anything that broke the series-counting half would make test 2
|
||||
# fire, and anything that broke the tun0 half would make test 1 quiet.
|
||||
#
|
||||
# VPNEgressUplinkLabelMissing is the WATCHDOG for the rule above: with no uplink label anywhere, the
|
||||
# staleness comparison has nothing to compare and is structurally INERT — present, evaluating, and
|
||||
# incapable of firing. Tests 3 and 4 pin that, because an inert critical alert is the failure mode this
|
||||
# whole group exists to prevent.
|
||||
#
|
||||
# Fixture values are prod's real shape: gateway k3s-prod-3, tun0 ifindex 139 with address
|
||||
# 10.150.192.86 (live label value `10.150.192.86-139`), reconnecting to ifindex 140.
|
||||
|
||||
rule_files:
|
||||
- generated.rules.yaml
|
||||
|
||||
evaluation_interval: 1m
|
||||
|
||||
tests:
|
||||
# =============================================================================================
|
||||
# 1. THE BUG — tun0 was re-created, the label did NOT follow. Steering is stale: FIRES.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: tunnel reconnect with no label update - VPNEgressSteeringStale fires
|
||||
input_series:
|
||||
# tun0 re-created at t=41m: ifindex 139 -> 140. One change inside the trailing hour.
|
||||
- series: 'node_network_iface_id{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
|
||||
values: '139+0x40 140+0x40'
|
||||
# tun0 is UP and passing traffic the whole time — this is what makes the failure deceptive, and
|
||||
# it is asserted below: VPNTunnelDown stays QUIET while egress is actually broken.
|
||||
- series: 'node_network_receive_bytes_total{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
|
||||
values: '0+120000x80'
|
||||
# The node label NEVER changes, so only ONE distinct uplink value exists in the window.
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3",label_homelab_keithjackson_dev_egress_gateway="true",label_homelab_keithjackson_dev_egress_uplink="10.150.192.86-139"}'
|
||||
values: '1+0x80'
|
||||
alert_rule_test:
|
||||
# 1 tunnel change > (1 distinct label value - 1) = 0, held for 15m.
|
||||
- eval_time: 60m
|
||||
alertname: VPNEgressSteeringStale
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
exp_annotations:
|
||||
summary: "VPN egress steering is stale — vpn-class pods have no egress"
|
||||
description: >-
|
||||
The gateway node's tun0 was re-created without the `egress-uplink` label following it, so
|
||||
Cilium's cached egress ifindex points at a destroyed interface: egress-class=vpn pods are
|
||||
being dropped, silently, while tun0 still looks healthy. Compare `kubectl get nodes -L
|
||||
homelab.keithjackson.dev/egress-uplink` against `cilium-dbg bpf egress list` on the
|
||||
gateway node; re-applying the CiliumEgressGatewayPolicy (or setting any node label)
|
||||
forces a fresh resolution.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
|
||||
# Not yet: at 45m the condition has been true only ~4m, under the 15m `for`.
|
||||
- eval_time: 45m
|
||||
alertname: VPNEgressSteeringStale
|
||||
exp_alerts: []
|
||||
# No gateway label AND no tun0 history, so VPNTunnelDown's `or` guard is false: a cluster with
|
||||
# no VPN gateway must not be told its tunnel is down forever.
|
||||
- eval_time: 45m
|
||||
alertname: VPNTunnelDown
|
||||
exp_alerts: []
|
||||
# The watchdog must stay quiet — the label IS present here.
|
||||
- eval_time: 60m
|
||||
alertname: VPNEgressUplinkLabelMissing
|
||||
exp_alerts: []
|
||||
# THE HEADLINE CLAIM, asserted rather than asserted-in-prose: the tunnel alert says NOTHING
|
||||
# while vpn-class egress is completely broken. If VPNTunnelDown covered this failure, the
|
||||
# staleness rule would not need to exist.
|
||||
- eval_time: 60m
|
||||
alertname: VPNTunnelDown
|
||||
exp_alerts: []
|
||||
|
||||
# =============================================================================================
|
||||
# 2. THE HEALTHY RECONNECT — same tunnel change, but the label followed it. STAYS QUIET.
|
||||
# This is the half that proves the rule is not simply "any reconnect alerts".
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: tunnel reconnect WITH the label following - stays quiet
|
||||
input_series:
|
||||
- series: 'node_network_iface_id{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
|
||||
values: '139+0x40 140+0x40'
|
||||
- series: 'node_network_receive_bytes_total{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
|
||||
values: '0+120000x80'
|
||||
# A label VALUE change retires one series and starts another. Both are inside the trailing
|
||||
# hour, so the window holds TWO distinct uplink values: 2 - 1 = 1, which is NOT < 1 change.
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3",label_homelab_keithjackson_dev_egress_gateway="true",label_homelab_keithjackson_dev_egress_uplink="10.150.192.86-139"}'
|
||||
values: '1+0x40 _x40'
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3",label_homelab_keithjackson_dev_egress_gateway="true",label_homelab_keithjackson_dev_egress_uplink="10.150.192.86-140"}'
|
||||
values: '_x41 1+0x39'
|
||||
alert_rule_test:
|
||||
- eval_time: 60m
|
||||
alertname: VPNEgressSteeringStale
|
||||
exp_alerts: []
|
||||
- eval_time: 75m
|
||||
alertname: VPNEgressSteeringStale
|
||||
exp_alerts: []
|
||||
- eval_time: 60m
|
||||
alertname: VPNEgressUplinkLabelMissing
|
||||
exp_alerts: []
|
||||
- eval_time: 60m
|
||||
alertname: VPNTunnelDown
|
||||
exp_alerts: []
|
||||
|
||||
# =============================================================================================
|
||||
# 3. THE INERT-ALERT CASE — a gateway node publishing NO uplink label at all.
|
||||
# VPNEgressSteeringStale cannot fire in this state, which is exactly why this warning exists.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: gateway node with no egress-uplink label - VPNEgressUplinkLabelMissing fires
|
||||
input_series:
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3",label_homelab_keithjackson_dev_egress_gateway="true"}'
|
||||
values: '1+0x60'
|
||||
# tun0 is reconnecting the whole time. The staleness rule STILL cannot fire, because it has no
|
||||
# label series to compare against — that is the inertness being alerted on.
|
||||
- series: 'node_network_iface_id{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
|
||||
values: '139+0x30 140+0x30'
|
||||
- series: 'node_network_receive_bytes_total{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
|
||||
values: '0+120000x60'
|
||||
alert_rule_test:
|
||||
- eval_time: 35m
|
||||
alertname: VPNEgressUplinkLabelMissing
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
exp_annotations:
|
||||
summary: "Egress gateway node publishes no egress-uplink label"
|
||||
description: >-
|
||||
A node carries `homelab.keithjackson.dev/egress-gateway=true` but no
|
||||
`homelab.keithjackson.dev/egress-uplink` label is visible in kube-state-metrics. Without
|
||||
it nothing can tell whether Cilium's egress steering followed the last tunnel reconnect,
|
||||
and `VPNEgressSteeringStale` is inert. Check the label on the node, then the
|
||||
`vpn-egress-resteer` service/timer on the gateway node, then the kube-state-metrics
|
||||
metricLabelsAllowlist.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
|
||||
# Under the 30m `for`.
|
||||
- eval_time: 20m
|
||||
alertname: VPNEgressUplinkLabelMissing
|
||||
exp_alerts: []
|
||||
# And the proof of inertness: the critical rule is silent despite a real reconnect.
|
||||
- eval_time: 55m
|
||||
alertname: VPNEgressSteeringStale
|
||||
exp_alerts: []
|
||||
- eval_time: 55m
|
||||
alertname: VPNTunnelDown
|
||||
exp_alerts: []
|
||||
|
||||
# =============================================================================================
|
||||
# 4. NO GATEWAY NODE AT ALL — neither rule may fire. Both are guarded by an `and on()` on the
|
||||
# egress-gateway label so they stay silent on a cluster with no VPN gateway, rather than
|
||||
# alerting forever about an absent feature.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: no egress-gateway node - both rules silent
|
||||
input_series:
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1"}'
|
||||
values: '1+0x60'
|
||||
- series: 'node_network_iface_id{job="node-exporter",instance="192.168.2.10:9100",device="eth0"}'
|
||||
values: '2+0x60'
|
||||
alert_rule_test:
|
||||
- eval_time: 45m
|
||||
alertname: VPNEgressUplinkLabelMissing
|
||||
exp_alerts: []
|
||||
- eval_time: 45m
|
||||
alertname: VPNEgressSteeringStale
|
||||
exp_alerts: []
|
||||
# No gateway label AND no tun0 history, so VPNTunnelDown's `or` guard is false: a cluster with
|
||||
# no VPN gateway must not be told its tunnel is down forever.
|
||||
- eval_time: 45m
|
||||
alertname: VPNTunnelDown
|
||||
exp_alerts: []
|
||||
@@ -157,6 +157,31 @@ kubectl -n kube-system exec <cilium-agent-on-the-gateway> -c cilium-agent -- \
|
||||
If they disagree, the re-steer path is broken — `journalctl -u vpn-egress-resteer` on the
|
||||
gateway node is the next stop.
|
||||
|
||||
### And now something watches it (2026-09-12)
|
||||
|
||||
The self-heal closed the datapath hole; it did not close the monitoring one, and a mechanism that
|
||||
silently stops working is the same outage again. Two rules in `platform/monitoring/alerts.yaml`:
|
||||
|
||||
- **`VPNEgressSteeringStale`** (critical, `for: 15m`) — fires when the live tun0 was re-created
|
||||
more times in the last hour than the `egress-uplink` label changed, i.e. Cilium was never told
|
||||
about a reconnect. This is the case `VPNTunnelDown` cannot see: tun0 present, bytes flowing,
|
||||
steering pointed at a destroyed ifindex.
|
||||
- **`VPNEgressUplinkLabelMissing`** (warning, `for: 30m`) — fires when a node carries
|
||||
`egress-gateway=true` but no `egress-uplink` label reaches kube-state-metrics, which is the state
|
||||
in which the rule above would be inert. The watchdog for the watchdog.
|
||||
|
||||
Both are gated on a gateway node existing (the same `kube_node_labels` guard `VPNTunnelDown` uses),
|
||||
so a cluster with no gateway cannot be made to fire them. `egress-uplink` had to be added to
|
||||
`kube-state-metrics.metricLabelsAllowlist` in `platform/monitoring/kube-prometheus-stack.yaml` for
|
||||
the label to exist as a metric at all — the two changes must ship together.
|
||||
|
||||
**What still is not watched:** "the label was written and Cilium still ignored it." Cilium OSS
|
||||
exposes no metric carrying the egress map's contents (the map is readable only through
|
||||
`cilium-dbg bpf egress list`), and the cilium-agent is not scraped on either cluster in the first
|
||||
place — no `cilium_*` or `hubble_*` series exist. Closing that last gap means either exporting the
|
||||
map's ifindex from the re-steer script through a node-exporter textfile collector, or a black-box
|
||||
probe from a vpn-class pod. Neither exists yet.
|
||||
|
||||
## Resilience — there is NO gateway failover (spec correction, escalated)
|
||||
|
||||
**ADR-0015 §Decision says "Two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)
|
||||
|
||||
+4
-2
@@ -84,8 +84,10 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
|
||||
- **`cert-expiry`** — a certificate is expiring. Almost always a late symptom of `cert-issuance`.
|
||||
|
||||
### Networking
|
||||
- **`vpn-down`** — the egress VPN tunnel. Includes the silent egress-gateway staleness that this alert
|
||||
cannot see, and what the kill-switch actually does.
|
||||
- **`vpn-down`** — the egress VPN tunnel. Explains what the kill-switch actually does, and the silent
|
||||
egress-gateway staleness this alert cannot see — which now has its own alerts and page below.
|
||||
- **`vpn-egress-stale`** — the tunnel is UP but `egress-class=vpn` pods have no egress: Cilium's cached
|
||||
ifindex outlived a reconnect. Also covers the watchdog for that rule going inert.
|
||||
- **`denied-flows`** — policy-denied flow spike. The Cilium hairpin/identity regression class.
|
||||
|
||||
### Backups and DR
|
||||
|
||||
@@ -85,3 +85,4 @@ configMapGenerator:
|
||||
- targets-down=targets-down.md
|
||||
- update-rollback=update-rollback.md
|
||||
- vpn-down=vpn-down.md
|
||||
- vpn-egress-stale=vpn-egress-stale.md
|
||||
@@ -0,0 +1,123 @@
|
||||
# vpn-egress-stale — VPN egress steering did not follow the tunnel
|
||||
|
||||
**Alerts:** `VPNEgressSteeringStale` (critical, 15m) and `VPNEgressUplinkLabelMissing` (warning, 30m)
|
||||
**File:** `platform/monitoring/alerts.yaml`
|
||||
**Related:** `/runbooks/vpn-down` is for the tunnel being DOWN. This page is for the tunnel being **up**
|
||||
while the pods that must use it have no egress at all.
|
||||
|
||||
## What this means here
|
||||
|
||||
`egress-class=vpn` pods leave the cluster through one gateway node's `tun0`, selected by the
|
||||
`vpn-egress-steering` CiliumEgressGatewayPolicy. Cilium resolves that interface to an **ifindex** and
|
||||
caches it. When OpenVPN reconnects it destroys and re-creates `tun0`, which gets a **new ifindex** — and
|
||||
Cilium does not re-resolve on its own. The gateway node publishes
|
||||
`homelab.keithjackson.dev/egress-uplink=<tun0 address>-<ifindex>` from the OpenVPN up-script on every
|
||||
connect, and that node write is what makes Cilium re-resolve.
|
||||
|
||||
If the label write does not happen, the cache keeps pointing at a destroyed interface. **`tun0` looks
|
||||
perfectly healthy the whole time** — it is up, it has an address, `VPNTunnelDown` stays quiet — while
|
||||
every vpn-class pod is silently dropped. That exact shape ran for roughly **80 minutes** undetected on
|
||||
the dev gateway, which is why these two alerts exist.
|
||||
|
||||
`VPNEgressSteeringStale` compares how many times `tun0`'s ifindex changed in the last hour against how
|
||||
many distinct `egress-uplink` label values appeared in the same hour. More tunnel changes than label
|
||||
changes means a reconnect happened that the label did not follow.
|
||||
|
||||
**`VPNEgressUplinkLabelMissing` is the watchdog for that comparison.** If the label is absent entirely,
|
||||
the staleness rule has nothing to compare against and is **inert** — present, evaluating, and incapable
|
||||
of firing. So the warning matters more than its severity suggests: it means your critical alert is
|
||||
switched off.
|
||||
|
||||
Today's gateway is **`k3s-prod-3`** (`homelab.keithjackson.dev/egress-gateway=true`), and the policy has
|
||||
been in place since early September. It is a **singleton** — there is no second gateway to fail over to.
|
||||
|
||||
## First commands
|
||||
|
||||
Run cluster commands through the pinned toolbox, which supplies the kube context:
|
||||
|
||||
```
|
||||
./infra/devcontainer/dev.sh run kubectl --context k3s-prod get nodes \
|
||||
-L homelab.keithjackson.dev/egress-gateway -L homelab.keithjackson.dev/egress-uplink
|
||||
```
|
||||
|
||||
You want exactly one node carrying `egress-gateway=true`, and that same node carrying a non-empty
|
||||
`egress-uplink` of the form `<address>-<ifindex>`.
|
||||
|
||||
```
|
||||
# 2. what Cilium currently believes, on the gateway node itself
|
||||
ssh k3s-prod-3 'sudo cilium-dbg bpf egress list'
|
||||
ssh k3s-prod-3 'ip -o link show tun0' # the REAL ifindex, first field
|
||||
```
|
||||
|
||||
Compare the ifindex in the label and in `bpf egress list` against `ip link`. **A mismatch is the whole
|
||||
bug.**
|
||||
|
||||
```
|
||||
# 3. is the re-steer mechanism itself alive?
|
||||
ssh k3s-prod-3 'systemctl status vpn-egress-resteer.service vpn-egress-resteer.timer --no-pager'
|
||||
ssh k3s-prod-3 'journalctl -u vpn-egress-resteer --since -2h --no-pager | tail -30'
|
||||
```
|
||||
|
||||
```
|
||||
# 4. are vpn-class pods actually broken, or only the metric?
|
||||
./infra/devcontainer/dev.sh run kubectl --context k3s-prod get ciliumegressgatewaypolicies.cilium.io \
|
||||
vpn-egress-steering -o yaml | head -40
|
||||
```
|
||||
|
||||
Prove egress from a pod the policy selects rather than inferring it — the observable symptom is
|
||||
"outbound traffic silently disappears", so test outbound traffic:
|
||||
|
||||
```
|
||||
./infra/devcontainer/dev.sh run kubectl --context k3s-prod -n qbittorrent exec deploy/qbittorrent -- \
|
||||
sh -c 'curl -s --max-time 10 https://ifconfig.me || echo EGRESS-FAILED'
|
||||
```
|
||||
|
||||
That should return the **VPN provider's** address. The gateway VIP or a timeout both mean egress is
|
||||
broken; your own ISP address means the kill-switch is not holding and traffic is leaking, which is worse
|
||||
than a drop — see `/runbooks/vpn-down`.
|
||||
|
||||
## Likely causes, most likely first
|
||||
|
||||
1. **A tunnel reconnect that the label did not follow.** The common case, and what the alert is built to
|
||||
catch. The OpenVPN up-script either did not run or could not write the node label (RBAC, expired
|
||||
kubeconfig on the node, API unreachable at that moment). Check `journalctl -u vpn-egress-resteer` and
|
||||
the up-script's own log around the reconnect time.
|
||||
2. **Cilium cached a dead ifindex despite the label changing.** The label write happened but the
|
||||
re-resolution did not take. Forcing any node-label write makes Cilium re-resolve — that is the same
|
||||
lever the up-script pulls, so it is a safe nudge, not a workaround:
|
||||
|
||||
./infra/devcontainer/dev.sh run kubectl --context k3s-prod annotate node k3s-prod-3 \
|
||||
egress-resteer-nudge="$(date -u +%s)" --overwrite
|
||||
|
||||
Re-applying the CiliumEgressGatewayPolicy has the same effect. Re-check `bpf egress list` after.
|
||||
3. **`metricLabelsAllowlist` lost the label** (this fires `VPNEgressUplinkLabelMissing`, not the stale
|
||||
alert). kube-state-metrics exposes **no** custom node labels by default; the allowlist entry in
|
||||
`platform/monitoring/kube-prometheus-stack.yaml` is what publishes them. If that entry is edited or
|
||||
dropped, `kube_node_labels` carries no `label_homelab_keithjackson_dev_egress_uplink` and the critical
|
||||
alert goes quiet without anything looking broken. Confirm what is actually exported:
|
||||
|
||||
./infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \
|
||||
prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
|
||||
/bin/promtool query instant http://localhost:9090 'kube_node_labels'
|
||||
|
||||
Note that KSM does **not** restart on a ConfigMap-only change; the deployment carries no checksum
|
||||
annotation over it, so a `rollout restart -n monitoring deploy/kube-prometheus-stack-kube-state-metrics`
|
||||
is required after editing the allowlist.
|
||||
4. **The gateway node was replaced or relabelled.** If `egress-gateway=true` moved, the up-script must be
|
||||
installed on the new node (`infra/ansible/roles/vpn_egress_gateway`). A gateway label with no
|
||||
up-script behind it produces exactly this alert on the next reconnect.
|
||||
|
||||
## When to escalate or stop
|
||||
|
||||
- **Stop and do not "fix" it by deleting the policy.** Removing `vpn-egress-steering` does not restore
|
||||
egress; it sends vpn-class traffic out of your **own** ISP connection, which is the leak the class
|
||||
exists to prevent. A broken tunnel that drops traffic is the intended failure mode.
|
||||
- **Escalate to a maintenance window** if the ifindex mismatch returns after every reconnect. That is the
|
||||
up-script or its RBAC, not a transient, and chasing it live means repeatedly breaking egress for the
|
||||
vpn-class workloads.
|
||||
- **This is a singleton gateway.** There is no failover. If `k3s-prod-3` needs to be drained while this is
|
||||
unresolved, accept that vpn-class egress is down for the duration rather than improvising a second
|
||||
gateway under time pressure.
|
||||
- If `VPNEgressUplinkLabelMissing` is firing and you cannot restore the label quickly, say so explicitly
|
||||
in the incident notes: for as long as it is missing, **`VPNEgressSteeringStale` is not watching
|
||||
anything**, so the absence of that critical alert proves nothing.
|
||||
Reference in new issue
Block a user