0749562ad0d6cf5b6b42aef79ec7e8d247da8a86
9
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
a3f4ddb971 |
merge: feat/egress-gateway-staleness-alert — finished so it can actually pass the gate
Brings the two alerts for the silent egress-gateway failure: VPNEgressSteeringStale
(critical) and VPNEgressUplinkLabelMissing (warning), plus the kube-state-metrics
allowlist entry that makes them non-inert.
WHY THE BRANCH SAT IN THE BACKLOG: it could not pass the merge gate. Three
whole-repo checks failed on it, and each one was a real gap rather than a nuisance:
* alert-runbooks-resolvable — both alerts link to /runbooks/vpn-egress-stale,
which did not exist. An alert firing now would hand the operator a 404.
* alert-rules-unit-tested — alerts.yaml is already a covered file, so new rules
in it must be tested or a "tested" file quietly grows untested rules.
* alert-metrics-have-scrape-targets — see the guard bug below.
WHAT THE ALERTS CATCH. egress-class=vpn pods leave through one gateway node's tun0;
Cilium resolves it to an ifindex and caches it. An OpenVPN reconnect destroys and
re-creates tun0 with a NEW ifindex, and the node label
homelab.keithjackson.dev/egress-uplink=<addr>-<ifindex> is what forces Cilium to
re-resolve. If the label does not follow the tunnel, the cache points at a destroyed
interface and every vpn-class pod is silently dropped — while tun0 stays up and
VPNTunnelDown says nothing. That ran ~80 minutes undetected on the dev gateway.
Verified this is live on prod, not inert: k3s-prod-3 carries egress-gateway=true and
egress-uplink=10.150.192.86-139, and vpn-egress-steering has been in place 14 days.
The label was exported on 0 series before this change, so the allowlist entry is
what switches the alerts on.
runbooks/vpn-egress-stale.md — written and wired into the configMapGenerator, with
the four required operator sections. It leads with proving egress from a pod the
policy selects, because the symptom is "outbound traffic silently disappears", and
it says plainly not to "fix" this by deleting the policy: that does not restore
egress, it sends vpn-class traffic out of the real ISP connection, which is the leak
the class exists to prevent.
platform/monitoring/tests/vpn-egress.test.yaml — four promtool fixtures. 1 and 2 are
the SAME tunnel reconnect with and without the label following it, which isolates the
mechanism the rule actually keys on: a label VALUE change is not a changing sample,
it retires one series and starts another. 3 is the inert case (gateway node, no uplink
label) where the critical rule CANNOT fire, which is why the warning exists.
Two fixes the branch needed that were defects in the checks themselves:
1. alert-metrics-have-scrape-targets asserted the KSM allowlist matched
`nodes=[...egress-gateway]` with a literal closing bracket — i.e. that the list
contained EXACTLY that one label. Adding a second label correctly (egress-uplink,
which these alerts need) therefore FAILED the check while the property it guards
was perfectly satisfied. Loosened to `[,\]]`: in the list, not the whole list. A
guard that breaks when you correctly extend the thing it guards teaches people to
edit the guard, which is how it stops guarding.
2. UNTESTED_ALERTS listed VPNTunnelDown, and the check flagged it as stale because
"asserted on" is a bare word match over the test file — my header COMMENT
mentioning VPNTunnelDown was enough to count as coverage. Rather than delete the
register entry and claim coverage that did not exist, the fixtures now assert
VPNTunnelDown QUIET in all four cases. That required making them faithful: they
had no node_network_receive_bytes_total{device="tun0"} at all, so VPNTunnelDown
would have been firing throughout unnoticed, because promtool only checks the
alertnames you name. A healthy tun0 passes traffic, and that silence IS the claim
these rules exist for.
NOTE FOR LATER, not fixed here: that word-match means any alert named in prose counts
as tested. Tightening it to `alertname:` positions could reveal other prose-only
"coverage", and the register is shrink-only, so the cascade wants doing deliberately
rather than at 01:00 unattended.
Conflicts resolved: kept HEAD's Velero custom-resource-state block and took only the
branch's one-line allowlist change (14 lines down to 1); inserted the branch's
2026-09-12 DECISIONS-LOG entry in date order rather than appending it out of sequence.
Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips,
0 unwired. The new unit tests pass under the pinned promtool 3.14.0.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
4f2ca8c1a3 |
test(conformance): record the L0 verifier for R-STATE-03 / R-ENV-02
A requirement reads PROVEN only when EVERY verifying test has a fresh ledger row — the state is the minimum across them, so a test with no row contributes not-derivable and one recorded verifier cannot manufacture green. R-STATE-03 carries two executable verifiers (L0 + L3). The L3 dr-restore proof landed in bd17b65; this records the L0 side, placeholder-guard-coupling, which also carries R-ENV-02. Worth noting it is the test that was permanently SKIPPED until earlier today — it was hiding a red, and now it runs and is recorded. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
bd17b65db9 |
test(conformance): R-STATE-03 PROVEN on prod — DR restore from R2, off-cluster
Phase 7's outstanding exit criterion. The only previous dr-restore proof was
recorded on the dev cluster (decommissioned 2026-09-12) against MinIO, so it
could not speak for prod or for today's Cloudflare R2 target.
Run 20260914T021505Z-24b8111, env=prod, kube v1.36.4+k3s1, chainsaw v0.2.12,
150.70s, 1 passed / 0 failed / 0 skipped. Sentinel
DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK.
What makes it a real proof rather than a green check: the test seeds a canary
on the replicated longhorn class, backs the namespace AND volume data up to
the off-cluster store, then DESTROYS every in-cluster copy — namespace, PVC
and PV — before restoring into a clean, previously-nonexistent namespace. It
asserts the restored PVC binds a DIFFERENT PersistentVolume
(pvc-f0c061e1 != pvc-4107d59d) and fails explicitly if it rebinds the
original, because that would be an in-cluster survival masquerading as a
restore. The canary came back byte-identical.
It touches no real service's namespace, PVC or PV — the scratch namespaces are
named after its own Chainsaw namespace and are destroyed by design.
This is the run whose first attempt was discarded for recording env=dev; the
recorder fix is
|
||
|
jakman85andClaude Opus 5
|
155df8ee3f |
test(conformance): R-NET-06 PROVEN on prod — vpn-killswitch re-run from a clean tree
Supersedes the -dirty row from the previous commit by the ledger's own
append-only / latest-wins convention (lib/status.py latest_by_test): nothing
is rewritten, a newer row simply outranks the older one.
runs/20260913T014923Z-9d0814a vpn-killswitch (L3) proven env=prod
git_sha 9d0814ae... (CLEAN — resolvable, so the freshness check passes)
Fault injection against LIVE prod, gateway k3s-prod-3. Timeline (UTC):
01:49:27 tunnel-stop Job applied
01:49:31 Job succeeded — host vpn-egress-tunnel STOPPED, tun0 gone
01:49:31..01:51:01 vpn-class blackholed for 8 consecutive polls, NO response
of any kind; internet-class unaffected on the WAN
01:51:02 self-healing finally: tunnel-restart Job
01:51:05 kubectl wait confirms the host brought the unit back
01:51:04 egress-uplink label AND the BPF egress map both move 134 -> 139
(measured independently, out of band, during the run)
01:51:08 vpn-class egress back on PIA 158.173.241.11
RECOVERY WAS UNAIDED and took ~3s from tunnel-up to a re-steered datapath —
far inside the ~2-minute convergence timer, so the EVENT-DRIVEN up-script
path did the work, not the timer backstop. This is the half that was
previously unprovable: before the egress-uplink label, Cilium held the stale
ifindex and vpn-class egress never came back.
WHAT THE NO-LEAK ASSERTION DOES AND DOES NOT PROVE: it proves vpn-class
fails CLOSED (zero packets out, never the WAN IP). It does NOT credit the
host nftables backstop, which provably cannot fire for pod traffic —
bpf_redirect() skips netfilter. Stopping the tunnel removes tun0 and Cilium
drops on its own because the cached egress ifindex is unusable. Fail-closed
is real; the mechanism is Cilium's drop, as the test header now says.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
9d0814ae61 |
test(conformance): R-NET-06 run against prod — L1 proven, L3 pass recorded -dirty
Two recorded runs against PROD (CONF_ENV=prod, context k3s-prod): 20260913T014321Z-19fc8bc egress-class-render (L1) proven 20260913T014437Z-19fc8bc vpn-killswitch (L3) proven, but git_sha -dirty The L3 run PASSED for real on prod — every declared sentinel present (NS_PRIVILEGED_OK, TUNNEL_HEALTHY_OK, NO_LEAK_OK, INTERNET_UNAFFECTED_OK, TUNNEL_SELF_HEAL_OK, TUNNEL_RECOVERED_OK), tunnel stopped 01:44:48Z and recovered unaided by 01:46:25Z. Its row is nevertheless NOT usable as proof: git_sha carries `-dirty`, so status.py's ancestry check cannot resolve it and the row degrades to stale-proof. Cause is a harness flaw, not the cluster: lib/record_run.py's git_sha() ignores UNTRACKED files (so a run's own runs/<id>/ does not mark it dirty) but conformance/runs/ledger.jsonl is TRACKED — so the FIRST --record run of a session appends a row, dirties the tree, and every SUBSEQUENT --record run in that same session is stamped -dirty. Recording one test poisons the proof of the next. Committed as-is rather than rewritten: the ledger is append-only and this is honest evidence of a real run. A clean re-run follows and supersedes it by latest-wins (lib/status.py latest_by_test). Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
380ea98a38 |
feat(conformance): R-STATE-03 PROVEN — DR restore from off-cluster backup, executed
The second requirement ever proven on this project, and the one the operator had attested was working but the suite could not attest to. Test authored 2026-08-31 carrying a `conformance.home.k3s/never-executed` annotation; executed for the first time tonight. test dr-restore run 20260902T001846Z-fdc3265 req R-STATE-03 (safety-critical, DR) junit passed / 0 failed / 0 skipped sentinel DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK proof proven sha |
||
|
jakman85
|
4cb5e7a8d4 |
feat(conformance): first PROVEN row — R-STATE-02 node-loss
Records the first requirement this project has ever PROVEN, in the only sense its own
definition of done accepts: a green, requirement-tagged, reproducible behaviour test,
executed against a real cluster, with the evidence committed.
test node-loss run 20260901T235117Z-e9a4b66
req R-STATE-02 (safety-critical) junit passed
sentinel NODE_LOSS_VOLUME_AVAILABLE_OK proof proven
sha
|
||
|
jakman85andClaude Opus 5
|
57d4df8c4f |
fix(conformance): ledger read tokens from Chainsaw's COMMAND echo, not real output
The run ledger mis-classified the FIRST real L3 execution — a genuine PASS recorded as
`skipped`. Diagnosis, from the preserved log, section-classified:
line 51 COMMAND echo NODE_LOSS_SKIPPED_OK <- script SOURCE
line 190 STDOUT NODE_LOSS_VOLUME_AVAILABLE_OK <- the only REAL emission
Chainsaw echoes each script into a `=== COMMAND` block before running it, and every
skip-gated test contains `echo <TEST>_SKIPPED_OK` as source text. record_run.py scanned the
raw log without distinguishing sections, so it matched the skip token in the echo.
NODE_LOSS_SKIPPED_OK never appeared in output at all.
SEVERITY: since every skip-gated test embeds its own skip token as source, and skip outranks
pass, THE LEDGER COULD NEVER HAVE RECORDED `proven` FOR ANY TEST WITH A SKIP GATE. It would
have silently recorded nothing forever while appearing to work. It failed CLOSED — the right
direction — but a ledger that always fails closed is an expensive way to write "unknown".
FIX: classify_sections() splits the log on indented `=== NAME` markers (a section runs to the
first non-blank line indented LESS than its marker). Verified against the real log's actual
markers, including that Go's column-0 `=== RUN/PAUSE/CONT` lines carry a test name and must
never open a section. Then:
PASS tokens are read from STDOUT ONLY — exactly where Chainsaw's own contains($stdout,…)
looks, so a source echo or stderr can never manufacture a proof.
SKIP tokens are read from every section that is NOT COMMAND, so an unknown future section
name fails closed in BOTH directions.
Lines outside any section prove nothing — this also closes a second false-positive vector:
a FAILING Chainsaw run prints the assertion expression, which contains both tokens.
A log with no markers is scanned whole exactly as before (such logs have no COMMAND echo).
SECOND BUG, same root: _LOG_HEADER_RE never matched a real Chainsaw log — the lines are
` l.go:52: <BS>| 23:51:18 | <ANSI>node-loss<ANSI> | …`, not `| … |`. EVERY genuine run was
silently falling into DEGRADED whole-log mode. Now strips ANSI/backspace noise first and
accepts the Go source prefix, so runs attribute per-testcase as designed.
Safety properties re-asserted, not assumed: skip still beats pass WITHIN real output; unknown
layers still require a sentinel; JUnit failed/error/skipped still outrank everything; missing
JUnit or log still exits 3. Negative control: with scan_texts forced section-blind in a
throwaway copy, exactly the new cases fail — they do catch this bug.
Includes the preserved run directory 20260901T235117Z-e9a4b66 (report.junit.xml + run.log)
for R-STATE-02 / node-loss — 131s, cordon -> evict -> volume stayed available -> reader
re-attached -> clean teardown, the first L3 fault-injection test ever executed on this
cluster. The ledger row is recorded in the FOLLOWING commit, deliberately: git_sha is sampled
at record time and a dirty tree records `-dirty`, which is unresolvable and correctly degrades
to stale-proof. Recording against this clean commit is the honest way to get a resolvable sha
— NOT a --git-sha override, which would be a lever for recording a clean sha over a dirty
tree and was deliberately not added.
Serves: R-TEST-01..04, R-STATE-02.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
e9a4b669d4 |
feat(conformance): committed run ledger — the only thing that can move PROVEN off zero
status.py has always reported PROVEN: 0 for all 51 requirements, and named the fix in its
own header: "a committed, machine-readable run ledger". There was none — _artifacts/ is
gitignored, the root .gitignore kills *.junit.xml, and a stray JUnit was surfaced only as
a quarantined mtime advisory. Built BEFORE the first real L3 execution so that run is
captured rather than reconstructed from memory.
THE TRAP THIS EXISTS TO AVOID. These tests exit 0 and Chainsaw records `passed` on a
SELF-SKIP exactly as on a real run, and Chainsaw's JUnit carries no stdout — only
properties. So committing the JUnit as proof would let DR_RESTORE_SKIPPED_OK be recorded
as proof of R-STATE-03: authoritative-looking and wrong, which is worse than no ledger.
The sentinel token is the ONLY thing distinguishing proven from skipped, and it lives
solely in the run log. The ledger therefore records the LOG alongside the JUnit and
derives proof from the sentinel, never from the JUnit result.
Sentinel tokens are DERIVED, not hand-listed: record_run.py parses every
contains($stdout,'TOKEN') out of each test's own chainsaw-test.yaml, so the list cannot
drift from the tests. 23 of 35 tests declare no sentinel at all.
decide() is the single place `proven` is emitted, ordered WORST-FIRST:
JUnit failure/error -> failed
JUnit <skipped/> -> junit-skipped
ANY declared skip token in the log -> skipped (beats a green JUnit AND beats a pass
token present in the same log)
no sentinel declared -> proven ONLY for L0/L1 (whose JUnit is truthful on its own, since
run_l1_offline.py maps exit 3 to <skipped/>); written as an
explicit exempt-set so an unknown/blank layer fails CLOSED
EVERY declared pass token present -> proven
otherwise -> unknown-sentinel-absent
Log attribution is a state machine over Chainsaw's per-test headers; a log with no
recognisable header falls back to whole-log scanning restricted to GLOBALLY-UNIQUE tokens,
so DNS_OK (owned by two tests) can never prove either. record_run.py exits 3 if the JUnit
or the log is missing — nothing recorded, never green.
STALENESS: a proven row is honoured only while `git merge-base --is-ancestor` holds
between the requirement's last implementing/test change and the row's git_sha; otherwise
it degrades to stale-proof. A -dirty sha, an unknown rev, or an uncommitted test path all
fail ancestry the same way. A requirement reads proven only when EVERY verifying test
does (worst-of), and the latest row per test wins — not best-ever.
REFUSAL POSTURE PRESERVED: with an empty ledger the report is unchanged in intent — every
requirement not-derivable, headline "PROVEN: 0 — NOT DERIVABLE". This replaces "not
derivable" with real evidence only where evidence exists, never with optimism.
--record is explicit, never implicit (a run that silently records itself is how a bad run
becomes permanent false proof), is rejected for --status/--coverage-only/--lint, and
re-execs under tee returning ${PIPESTATUS[0]} so recording can never soften a gate. It
rm -f's a stale _artifacts/report.junit.xml first — a hazard hit live during development,
where a stale JUnit got paired with a fresh log.
Selftests: record_run.py 23 checks (incl. end-to-end Chainsaw-passed + NODE_LOSS_SKIPPED_OK
-> proof "skipped"), status.py +16 against a throwaway git repo (self-skip -> skipped;
proven-at-older-sha -> stale-proof; dirty/unresolvable -> stale-proof; worst-of;
latest-row-wins; malformed lines dropped). No regressions: --coverage-only, --status,
--lint, --strict-coverage and the merge-gate shape all rc=0 and unchanged; nothing in ci/
references status.py.
Root .gitignore needed no change — conformance/runs/ is not itself excluded, so the deeper
negations override the root's global *.junit.xml. Verified with git check-ignore.
Serves: R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|