jakman85andClaude Opus 5
a3f4ddb971
Brings the two alerts for the silent egress-gateway failure: VPNEgressSteeringStale
(critical) and VPNEgressUplinkLabelMissing (warning), plus the kube-state-metrics
allowlist entry that makes them non-inert.
WHY THE BRANCH SAT IN THE BACKLOG: it could not pass the merge gate. Three
whole-repo checks failed on it, and each one was a real gap rather than a nuisance:
* alert-runbooks-resolvable — both alerts link to /runbooks/vpn-egress-stale,
which did not exist. An alert firing now would hand the operator a 404.
* alert-rules-unit-tested — alerts.yaml is already a covered file, so new rules
in it must be tested or a "tested" file quietly grows untested rules.
* alert-metrics-have-scrape-targets — see the guard bug below.
WHAT THE ALERTS CATCH. egress-class=vpn pods leave through one gateway node's tun0;
Cilium resolves it to an ifindex and caches it. An OpenVPN reconnect destroys and
re-creates tun0 with a NEW ifindex, and the node label
homelab.keithjackson.dev/egress-uplink=<addr>-<ifindex> is what forces Cilium to
re-resolve. If the label does not follow the tunnel, the cache points at a destroyed
interface and every vpn-class pod is silently dropped — while tun0 stays up and
VPNTunnelDown says nothing. That ran ~80 minutes undetected on the dev gateway.
Verified this is live on prod, not inert: k3s-prod-3 carries egress-gateway=true and
egress-uplink=10.150.192.86-139, and vpn-egress-steering has been in place 14 days.
The label was exported on 0 series before this change, so the allowlist entry is
what switches the alerts on.
runbooks/vpn-egress-stale.md — written and wired into the configMapGenerator, with
the four required operator sections. It leads with proving egress from a pod the
policy selects, because the symptom is "outbound traffic silently disappears", and
it says plainly not to "fix" this by deleting the policy: that does not restore
egress, it sends vpn-class traffic out of the real ISP connection, which is the leak
the class exists to prevent.
platform/monitoring/tests/vpn-egress.test.yaml — four promtool fixtures. 1 and 2 are
the SAME tunnel reconnect with and without the label following it, which isolates the
mechanism the rule actually keys on: a label VALUE change is not a changing sample,
it retires one series and starts another. 3 is the inert case (gateway node, no uplink
label) where the critical rule CANNOT fire, which is why the warning exists.
Two fixes the branch needed that were defects in the checks themselves:
1. alert-metrics-have-scrape-targets asserted the KSM allowlist matched
`nodes=[...egress-gateway]` with a literal closing bracket — i.e. that the list
contained EXACTLY that one label. Adding a second label correctly (egress-uplink,
which these alerts need) therefore FAILED the check while the property it guards
was perfectly satisfied. Loosened to `[,\]]`: in the list, not the whole list. A
guard that breaks when you correctly extend the thing it guards teaches people to
edit the guard, which is how it stops guarding.
2. UNTESTED_ALERTS listed VPNTunnelDown, and the check flagged it as stale because
"asserted on" is a bare word match over the test file — my header COMMENT
mentioning VPNTunnelDown was enough to count as coverage. Rather than delete the
register entry and claim coverage that did not exist, the fixtures now assert
VPNTunnelDown QUIET in all four cases. That required making them faithful: they
had no node_network_receive_bytes_total{device="tun0"} at all, so VPNTunnelDown
would have been firing throughout unnoticed, because promtool only checks the
alertnames you name. A healthy tun0 passes traffic, and that silence IS the claim
these rules exist for.
NOTE FOR LATER, not fixed here: that word-match means any alert named in prose counts
as tested. Tightening it to `alertname:` positions could reveal other prose-only
"coverage", and the register is shrink-only, so the cascade wants doing deliberately
rather than at 01:00 unattended.
Conflicts resolved: kept HEAD's Velero custom-resource-state block and took only the
branch's one-line allowlist change (14 lines down to 1); inserted the branch's
2026-09-12 DECISIONS-LOG entry in date order rather than appending it out of sequence.
Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips,
0 unwired. The new unit tests pass under the pinned promtool 3.14.0.
Co-Authored-By: Claude Opus 5 <[email protected]>
Conformance tests — layout & requirement-tagging convention
Every test is a Chainsaw Test in its own directory:
tests/
l0-static/ L0 — whole-repo invariants over the tracked tree (no cluster)
l1-rendering/ L1 — ServiceDefinition -> rendered resources match golden
l2-behavioral/ L2 — real runtime behavior (probes against a cluster)
l3-fault-injection/ L3 — break something; assert fail-closed / no-data-loss
meta/ harness self-checks (R-TEST-*), cluster-independent
<test-name>/chainsaw-test.yaml
The layer a test belongs to is its subtree and its .../layer annotation; the
coverage generator cross-checks the two.
L0 static (l0-static/) — what belongs here, and what does not
docs/14's L0 row is "Pre-apply, in CI · manifests/ServiceDefinitions obey invariants".
Kyverno (platform/kyverno/) covers the invariants expressible over a single
resource. This subtree covers the L0 invariants that are properties of the repository
as a whole and that no per-resource policy can state — e.g. no plaintext secret exists
anywhere in the tracked tree, no tracked RoleBinding grants a workload identity a
privileged role, the spec-immutability gate actually blocks a modified spec.
Each l0-static/<name>/ dir carries a check.sh: a self-contained offline assertion
script (git + Python stdlib/PyYAML; no cluster, no network) that exits 0/1. It is executed:
- by
conformance/run.sh --layer l0(and as the offline leg of a full run), vialib/run_l1_offline.py, producing a REAL pass/fail JUnit entry — never a skip; and - by the merge gate's
ci/l0.shstage, so these assertions run on every PR; and - by the wrapping Chainsaw
Test, which runs the SAMEcheck.sh, so an in-cluster run executes byte-identical assertions.
Two authoring rules specific to this layer:
- State the gap. A statically-provable half of a requirement is still only half.
Every
check.shheader ends with an explicit "what this does NOT prove" list naming the cluster-layer remainder. Covering a requirement at L0 does not finish it. - Exemptions must be self-policing. Where a check must tolerate something (a
deliberately-plaintext policy fixture, an
.exampletemplate), the exemption is conditioned on a property that would break if the exemption were abused — e.g. the plaintext Secret fixture is exempt only while akyverno-test.yamlasserts the policy rejects it. Never a bare path allowlist.
Requirement-tagging convention (authoritative)
A test declares the requirement(s) it verifies three ways; the label is the source of truth, the annotations must agree with it (the coverage generator fails on drift):
metadata:
name: default-deny
labels:
# ONE label per verified R-* ID. Machine-selectable + coverage source of truth.
conformance.home.k3s/R-NET-01: "true"
annotations:
# Human-readable canonical list — MUST equal the label set.
conformance.home.k3s/requirements: "R-NET-01"
# Declared layer (L0|L1|L2|L3|META) — must match the subtree.
conformance.home.k3s/layer: "L2"
# true => R-TEST-05 requires this to be (or have) an L3 fault-injection test.
conformance.home.k3s/safety-critical: "false"
# Given/When/Then intent, for the coverage report and reviewers.
conformance.home.k3s/intent: "Given ... When ... Then ..."
Why a label per requirement (not a comma list): Kubernetes label values can't hold
a comma-separated list, and chainsaw test --selector selects on label keys/values.
One conformance.home.k3s/<R-ID>=true label per requirement makes both subset-selection
and coverage extraction exact and robust.
How this drives the tooling
- Subset runs (merge gate):
run.sh --requirements R-NET-01,R-NET-06resolves the R-IDs to the set of test dirs carrying those labels (union, deduped) and runs each once. - Coverage (
lib/coverage.py): reads these labels from everychainsaw-test.yaml, maps each registry requirement -> its verifying test(s), and flags gaps + any label/annotation mismatch + tags not in the registry. - JUnit: results (real or skipped-scaffold) carry the R-IDs in
classname+ properties, so the JUnit itself stays traceable (R-TEST-04).
Authoring rules (R-TEST-01/02/06)
-
Behavior, not existence. Assert what the system does (a probe cannot connect), never that an object merely exists (a policy can exist and not enforce).
-
Given/When/Then structure with
given-*/when-*/then-*step names. -
Hermetic & self-cleaning. Rely on Chainsaw's per-test namespace + auto-teardown; put any out-of-namespace fixture cleanup in a
finallyblock. A leaked fixture is a failure. -
Bounded, deterministic waits. Use assert/error polling within the config timeouts; never
sleep-and-hope. -
Safety-critical => fault-injection. For a safety-critical requirement, actually break the thing and assert fail-closed / no-data-loss, with
finallyrestoring state. -
Never weaken a test to go green (R-TEST-06). If the spec can't be met, escalate to the PM — do not relax the assertion or the requirement.
-
Never write a per-environment value into a test (R-ENV-02). No hostname suffix, no Gateway VIP, no LB-pool range. One authored suite must run green against dev and prod, so a test asks the cluster it is pointed at what it is. The mechanism is a chainsaw binding over the same ConfigMap Flux substitutes from:
spec: bindings: - name: baseDomain value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)Declare it at test level (step
outputs/bindingsare step-scoped), then consume it:($baseDomain)for a podenv:value,(join('.', ['myhost', $baseDomain]))where a full hostname is needed — including inside afixtures/*.yamlapplied withapply: file:, which chainsaw templates with the same bindings. Prefer reading the actual observed value over restating a range (e.g. assert the DNS answer equalsGATEWAY_VIP, or equals the LB IP the Service was assigned) — it is both stricter and environment-independent. Never take the environment from a CLI flag or an exported variable: a flag someone forgets is how a suite ends up silently asserting the other cluster's expectations. L1 goldens are the one exception: they are rendering fixtures and carry the literal${BASE_DOMAIN}token.
How a scaffold declares itself (and how status.py tells one from a verifier)
Scaffolds in this tree carry TODO(T-x.y) markers where a fixture/assertion lands as a
later roadmap phase delivers the capability under test. lib/status.py classifies each
test EXECUTABLE or SCAFFOLD from the test's own files — chainsaw-test.yaml,
render-check.sh, check.sh — and either signal below outranks
conformance.home.k3s/executable: "true", so a placeholder body can never be reported as
an executable test. Declare a scaffold in both positions:
# STATUS: scaffold — <what is missing>in the file's header comment block. The declared state must be the first token afterSTATUS:(executable tests say# STATUS: EXECUTABLE/IMPLEMENTED/RUNNABLE).- A
TODO/FIXMEmarker carrying a message, in the body — the file from its first non-comment line onward.echo "TODO(T-4.2): inject a fault to fire a rule", or an inline comment on a missing fixture:file: fixtures/smtp-sink.yaml # TODO(T-4.1): hermetic SMTP capture sink.
What is not a scaffold marker, and why the distinction is load-bearing: a marker
mentioned in the header prose (the Given/When/Then and the mandatory "what this does
NOT prove" list), and a marker that is the entire content of a quoted literal — a
placeholder detector's own vocabulary, e.g. SEED = ["TBD", "TODO", "FIXME", ...] in
l0-static/placeholder-guard-coupling/check.sh. A whole-file "TODO" in text test used to
classify that 32-assertion check as a scaffold and, through it, report the safety-critical
R-STATE-03 as exec=1 scaffold=1. If your check must hold a marker as data, keep each
idiom a standalone quoted literal in a list; a marker with text attached (a regex
alternation, several idioms in one string) still classifies as a scaffold, because
ambiguity here only ever downgrades.