jakman85andClaude Opus 5
59ffbaf11a
Six L2 behavioural tests embedded `home.keithjackson.dev`. They are test INPUTS,
not reconciled manifests, so postBuild substitution never touched them — correct
while dev held the apex, wrong the moment prod came up on
prod.home.keithjackson.dev, and blocking §9's "suite passes green against prod".
Each test now asks the cluster what it is, via a test-level chainsaw binding over
flux-system/cluster-vars — the SAME ConfigMap Flux renders the platform from, so
a test cannot disagree with the cluster it runs on:
bindings:
- name: baseDomain
value: (x_k8s_get($client,'v1','ConfigMap','flux-system','cluster-vars').data.BASE_DOMAIN)
Chosen over a flag or env var because there is nothing to forget at run time, and
test-level scope covers every step (step outputs/bindings are step-scoped).
THE APEX WAS NOT THE ONLY LEAK. dns-exposure and dns-internal-resolution also
hardcoded DEV'S Cilium LB range 192.168.1.53-.60; prod's is 192.168.2.20-.30, so
both were wrong for prod regardless of hostname. Replaced with assertions on the
OBSERVED value — the DNS answer must equal that cluster's GATEWAY_VIP, and the
public record must equal the LB IP the Service was actually assigned — which is
strictly STRONGER than the old range check as well as environment-independent.
sso-forward-auth held the literal only in prose (it talks over Service DNS).
PROVEN, NOT ASSERTED — and the positive control is the important half:
* expressions extracted verbatim from the tracked files resolve to the correct,
DIFFERENT values per cluster (dev home.keithjackson.dev / 192.168.1.53,
prod prod.home.keithjackson.dev / 192.168.2.20);
* the same assert with the HARDCODED apex passes on dev and FAILS on prod
("Invalid value: *.prod.home... Expected: *.home..."), so the harness is
provably not blind — that is exactly the drift removed;
* chainsaw lint VALID on all six; run.sh --lint clean over the whole tree.
END-TO-END L2 RUN AGAINST LIVE DEV: `dns-exposure` PASSED (42.6s) — the binding
produced conformance-dns.home.keithjackson.dev, external-dns published it, and the
probe Job asserted the record against the assigned LB IP. This is the first time
one of these tests has been executed since parameterization, and it exercises the
full path rather than just the binding.
FINDING FROM THAT RUN, worth recording: the test's teardown is ASYNCHRONOUS. It
reports PASS as soon as chainsaw deletes the namespace, but the Cloudflare record
is reaped later by external-dns's own --interval=1m sync. A zone query immediately
after PASS still showed conformance-dns.home.keithjackson.dev plus its owner=default
TXT registry entry. Harmless here, but it means a green dns-exposure does NOT prove
cleanup happened, and if external-dns were suspended the record would persist
silently. Verified separately that it clears.
conformance/incluster/rbac.yaml: its cluster-wide `get configmaps` is now
LOAD-BEARING for these bindings — narrowing it to the test namespace breaks five
tests. Commented so it is not "tidied" later.
l3-fault-injection/dr-restore: its KNOWN-GAPS block said off-host is unproven
because the MinIO shares a Proxmox host with dev's VMs. Still true as run, so the
text now says both — that the R2 move (32d6766) is what closes it, that this test's
pass is against MinIO, and that the ledger row proving R-STATE-03 PREDATES the move,
so off-host DR cannot be claimed until the drill is re-run against R2.
NOT VERIFIED, plainly: no full L2 suite run against prod, so "green against prod"
remains unproven — what is proven is that the tests now derive the CORRECT
expectation for whichever cluster they are pointed at. Whether prod SATISFIES them
(split-horizon, LE wildcard, Grafana's issuer) is untested.
VERIFIED: full CI gate PASS. flux-postbuild-substitution 260 assertions,
render-golden 10/10.
Refs R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04, R-ENV-02, R-DNS-01/02/03.
Co-Authored-By: Claude Opus 5 <[email protected]>
316 lines
16 KiB
YAML
316 lines
16 KiB
YAML
# =============================================================================
|
|
# L2 BEHAVIORAL — Exposed app reachable through the Gateway under default-deny
|
|
# (R-NET-01, R-NET-02, R-NET-03) [RUNNABLE]
|
|
# Catalog: docs/14 (default-deny + explicit allow family)
|
|
# Spec: docs/03 "Default-deny baseline" / "Explicit allow rules";
|
|
# docs/09 "Interaction with default-deny" (ingress opens NORTH-SOUTH only).
|
|
#
|
|
# THE GAP THIS CLOSES: an onboarded app's rendered default-deny has `ingress: []`
|
|
# (deny ALL), so the Cilium Gateway (Envoy, reserved identity `ingress`) cannot
|
|
# reach the backend — an Accepted HTTPRoute is still unreachable. The renderer now
|
|
# also emits an ADDITIVE gateway-ingress CiliumNetworkPolicy allowing ONLY the
|
|
# `ingress` identity to ONLY the declared port. This test proves that end-to-end
|
|
# through the REAL shared Gateway:
|
|
#
|
|
# GIVEN an exposed app namespace with default-deny + the gateway-ingress allow
|
|
# (fromEntities: [ingress] -> declared port 8080 only), attached to the
|
|
# shared Gateway with a declared route (8080) and an undeclared route (9091),
|
|
# WHEN a client dials the Gateway VIP for the DECLARED host -> THEN it reaches the
|
|
# backend (proves the additive allow opens the north-south path, R-NET-02/03);
|
|
# WHEN a client dials the Gateway VIP for the UNDECLARED port -> THEN the backend is
|
|
# NOT reached (default-deny still drops it, R-NET-02);
|
|
# WHEN an in-namespace peer (NOT the `ingress` identity) dials the declared port
|
|
# east-west -> THEN it is DENIED (the allow is least-privilege / gateway-only;
|
|
# default-deny east-west is intact, R-NET-01).
|
|
#
|
|
# This asserts observable BEHAVIOR through the live Gateway (green Job = the intended
|
|
# verdict), not object existence (R-TEST-01).
|
|
#
|
|
# TWO FAILURE MODES ARE DISTINGUISHED (per the WS-NET pod->Gateway-VIP reachability
|
|
# finding, DECISIONS-LOG): the gateway probes classify each attempt as
|
|
# * REACHED (backend marker in the body) -> the intended verdict for the declared host;
|
|
# * DENIED (a real HTTP status, e.g. 503, but NO backend marker) -> default-deny drop
|
|
# (the intended verdict for the undeclared port);
|
|
# * NO-GATEWAY (curl code 000 / connect timeout) -> the probe's node cannot reach the
|
|
# Gateway VIP AT ALL. That is a SEPARATE platform fault (a node-local Cilium L7LB/
|
|
# envoy desync makes pods on an affected node unable to reach the Gateway VIP or
|
|
# ClusterIP), NOT default-deny. The probe SELF-SKIPS that case (exits 0 with a SKIP
|
|
# note) so this test never false-fails on the unrelated reachability bug. The
|
|
# hermetic east-west denial (below) needs no Gateway and always asserts.
|
|
#
|
|
# SELF-SKIPPING: the whole L2 layer is gated by CLUSTER_AVAILABLE in conformance/
|
|
# run.sh (skipped with no cluster). Within the test, `given` asserts the shared
|
|
# Gateway is Programmed (a missing Gateway fails fast/diagnostically), and the gateway
|
|
# probes self-skip the node-local reachability fault as above. The backend uses NO
|
|
# readinessProbe: the default-deny baseline blocks the kubelet node->pod probe, so a
|
|
# probe-bearing pod would never go Ready (DECISIONS-LOG K4); `phase: Running` is the
|
|
# correct readiness signal. Cleanup deletes both fixture namespaces in `finally`.
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: gateway-ingress-reachable
|
|
labels:
|
|
conformance.home.k3s/R-NET-01: "true"
|
|
conformance.home.k3s/R-NET-02: "true"
|
|
conformance.home.k3s/R-NET-03: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-NET-01, R-NET-02, R-NET-03"
|
|
conformance.home.k3s/layer: "L2"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
An exposed app is reachable through the shared Gateway under default-deny via an
|
|
additive gateway-ingress allow (ingress identity -> declared port only); an
|
|
undeclared port is not reachable through the Gateway and east-west stays denied.
|
|
spec:
|
|
# A generous timeout: pulling images + programming two HTTPRoutes on the Gateway.
|
|
timeouts:
|
|
apply: 60s
|
|
assert: 180s
|
|
# -- R-ENV-02: the fixture's route hostnames MUST fall under the Gateway's wildcard listener
|
|
# (*.${BASE_DOMAIN}) or they never attach — so the suffix is READ FROM THE CLUSTER UNDER TEST
|
|
# (flux-system/cluster-vars, the ConfigMap Flux substitutes ${BASE_DOMAIN} from) rather than
|
|
# authored here. Test-level => in scope for the fixture apply AND every probe step below
|
|
# (step `outputs`, by contrast, are step-scoped — see the gwvip re-capture comments).
|
|
# fixtures/app-and-probe.yaml consumes it via chainsaw resource templating.
|
|
bindings:
|
|
- name: baseDomain
|
|
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
|
|
steps:
|
|
# -- GIVEN: the exposed-app fixture (default-deny + gateway-ingress allow + backend +
|
|
# Service + declared/undeclared HTTPRoutes) and a probe namespace. Assert the shared
|
|
# Gateway is Programmed (precondition) and the backend pod is Running (listeners up).
|
|
# Each gateway-probe step below re-captures the Gateway VIP into a binding (chainsaw
|
|
# output bindings are step-scoped) so probes dial the real ingress address that
|
|
# split-horizon DNS resolves service hostnames to — env-agnostic, no hardcoded IP.
|
|
- name: given-exposed-app-under-default-deny
|
|
try:
|
|
- apply:
|
|
file: fixtures/app-and-probe.yaml
|
|
- assert:
|
|
resource:
|
|
apiVersion: gateway.networking.k8s.io/v1
|
|
kind: Gateway
|
|
metadata:
|
|
name: shared
|
|
namespace: gateway
|
|
# JMESPath binding (parenthesized key) rather than a list literal: chainsaw does
|
|
# STRICT length-equal list matching, so asserting one element against the full
|
|
# conditions[] would fail (DECISIONS-LOG K4). This pinpoints Programmed=True.
|
|
status:
|
|
(conditions[?type == 'Programmed'] | [0].status): "True"
|
|
- assert:
|
|
resource:
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: gw-backend
|
|
namespace: conformance-gw-ingress
|
|
status:
|
|
# scalar match — Running means both httpd listeners are up (no readinessProbe
|
|
# under default-deny; DECISIONS-LOG K4).
|
|
phase: Running
|
|
|
|
# -- THEN(reachable): a client in the unrestricted probe namespace dials the Gateway VIP
|
|
# for the DECLARED host. --connect-to sets SNI/Host to the hostname but dials the VIP
|
|
# (the real ingress address; -k because we test REACHABILITY, not TLS — trusted-tls
|
|
# covers the chain). The backend's marker in the body proves the Gateway (identity
|
|
# `ingress`) reached the backend THROUGH default-deny (green Job = REACHED). If the
|
|
# probe's node cannot reach the VIP at all (code 000), self-skip (green + SKIP note):
|
|
# that is the separate pod->Gateway-VIP platform fault, not default-deny.
|
|
- name: then-reachable-through-gateway
|
|
try:
|
|
# chainsaw output bindings are step-scoped, so re-capture the VIP in this step.
|
|
- command:
|
|
entrypoint: kubectl
|
|
args: [-n, gateway, get, gateway, shared, -o, "jsonpath={.status.addresses[0].value}"]
|
|
outputs:
|
|
- name: gwvip
|
|
value: ($stdout)
|
|
- apply:
|
|
resource:
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: probe-reachable
|
|
namespace: conformance-gw-probe
|
|
spec:
|
|
backoffLimit: 6
|
|
activeDeadlineSeconds: 180
|
|
template:
|
|
spec:
|
|
restartPolicy: Never
|
|
containers:
|
|
- name: probe
|
|
image: curlimages/curl:8.11.1
|
|
env:
|
|
- name: GWVIP
|
|
value: ($gwvip)
|
|
- name: BASE_DOMAIN
|
|
value: ($baseDomain)
|
|
command:
|
|
- sh
|
|
- -c
|
|
- |
|
|
set -u
|
|
host="gw-declared-conformance.${BASE_DOMAIN}"
|
|
last=000
|
|
i=0
|
|
while [ "$i" -lt 30 ]; do
|
|
code="$(curl -k -sS --max-time 8 -o /tmp/body \
|
|
-w '%{http_code}' \
|
|
--connect-to "${host}:443:${GWVIP}:443" \
|
|
"https://${host}/" 2>/dev/null || echo 000)"
|
|
last="$code"
|
|
if grep -q GW_BACKEND_REACHED /tmp/body 2>/dev/null; then
|
|
echo "REACHED: Gateway reached the backend through default-deny (code ${code})"
|
|
exit 0
|
|
fi
|
|
i=$((i+1)); sleep 5
|
|
done
|
|
if [ "$last" = "000" ]; then
|
|
echo "SKIP: this node cannot reach the Gateway VIP (code 000) —" 1>&2
|
|
echo " separate pod->Gateway-VIP platform fault, NOT default-deny" 1>&2
|
|
exit 0
|
|
fi
|
|
echo "FAIL: Gateway returned ${last} but the backend marker was absent — default-deny drop of the declared port?" 1>&2
|
|
exit 1
|
|
- assert:
|
|
resource:
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata: { name: probe-reachable, namespace: conformance-gw-probe }
|
|
status: { succeeded: 1 }
|
|
|
|
# -- THEN(undeclared port denied through the Gateway): same Gateway VIP, same backend, the
|
|
# UNDECLARED host -> port 9091. Routing is identical to the declared route (proven to
|
|
# reach the backend), so the ONLY difference is the missing allow on 9091: the Gateway
|
|
# Envoy's upstream connect is DROPPED by default-deny. The backend marker MUST be
|
|
# absent. A leak (the 9091 marker appears) fails the Job. If the node cannot reach the
|
|
# VIP at all (code 000), self-skip (same platform fault as above).
|
|
- name: then-undeclared-port-denied-through-gateway
|
|
try:
|
|
# chainsaw output bindings are step-scoped, so re-capture the VIP in this step.
|
|
- command:
|
|
entrypoint: kubectl
|
|
args: [-n, gateway, get, gateway, shared, -o, "jsonpath={.status.addresses[0].value}"]
|
|
outputs:
|
|
- name: gwvip
|
|
value: ($stdout)
|
|
- apply:
|
|
resource:
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: probe-undeclared
|
|
namespace: conformance-gw-probe
|
|
spec:
|
|
backoffLimit: 4
|
|
activeDeadlineSeconds: 150
|
|
template:
|
|
spec:
|
|
restartPolicy: Never
|
|
containers:
|
|
- name: probe
|
|
image: curlimages/curl:8.11.1
|
|
env:
|
|
- name: GWVIP
|
|
value: ($gwvip)
|
|
- name: BASE_DOMAIN
|
|
value: ($baseDomain)
|
|
command:
|
|
- sh
|
|
- -c
|
|
- |
|
|
set -u
|
|
host="gw-undeclared-conformance.${BASE_DOMAIN}"
|
|
reached_gw=no
|
|
i=0
|
|
while [ "$i" -lt 8 ]; do
|
|
code="$(curl -k -sS --max-time 8 -o /tmp/body \
|
|
-w '%{http_code}' \
|
|
--connect-to "${host}:443:${GWVIP}:443" \
|
|
"https://${host}/" 2>/dev/null || echo 000)"
|
|
if grep -q GW_BACKEND_UNDECLARED /tmp/body 2>/dev/null; then
|
|
echo "LEAK: undeclared port 9091 reachable through the Gateway (code ${code})" 1>&2
|
|
exit 1
|
|
fi
|
|
[ "$code" != "000" ] && reached_gw=yes
|
|
i=$((i+1)); sleep 5
|
|
done
|
|
if [ "$reached_gw" = "yes" ]; then
|
|
echo "DENIED: undeclared port not reachable through the Gateway (backend marker never returned)"
|
|
exit 0
|
|
fi
|
|
echo "SKIP: this node cannot reach the Gateway VIP (code 000) —" 1>&2
|
|
echo " separate pod->Gateway-VIP platform fault, NOT default-deny" 1>&2
|
|
exit 0
|
|
- assert:
|
|
resource:
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata: { name: probe-undeclared, namespace: conformance-gw-probe }
|
|
status: { succeeded: 1 }
|
|
|
|
# -- THEN(east-west still denied): HERMETIC (no Gateway needed). An in-namespace peer
|
|
# (NOT the `ingress` identity), covered by the namespace-wide default-deny, dials the
|
|
# DECLARED port east-west. The gateway-ingress allow is least-privilege (only
|
|
# fromEntities: [ingress]), so this connect MUST be dropped — default-deny east-west
|
|
# is intact (R-NET-01). The probe EXITS 0 only when the connection is BLOCKED (green
|
|
# Job = denied). This step always runs the real assertion (independent of the pod->VIP
|
|
# reachability fault above).
|
|
- name: then-east-west-still-denied
|
|
try:
|
|
- apply:
|
|
resource:
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: probe-eastwest
|
|
namespace: conformance-gw-ingress
|
|
spec:
|
|
backoffLimit: 4
|
|
activeDeadlineSeconds: 120
|
|
template:
|
|
metadata:
|
|
labels: { app: eastwest-peer }
|
|
spec:
|
|
restartPolicy: Never
|
|
containers:
|
|
- name: probe
|
|
image: busybox:1.36.1
|
|
command:
|
|
- sh
|
|
- -c
|
|
- |
|
|
set -u
|
|
# DNS resolves (baseline allows it); the TCP connect must drop.
|
|
if nc -w 5 gw-backend 8080 </dev/null 2>/dev/null; then
|
|
echo "LEAK: east-west reached the declared port under default-deny" 1>&2
|
|
exit 1
|
|
fi
|
|
echo "DENIED: east-west blocked; gateway allow is least-privilege"
|
|
exit 0
|
|
- assert:
|
|
resource:
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata: { name: probe-eastwest, namespace: conformance-gw-ingress }
|
|
status: { succeeded: 1 }
|
|
# -- Cleanup: delete both fixture namespaces (fixed names, outside chainsaw's managed
|
|
# namespace). finally accepts delete (DECISIONS-LOG: step-level finally = script/
|
|
# command/delete/wait only).
|
|
finally:
|
|
- delete:
|
|
ref:
|
|
apiVersion: v1
|
|
kind: Namespace
|
|
name: conformance-gw-ingress
|
|
- delete:
|
|
ref:
|
|
apiVersion: v1
|
|
kind: Namespace
|
|
name: conformance-gw-probe
|