jakman85andClaude Opus 5
59ffbaf11a
Six L2 behavioural tests embedded `home.keithjackson.dev`. They are test INPUTS,
not reconciled manifests, so postBuild substitution never touched them — correct
while dev held the apex, wrong the moment prod came up on
prod.home.keithjackson.dev, and blocking §9's "suite passes green against prod".
Each test now asks the cluster what it is, via a test-level chainsaw binding over
flux-system/cluster-vars — the SAME ConfigMap Flux renders the platform from, so
a test cannot disagree with the cluster it runs on:
bindings:
- name: baseDomain
value: (x_k8s_get($client,'v1','ConfigMap','flux-system','cluster-vars').data.BASE_DOMAIN)
Chosen over a flag or env var because there is nothing to forget at run time, and
test-level scope covers every step (step outputs/bindings are step-scoped).
THE APEX WAS NOT THE ONLY LEAK. dns-exposure and dns-internal-resolution also
hardcoded DEV'S Cilium LB range 192.168.1.53-.60; prod's is 192.168.2.20-.30, so
both were wrong for prod regardless of hostname. Replaced with assertions on the
OBSERVED value — the DNS answer must equal that cluster's GATEWAY_VIP, and the
public record must equal the LB IP the Service was actually assigned — which is
strictly STRONGER than the old range check as well as environment-independent.
sso-forward-auth held the literal only in prose (it talks over Service DNS).
PROVEN, NOT ASSERTED — and the positive control is the important half:
* expressions extracted verbatim from the tracked files resolve to the correct,
DIFFERENT values per cluster (dev home.keithjackson.dev / 192.168.1.53,
prod prod.home.keithjackson.dev / 192.168.2.20);
* the same assert with the HARDCODED apex passes on dev and FAILS on prod
("Invalid value: *.prod.home... Expected: *.home..."), so the harness is
provably not blind — that is exactly the drift removed;
* chainsaw lint VALID on all six; run.sh --lint clean over the whole tree.
END-TO-END L2 RUN AGAINST LIVE DEV: `dns-exposure` PASSED (42.6s) — the binding
produced conformance-dns.home.keithjackson.dev, external-dns published it, and the
probe Job asserted the record against the assigned LB IP. This is the first time
one of these tests has been executed since parameterization, and it exercises the
full path rather than just the binding.
FINDING FROM THAT RUN, worth recording: the test's teardown is ASYNCHRONOUS. It
reports PASS as soon as chainsaw deletes the namespace, but the Cloudflare record
is reaped later by external-dns's own --interval=1m sync. A zone query immediately
after PASS still showed conformance-dns.home.keithjackson.dev plus its owner=default
TXT registry entry. Harmless here, but it means a green dns-exposure does NOT prove
cleanup happened, and if external-dns were suspended the record would persist
silently. Verified separately that it clears.
conformance/incluster/rbac.yaml: its cluster-wide `get configmaps` is now
LOAD-BEARING for these bindings — narrowing it to the test namespace breaks five
tests. Commented so it is not "tidied" later.
l3-fault-injection/dr-restore: its KNOWN-GAPS block said off-host is unproven
because the MinIO shares a Proxmox host with dev's VMs. Still true as run, so the
text now says both — that the R2 move (32d6766) is what closes it, that this test's
pass is against MinIO, and that the ledger row proving R-STATE-03 PREDATES the move,
so off-host DR cannot be claimed until the drill is re-run against R2.
NOT VERIFIED, plainly: no full L2 suite run against prod, so "green against prod"
remains unproven — what is proven is that the tests now derive the CORRECT
expectation for whichever cluster they are pointed at. Whether prod SATISFIES them
(split-horizon, LE wildcard, Grafana's issuer) is untested.
VERIFIED: full CI gate PASS. flux-postbuild-substitution 260 assertions,
render-golden 10/10.
Refs R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04, R-ENV-02, R-DNS-01/02/03.
Co-Authored-By: Claude Opus 5 <[email protected]>
115 lines
6.0 KiB
YAML
115 lines
6.0 KiB
YAML
# =============================================================================
|
|
# L2 BEHAVIORAL — split-horizon INTERNAL resolution (R-DNS-01 / R-DNS-03) [RUNNABLE]
|
|
# Catalog: docs/14 "DNS/exposure (R-DNS-*)"
|
|
# Spec: docs/09 "Split-horizon resolution (R-DNS-03)" — internal clients resolve
|
|
# <svc>.${BASE_DOMAIN} to the INTERNAL Gateway VIP, not the public hairpin.
|
|
# Impl: platform/coredns/coredns-custom.yaml (k3s CoreDNS `import *.server`; template plugin
|
|
# answers the whole ${BASE_DOMAIN} zone with ${GATEWAY_VIP}).
|
|
#
|
|
# ENVIRONMENT-AGNOSTIC (R-ENV-02): this test asks the CLUSTER IT IS POINTED AT for its own
|
|
# ${BASE_DOMAIN} and ${GATEWAY_VIP} — the same flux-system/cluster-vars ConfigMap Flux
|
|
# substitutes from — via test-level chainsaw bindings (see `spec.bindings`). Nothing here is
|
|
# per-environment and no flag is passed at run time, so the suite cannot be run against the
|
|
# wrong expectation: dev asserts dev's zone/VIP, prod asserts prod's.
|
|
#
|
|
# GIVEN the split-horizon coredns-custom ConfigMap is reconciled into k3s CoreDNS,
|
|
# WHEN an in-cluster pod resolves a platform hostname over its PUBLIC name
|
|
# (keycloak.${BASE_DOMAIN}) using the default cluster resolver (CoreDNS 10.43.0.10),
|
|
# THEN the answer is EXACTLY that cluster's ${GATEWAY_VIP} (the address coredns-custom is
|
|
# templated with, and the `shared` Gateway's own status address) — NOT a public/off-cluster
|
|
# IP. This is what lets pod->public-hostname traffic hairpin through the
|
|
# Gateway with the correct SNI (*.${BASE_DOMAIN}) so the wildcard cert validates
|
|
# (R-DNS-01 "services reachable at <svc>.${BASE_DOMAIN}"; R-DNS-03 split-horizon).
|
|
#
|
|
# Complements dns-exposure (R-DNS-02, the EXTERNAL/public-record side). Together they cover the
|
|
# split-horizon pair: external answer = public record (R-DNS-02); internal answer = Gateway VIP
|
|
# (R-DNS-01/03).
|
|
#
|
|
# CLEANUP: chainsaw deletes the per-test namespace (Job included) at teardown. Read-only otherwise.
|
|
# chainsaw v0.2.12: Job + status assert; no test-level finally.
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: dns-internal-resolution
|
|
labels:
|
|
conformance.home.k3s/R-DNS-01: "true"
|
|
conformance.home.k3s/R-DNS-03: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-DNS-01,R-DNS-03"
|
|
conformance.home.k3s/layer: "L2"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
An in-cluster pod resolving keycloak.${BASE_DOMAIN} via cluster DNS gets that cluster's
|
|
internal Gateway VIP, not the public hairpin — split-horizon in-cluster resolution.
|
|
spec:
|
|
timeouts:
|
|
apply: 60s
|
|
assert: 2m
|
|
exec: 2m
|
|
# -- R-ENV-02: the environment's identity is READ FROM THE CLUSTER UNDER TEST, never authored
|
|
# here. flux-system/cluster-vars is the ConfigMap Flux itself substitutes from
|
|
# (clusters/base/cluster-vars.yaml + the dev/prod overlay patch), so these bindings are the
|
|
# same source of truth the platform is rendered from. Test-level => in scope for every step
|
|
# (unlike step `outputs`, which are step-scoped).
|
|
bindings:
|
|
- name: baseDomain
|
|
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
|
|
- name: gatewayVip
|
|
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.GATEWAY_VIP)
|
|
steps:
|
|
- name: then-cluster-dns-answers-with-the-internal-gateway-vip
|
|
try:
|
|
- apply:
|
|
resource:
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: dns-internal-probe
|
|
spec:
|
|
backoffLimit: 2
|
|
activeDeadlineSeconds: 100
|
|
template:
|
|
spec:
|
|
restartPolicy: Never
|
|
containers:
|
|
- name: probe
|
|
image: busybox:1.36.1
|
|
# The cluster's OWN identity, resolved from flux-system/cluster-vars by the
|
|
# test-level bindings — never a literal (R-ENV-02).
|
|
env:
|
|
- name: BASE_DOMAIN
|
|
value: ($baseDomain)
|
|
- name: GATEWAY_VIP
|
|
value: ($gatewayVip)
|
|
command:
|
|
- sh
|
|
- -c
|
|
- |
|
|
set -u
|
|
host="keycloak.${BASE_DOMAIN}"
|
|
# Resolve via the DEFAULT cluster resolver (CoreDNS, /etc/resolv.conf ->
|
|
# 10.43.0.10). Split-horizon must answer with THIS cluster's Gateway VIP
|
|
# (the address coredns-custom is templated with), proving pods hairpin
|
|
# through the Gateway. Exact match on ${GATEWAY_VIP}: stricter than the
|
|
# old hardcoded 192.168.1.53-.60 range AND correct in both environments.
|
|
out="$(nslookup "$host" 2>/dev/null)"
|
|
echo "$out"
|
|
ip="$(echo "$out" | awk '/^Address: /{print $2}' | grep -Fx "${GATEWAY_VIP}" | head -1)"
|
|
if [ -n "$ip" ]; then
|
|
echo "INTERNAL_RESOLUTION_OK $host -> $ip (in-cluster Gateway VIP)"
|
|
exit 0
|
|
fi
|
|
echo "FAIL: $host did not resolve to this cluster's Gateway VIP (${GATEWAY_VIP});"
|
|
echo " split-horizon coredns-custom not in effect (got a public/off-cluster answer)."
|
|
exit 1
|
|
- assert:
|
|
resource:
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: dns-internal-probe
|
|
status:
|
|
succeeded: 1
|