From 59ffbaf11a4ea9c36a910ac2c2df6a296943178a Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 5 Sep 2026 12:05:05 -0500 Subject: [PATCH] feat(conformance): L2 tests derive their environment from the cluster, not a literal MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Six L2 behavioural tests embedded `home.keithjackson.dev`. They are test INPUTS, not reconciled manifests, so postBuild substitution never touched them — correct while dev held the apex, wrong the moment prod came up on prod.home.keithjackson.dev, and blocking §9's "suite passes green against prod". Each test now asks the cluster what it is, via a test-level chainsaw binding over flux-system/cluster-vars — the SAME ConfigMap Flux renders the platform from, so a test cannot disagree with the cluster it runs on: bindings: - name: baseDomain value: (x_k8s_get($client,'v1','ConfigMap','flux-system','cluster-vars').data.BASE_DOMAIN) Chosen over a flag or env var because there is nothing to forget at run time, and test-level scope covers every step (step outputs/bindings are step-scoped). THE APEX WAS NOT THE ONLY LEAK. dns-exposure and dns-internal-resolution also hardcoded DEV'S Cilium LB range 192.168.1.53-.60; prod's is 192.168.2.20-.30, so both were wrong for prod regardless of hostname. Replaced with assertions on the OBSERVED value — the DNS answer must equal that cluster's GATEWAY_VIP, and the public record must equal the LB IP the Service was actually assigned — which is strictly STRONGER than the old range check as well as environment-independent. sso-forward-auth held the literal only in prose (it talks over Service DNS). PROVEN, NOT ASSERTED — and the positive control is the important half: * expressions extracted verbatim from the tracked files resolve to the correct, DIFFERENT values per cluster (dev home.keithjackson.dev / 192.168.1.53, prod prod.home.keithjackson.dev / 192.168.2.20); * the same assert with the HARDCODED apex passes on dev and FAILS on prod ("Invalid value: *.prod.home... Expected: *.home..."), so the harness is provably not blind — that is exactly the drift removed; * chainsaw lint VALID on all six; run.sh --lint clean over the whole tree. END-TO-END L2 RUN AGAINST LIVE DEV: `dns-exposure` PASSED (42.6s) — the binding produced conformance-dns.home.keithjackson.dev, external-dns published it, and the probe Job asserted the record against the assigned LB IP. This is the first time one of these tests has been executed since parameterization, and it exercises the full path rather than just the binding. FINDING FROM THAT RUN, worth recording: the test's teardown is ASYNCHRONOUS. It reports PASS as soon as chainsaw deletes the namespace, but the Cloudflare record is reaped later by external-dns's own --interval=1m sync. A zone query immediately after PASS still showed conformance-dns.home.keithjackson.dev plus its owner=default TXT registry entry. Harmless here, but it means a green dns-exposure does NOT prove cleanup happened, and if external-dns were suspended the record would persist silently. Verified separately that it clears. conformance/incluster/rbac.yaml: its cluster-wide `get configmaps` is now LOAD-BEARING for these bindings — narrowing it to the test namespace breaks five tests. Commented so it is not "tidied" later. l3-fault-injection/dr-restore: its KNOWN-GAPS block said off-host is unproven because the MinIO shares a Proxmox host with dev's VMs. Still true as run, so the text now says both — that the R2 move (32d6766) is what closes it, that this test's pass is against MinIO, and that the ledger row proving R-STATE-03 PREDATES the move, so off-host DR cannot be claimed until the drill is re-run against R2. NOT VERIFIED, plainly: no full L2 suite run against prod, so "green against prod" remains unproven — what is proven is that the tests now derive the CORRECT expectation for whichever cluster they are pointed at. Whether prod SATISFIES them (split-horizon, LE wildcard, Grafana's issuer) is untested. VERIFIED: full CI gate PASS. flux-postbuild-substitution 260 assertions, render-golden 10/10. Refs R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04, R-ENV-02, R-DNS-01/02/03. Co-Authored-By: Claude Opus 5 --- conformance/incluster/rbac.yaml | 7 +++ conformance/tests/README.md | 21 +++++++ .../dns-exposure/chainsaw-test.yaml | 58 ++++++++++++++++--- .../chainsaw-test.yaml | 54 ++++++++++++----- .../chainsaw-test.yaml | 17 +++++- .../fixtures/app-and-probe.yaml | 13 ++++- .../oidc-sso-grafana/chainsaw-test.yaml | 24 ++++++-- .../sso-forward-auth/chainsaw-test.yaml | 7 ++- .../trusted-tls/chainsaw-test.yaml | 19 +++++- .../dr-restore/chainsaw-test.yaml | 15 +++-- 10 files changed, 196 insertions(+), 39 deletions(-) diff --git a/conformance/incluster/rbac.yaml b/conformance/incluster/rbac.yaml index 36b7b79..72150cf 100644 --- a/conformance/incluster/rbac.yaml +++ b/conformance/incluster/rbac.yaml @@ -57,6 +57,13 @@ rules: verbs: [create, delete, get, list, watch, patch] - apiGroups: [""] resources: [pods, services, configmaps, secrets, persistentvolumeclaims] + # LOAD-BEARING `get` ON CONFIGMAPS, BEYOND FIXTURE CREATION (R-ENV-02): the environment- + # agnostic L2 tests (dns-exposure, dns-internal-resolution, gateway-ingress-reachable, + # oidc-sso-grafana, trusted-tls) resolve the cluster's own BASE_DOMAIN / GATEWAY_VIP at run + # time from `flux-system/cluster-vars` — the same ConfigMap Flux substitutes from — via + # chainsaw `x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars')` bindings. + # If this rule is ever narrowed to the test's own namespace, those bindings fail and the + # tests error out. Narrow it only alongside a namespace-scoped read of that ConfigMap. verbs: [create, delete, get, list, watch, patch] - apiGroups: [""] resources: [pods/exec] diff --git a/conformance/tests/README.md b/conformance/tests/README.md index 8fcf3a7..71850b7 100644 --- a/conformance/tests/README.md +++ b/conformance/tests/README.md @@ -93,6 +93,27 @@ and coverage extraction exact and robust. break the thing and assert fail-closed / no-data-loss, with `finally` restoring state. - **Never weaken a test to go green (R-TEST-06).** If the spec can't be met, escalate to the PM — do not relax the assertion or the requirement. +- **Never write a per-environment value into a test (R-ENV-02).** No hostname suffix, no + Gateway VIP, no LB-pool range. One authored suite must run green against dev *and* prod, + so a test **asks the cluster it is pointed at** what it is. The mechanism is a chainsaw + binding over the same ConfigMap Flux substitutes from: + + ```yaml + spec: + bindings: + - name: baseDomain + value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN) + ``` + + Declare it at **test level** (step `outputs`/`bindings` are step-scoped), then consume it: + `($baseDomain)` for a pod `env:` value, `(join('.', ['myhost', $baseDomain]))` where a + full hostname is needed — including inside a `fixtures/*.yaml` applied with `apply: file:`, + which chainsaw templates with the same bindings. Prefer reading the *actual* observed value + over restating a range (e.g. assert the DNS answer equals `GATEWAY_VIP`, or equals the LB IP + the Service was assigned) — it is both stricter and environment-independent. Never take the + environment from a CLI flag or an exported variable: a flag someone forgets is how a suite + ends up silently asserting the other cluster's expectations. L1 goldens are the one + exception: they are rendering fixtures and carry the literal `${BASE_DOMAIN}` token. Scaffolds in this tree carry `TODO(T-x.y)` markers where a fixture/assertion lands as a later roadmap phase delivers the capability under test. `lib/status.py` treats such a diff --git a/conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml b/conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml index 9b3eee0..244d542 100644 --- a/conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml +++ b/conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml @@ -3,11 +3,18 @@ # Catalog: docs/14 "DNS/exposure (R-DNS-*)" # Spec: docs/09 (ExternalDNS auto-manages records; split-horizon foundation) # +# ENVIRONMENT-AGNOSTIC (R-ENV-02): the hostname is built from the ${BASE_DOMAIN} of the CLUSTER +# UNDER TEST (read from flux-system/cluster-vars via a test-level chainsaw binding), and the +# expected answer is the LB IP that THIS Service was actually assigned (read back from its own +# status) — so neither the name nor the address is authored per environment, and no run-time flag +# can be forgotten. Requesting a record under the cluster's own suffix is also what keeps the two +# environments from ever describing the same public name (the 2026-09-04 incident precondition). +# # GIVEN a LoadBalancer Service annotated with an external-dns hostname, # WHEN ExternalDNS reconciles (source=service, provider=cloudflare, policy=sync), # THEN a PUBLIC DNS record for that hostname is auto-created, pointing at the -# Service's Cilium-assigned LB IP (R-DNS-02). The record resolves to the -# cluster LB IP (Cilium L2 `shared` pool 192.168.1.53-.60), i.e. clients reach the cluster endpoint. +# Service's Cilium-assigned LB IP (R-DNS-02) — asserted as an EXACT match against that +# Service's own `status.loadBalancer.ingress[0].ip`, i.e. clients reach the cluster endpoint. # # CLEANUP: chainsaw deletes the per-test namespace (Service included) at teardown; # ExternalDNS policy=sync then REMOVES the record from Cloudflare on its next pass. @@ -38,6 +45,13 @@ spec: apply: 90s assert: 5m exec: 6m + # -- R-ENV-02: the environment's hostname suffix is READ FROM THE CLUSTER UNDER TEST. + # flux-system/cluster-vars is the ConfigMap Flux substitutes ${BASE_DOMAIN} from, so this + # binding is the same source of truth the platform is rendered from — dev asks dev, prod + # asks prod, and there is no per-environment fork and no flag to forget. + bindings: + - name: baseDomain + value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN) steps: - name: given-a-service-requesting-a-dns-name try: @@ -48,7 +62,8 @@ spec: metadata: name: dns-probe-svc annotations: - external-dns.alpha.kubernetes.io/hostname: conformance-dns.home.keithjackson.dev + # conformance-dns. — never a literal apex. + external-dns.alpha.kubernetes.io/hostname: (join('.', ['conformance-dns', $baseDomain])) spec: type: LoadBalancer # No real backend needed — Cilium LB-IPAM assigns a VIP regardless, and @@ -59,7 +74,27 @@ spec: - name: http port: 80 targetPort: 80 + # Wait for Cilium LB-IPAM to actually assign the VIP: it is the value ExternalDNS will + # publish and the value the probe below asserts EXACTLY. `|| ` + empty-array literal + # keeps the expression nil-safe while the address is still absent, so the assert POLLS + # (fails and retries) instead of erroring out on the first evaluation. + - assert: + resource: + apiVersion: v1 + kind: Service + metadata: + name: dns-probe-svc + status: + (length(loadBalancer.ingress || `[]`)): 1 - name: then-externaldns-creates-the-public-record + # Step-scoped binding: read back the LB IP THIS Service was assigned (the previous step's + # assert guarantees it exists by now — step bindings are evaluated when the step runs). + # Expecting that exact address is both stricter than the old hardcoded 192.168.1.53-.60 + # range and correct in every environment: dev's pool is 192.168.1.53-.60, prod's is + # 192.168.2.20-.30, and neither is written down here. + bindings: + - name: lbIp + value: (x_k8s_get($client, 'v1', 'Service', $namespace, 'dns-probe-svc').status.loadBalancer.ingress[0].ip) try: - apply: resource: @@ -76,20 +111,27 @@ spec: containers: - name: probe image: busybox:1.36.1 + # Both values come from the cluster under test: the suffix from + # flux-system/cluster-vars, the address from this Service's own status. + env: + - name: BASE_DOMAIN + value: ($baseDomain) + - name: LB_IP + value: ($lbIp) command: - sh - -c - | set -u - host="conformance-dns.home.keithjackson.dev" + host="conformance-dns.${BASE_DOMAIN}" # Poll PUBLIC DNS (1.1.1.1) until the auto-created A record appears and - # points into the cluster LB range (Cilium L2 `shared` pool 192.168.1.53-.60). Proves ExternalDNS created it. + # points at THIS Service's assigned LB IP. Proves ExternalDNS created it. for i in $(seq 1 28); do - ip="$(nslookup "$host" 1.1.1.1 2>/dev/null | grep -oE '192\.168\.1\.(5[3-9]|60)' | head -1)" + ip="$(nslookup "$host" 1.1.1.1 2>/dev/null | awk '/^Address: /{print $2}' | grep -Fx "${LB_IP}" | head -1)" if [ -n "$ip" ]; then echo "RECORD_FOUND $host -> $ip"; exit 0; fi - echo "poll $i: record not yet published"; sleep 10 + echo "poll $i: record not yet published as ${LB_IP}"; sleep 10 done - echo "FAIL: ExternalDNS did not publish the record within timeout"; exit 1 + echo "FAIL: ExternalDNS did not publish ${host} -> ${LB_IP} within timeout"; exit 1 - assert: resource: apiVersion: batch/v1 diff --git a/conformance/tests/l2-behavioral/dns-internal-resolution/chainsaw-test.yaml b/conformance/tests/l2-behavioral/dns-internal-resolution/chainsaw-test.yaml index ee68a36..ecb2e1a 100644 --- a/conformance/tests/l2-behavioral/dns-internal-resolution/chainsaw-test.yaml +++ b/conformance/tests/l2-behavioral/dns-internal-resolution/chainsaw-test.yaml @@ -2,17 +2,24 @@ # L2 BEHAVIORAL — split-horizon INTERNAL resolution (R-DNS-01 / R-DNS-03) [RUNNABLE] # Catalog: docs/14 "DNS/exposure (R-DNS-*)" # Spec: docs/09 "Split-horizon resolution (R-DNS-03)" — internal clients resolve -# .home.keithjackson.dev to the INTERNAL Gateway VIP, not the public hairpin. +# .${BASE_DOMAIN} to the INTERNAL Gateway VIP, not the public hairpin. # Impl: platform/coredns/coredns-custom.yaml (k3s CoreDNS `import *.server`; template plugin -# answers the whole home.keithjackson.dev zone with ${GATEWAY_VIP}). +# answers the whole ${BASE_DOMAIN} zone with ${GATEWAY_VIP}). +# +# ENVIRONMENT-AGNOSTIC (R-ENV-02): this test asks the CLUSTER IT IS POINTED AT for its own +# ${BASE_DOMAIN} and ${GATEWAY_VIP} — the same flux-system/cluster-vars ConfigMap Flux +# substitutes from — via test-level chainsaw bindings (see `spec.bindings`). Nothing here is +# per-environment and no flag is passed at run time, so the suite cannot be run against the +# wrong expectation: dev asserts dev's zone/VIP, prod asserts prod's. # # GIVEN the split-horizon coredns-custom ConfigMap is reconciled into k3s CoreDNS, # WHEN an in-cluster pod resolves a platform hostname over its PUBLIC name -# (keycloak.home.keithjackson.dev) using the default cluster resolver (CoreDNS 10.43.0.10), -# THEN the answer is the in-cluster Gateway VIP (the Cilium L2 `shared` gateway pool 192.168.1.53-.60) — NOT a -# public/off-cluster IP. This is what lets pod->public-hostname traffic hairpin through the -# Gateway with the correct SNI (*.home.keithjackson.dev) so the wildcard cert validates -# (R-DNS-01 "services reachable at .home..."; R-DNS-03 split-horizon). +# (keycloak.${BASE_DOMAIN}) using the default cluster resolver (CoreDNS 10.43.0.10), +# THEN the answer is EXACTLY that cluster's ${GATEWAY_VIP} (the address coredns-custom is +# templated with, and the `shared` Gateway's own status address) — NOT a public/off-cluster +# IP. This is what lets pod->public-hostname traffic hairpin through the +# Gateway with the correct SNI (*.${BASE_DOMAIN}) so the wildcard cert validates +# (R-DNS-01 "services reachable at .${BASE_DOMAIN}"; R-DNS-03 split-horizon). # # Complements dns-exposure (R-DNS-02, the EXTERNAL/public-record side). Together they cover the # split-horizon pair: external answer = public record (R-DNS-02); internal answer = Gateway VIP @@ -34,13 +41,23 @@ metadata: conformance.home.k3s/safety-critical: "false" conformance.home.k3s/executable: "true" conformance.home.k3s/intent: >- - An in-cluster pod resolving keycloak.home.keithjackson.dev via cluster DNS gets the internal - Gateway VIP (192.168.1.53-.60), not the public hairpin — split-horizon in-cluster resolution. + An in-cluster pod resolving keycloak.${BASE_DOMAIN} via cluster DNS gets that cluster's + internal Gateway VIP, not the public hairpin — split-horizon in-cluster resolution. spec: timeouts: apply: 60s assert: 2m exec: 2m + # -- R-ENV-02: the environment's identity is READ FROM THE CLUSTER UNDER TEST, never authored + # here. flux-system/cluster-vars is the ConfigMap Flux itself substitutes from + # (clusters/base/cluster-vars.yaml + the dev/prod overlay patch), so these bindings are the + # same source of truth the platform is rendered from. Test-level => in scope for every step + # (unlike step `outputs`, which are step-scoped). + bindings: + - name: baseDomain + value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN) + - name: gatewayVip + value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.GATEWAY_VIP) steps: - name: then-cluster-dns-answers-with-the-internal-gateway-vip try: @@ -59,23 +76,32 @@ spec: containers: - name: probe image: busybox:1.36.1 + # The cluster's OWN identity, resolved from flux-system/cluster-vars by the + # test-level bindings — never a literal (R-ENV-02). + env: + - name: BASE_DOMAIN + value: ($baseDomain) + - name: GATEWAY_VIP + value: ($gatewayVip) command: - sh - -c - | set -u - host="keycloak.home.keithjackson.dev" + host="keycloak.${BASE_DOMAIN}" # Resolve via the DEFAULT cluster resolver (CoreDNS, /etc/resolv.conf -> - # 10.43.0.10). Split-horizon must answer with the in-cluster Gateway VIP - # (Cilium L2 `shared` pool 192.168.1.53-.60), proving pods hairpin through the Gateway. + # 10.43.0.10). Split-horizon must answer with THIS cluster's Gateway VIP + # (the address coredns-custom is templated with), proving pods hairpin + # through the Gateway. Exact match on ${GATEWAY_VIP}: stricter than the + # old hardcoded 192.168.1.53-.60 range AND correct in both environments. out="$(nslookup "$host" 2>/dev/null)" echo "$out" - ip="$(echo "$out" | awk '/^Address: /{print $2}' | grep -E '^192\.168\.1\.(5[3-9]|60)$' | head -1)" + ip="$(echo "$out" | awk '/^Address: /{print $2}' | grep -Fx "${GATEWAY_VIP}" | head -1)" if [ -n "$ip" ]; then echo "INTERNAL_RESOLUTION_OK $host -> $ip (in-cluster Gateway VIP)" exit 0 fi - echo "FAIL: $host did not resolve to the in-cluster Gateway VIP (192.168.1.53-.60);" + echo "FAIL: $host did not resolve to this cluster's Gateway VIP (${GATEWAY_VIP});" echo " split-horizon coredns-custom not in effect (got a public/off-cluster answer)." exit 1 - assert: diff --git a/conformance/tests/l2-behavioral/gateway-ingress-reachable/chainsaw-test.yaml b/conformance/tests/l2-behavioral/gateway-ingress-reachable/chainsaw-test.yaml index 151aa03..32301b4 100644 --- a/conformance/tests/l2-behavioral/gateway-ingress-reachable/chainsaw-test.yaml +++ b/conformance/tests/l2-behavioral/gateway-ingress-reachable/chainsaw-test.yaml @@ -68,6 +68,15 @@ spec: timeouts: apply: 60s assert: 180s + # -- R-ENV-02: the fixture's route hostnames MUST fall under the Gateway's wildcard listener + # (*.${BASE_DOMAIN}) or they never attach — so the suffix is READ FROM THE CLUSTER UNDER TEST + # (flux-system/cluster-vars, the ConfigMap Flux substitutes ${BASE_DOMAIN} from) rather than + # authored here. Test-level => in scope for the fixture apply AND every probe step below + # (step `outputs`, by contrast, are step-scoped — see the gwvip re-capture comments). + # fixtures/app-and-probe.yaml consumes it via chainsaw resource templating. + bindings: + - name: baseDomain + value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN) steps: # -- GIVEN: the exposed-app fixture (default-deny + gateway-ingress allow + backend + # Service + declared/undeclared HTTPRoutes) and a probe namespace. Assert the shared @@ -139,12 +148,14 @@ spec: env: - name: GWVIP value: ($gwvip) + - name: BASE_DOMAIN + value: ($baseDomain) command: - sh - -c - | set -u - host="gw-declared-conformance.home.keithjackson.dev" + host="gw-declared-conformance.${BASE_DOMAIN}" last=000 i=0 while [ "$i" -lt 30 ]; do @@ -207,12 +218,14 @@ spec: env: - name: GWVIP value: ($gwvip) + - name: BASE_DOMAIN + value: ($baseDomain) command: - sh - -c - | set -u - host="gw-undeclared-conformance.home.keithjackson.dev" + host="gw-undeclared-conformance.${BASE_DOMAIN}" reached_gw=no i=0 while [ "$i" -lt 8 ]; do diff --git a/conformance/tests/l2-behavioral/gateway-ingress-reachable/fixtures/app-and-probe.yaml b/conformance/tests/l2-behavioral/gateway-ingress-reachable/fixtures/app-and-probe.yaml index 02f83a8..01afc0b 100644 --- a/conformance/tests/l2-behavioral/gateway-ingress-reachable/fixtures/app-and-probe.yaml +++ b/conformance/tests/l2-behavioral/gateway-ingress-reachable/fixtures/app-and-probe.yaml @@ -10,6 +10,13 @@ # The namespace opts into the shared Gateway (gateway-access: shared) exactly like a # rendered app namespace, so its HTTPRoutes attach to the real platform Gateway and the # probe exercises the true north-south path (Gateway Envoy -> backend), not a stand-in. +# +# ENVIRONMENT-AGNOSTIC HOSTNAMES (R-ENV-02): the two HTTPRoute hostnames are NOT literals. They +# are chainsaw-templated from the `baseDomain` binding declared in ../chainsaw-test.yaml, which +# reads BASE_DOMAIN out of the flux-system/cluster-vars ConfigMap of the cluster under test — the +# same ConfigMap Flux substitutes ${BASE_DOMAIN} from. They MUST fall under the shared Gateway's +# wildcard listener (*.${BASE_DOMAIN}) or the routes never attach, which is precisely why a +# hardcoded suffix silently broke this test on any cluster that did not hold the apex. --- apiVersion: v1 kind: Namespace @@ -134,7 +141,8 @@ spec: namespace: gateway sectionName: https hostnames: - - gw-declared-conformance.home.keithjackson.dev + # gw-declared-conformance. — see the header. + - (join('.', ['gw-declared-conformance', $baseDomain])) rules: - matches: - path: @@ -158,7 +166,8 @@ spec: namespace: gateway sectionName: https hostnames: - - gw-undeclared-conformance.home.keithjackson.dev + # gw-undeclared-conformance. — see the header. + - (join('.', ['gw-undeclared-conformance', $baseDomain])) rules: - matches: - path: diff --git a/conformance/tests/l2-behavioral/oidc-sso-grafana/chainsaw-test.yaml b/conformance/tests/l2-behavioral/oidc-sso-grafana/chainsaw-test.yaml index 6fb93c1..4ef93fd 100644 --- a/conformance/tests/l2-behavioral/oidc-sso-grafana/chainsaw-test.yaml +++ b/conformance/tests/l2-behavioral/oidc-sso-grafana/chainsaw-test.yaml @@ -3,10 +3,15 @@ # Spec: docs/15 identity-and-access; ADR-0014. Impl: Grafana as a Keycloak OIDC client # (platform/monitoring/kube-prometheus-stack.yaml auth.generic_oauth) — the self-contained SSO proof. # +# ENVIRONMENT-AGNOSTIC (R-ENV-02): the expected IdP host is built from the ${BASE_DOMAIN} of the +# CLUSTER UNDER TEST, read from flux-system/cluster-vars (the ConfigMap Flux substitutes from) by +# a test-level chainsaw binding — so this asserts each cluster redirects to ITS OWN Keycloak, which +# is exactly what would catch a cross-environment OIDC issuer left pointing at the other cluster. +# # GIVEN Grafana is deployed and configured as an OIDC client of the Keycloak `homelab` realm, # WHEN an unauthenticated browser starts the OIDC login (GET /login/generic_oauth) # -> THEN Grafana redirects (302) to the CENTRAL IdP's authorize endpoint on -# keycloak.home.keithjackson.dev/realms/homelab — proving SSO is driven by the one IdP +# keycloak.${BASE_DOMAIN}/realms/homelab — proving SSO is driven by the one IdP # (R-IDP-01: central Identity Provider as the single user directory). # # Goes GREEN once the grafana-oidc client-secret hand-off is filled and Keycloak + Grafana are up. @@ -30,6 +35,11 @@ spec: apply: 60s assert: 3m exec: 3m + # -- R-ENV-02: the IdP hostname suffix is READ FROM THE CLUSTER UNDER TEST + # (flux-system/cluster-vars — the ConfigMap Flux substitutes ${BASE_DOMAIN} from). + bindings: + - name: baseDomain + value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN) steps: - name: given-grafana-is-an-oidc-client try: @@ -59,6 +69,9 @@ spec: containers: - name: probe image: busybox:1.36.1 + env: + - name: BASE_DOMAIN + value: ($baseDomain) command: - sh - -c @@ -68,13 +81,14 @@ spec: # busybox wget has no --max-redirect; -S dumps every response header # (incl. the initial 302 Location to the IdP) before following, so the # grep below still matches the Keycloak authorize URL. (Grafana verified - # live: 302 -> keycloak.home.keithjackson.dev/realms/homelab/.../auth.) + # live: 302 -> keycloak.${BASE_DOMAIN}/realms/homelab/.../auth.) + idp="keycloak.${BASE_DOMAIN}/realms/homelab" out="$(wget -S -O /dev/null "$svc/login/generic_oauth" 2>&1 || true)" echo "$out" - if echo "$out" | grep -qi 'keycloak.home.keithjackson.dev/realms/homelab'; then - echo GRAFANA_OIDC_REDIRECT_OK + if echo "$out" | grep -qiF "$idp"; then + echo "GRAFANA_OIDC_REDIRECT_OK -> ${idp}" else - echo "FAIL: Grafana did not redirect to the Keycloak homelab realm"; exit 1 + echo "FAIL: Grafana did not redirect to ${idp} (this cluster's Keycloak homelab realm)"; exit 1 fi - assert: resource: diff --git a/conformance/tests/l2-behavioral/sso-forward-auth/chainsaw-test.yaml b/conformance/tests/l2-behavioral/sso-forward-auth/chainsaw-test.yaml index 79d6ee2..24a72ad 100644 --- a/conformance/tests/l2-behavioral/sso-forward-auth/chainsaw-test.yaml +++ b/conformance/tests/l2-behavioral/sso-forward-auth/chainsaw-test.yaml @@ -3,8 +3,13 @@ # Catalog: docs/14 onboarding (auth wiring) + docs/15 identity-and-access. # Impl: platform/oauth2-proxy (forward-auth against the Keycloak `homelab` realm), ADR-0014. # +# ENVIRONMENT-AGNOSTIC (R-ENV-02): this test names NO hostname. It talks to oauth2-proxy over its +# in-cluster Service DNS and asserts HTTP behaviour, so the same authored test is correct on dev and +# on prod. The issuer host below is written as ${BASE_DOMAIN} because that is what the platform +# substitutes; it is prose, not an input — nothing here needs parameterizing. +# # GIVEN oauth2-proxy is deployed and has validated the Keycloak OIDC issuer (it only becomes Available -# after successfully discovering https://keycloak.home.keithjackson.dev/realms/homelab — this is +# after successfully discovering https://keycloak.${BASE_DOMAIN}/realms/homelab — this is # the proxy->Keycloak flow working), # WHEN an UNAUTHENTICATED request hits the forward-auth decision endpoint (/oauth2/auth) # -> THEN it is challenged: HTTP 401 (no valid Keycloak session => access denied), and a normal diff --git a/conformance/tests/l2-behavioral/trusted-tls/chainsaw-test.yaml b/conformance/tests/l2-behavioral/trusted-tls/chainsaw-test.yaml index d6c7f71..2cc6485 100644 --- a/conformance/tests/l2-behavioral/trusted-tls/chainsaw-test.yaml +++ b/conformance/tests/l2-behavioral/trusted-tls/chainsaw-test.yaml @@ -3,8 +3,13 @@ # Catalog: docs/14 "Trusted TLS (R-TLS-01/02)" # Spec: docs/09 (Gateway API + DNS-01 wildcard), ADR-0006 (LE issuers) # +# ENVIRONMENT-AGNOSTIC (R-ENV-02): the hostnames are built from the ${BASE_DOMAIN} of the CLUSTER +# UNDER TEST, read from flux-system/cluster-vars (the ConfigMap Flux substitutes from) by a +# test-level chainsaw binding. The same authored test therefore verifies dev's wildcard on dev and +# prod's on prod, with nothing to pass at run time. +# # GIVEN the shared Gateway terminating TLS with the cert-manager wildcard cert, -# WHEN an in-cluster probe dials it by hostname (SNI *.home.keithjackson.dev) and +# WHEN an in-cluster probe dials it by hostname (SNI *.${BASE_DOMAIN}) and # verifies the chain against the SYSTEM trust store, # THEN the served cert is non-self-signed, unexpired, covers the wildcard SAN, and # chains to a PUBLICLY-TRUSTED CA root (R-TLS-01). The wildcard SAN also covers @@ -36,6 +41,11 @@ metadata: that chains to a publicly-trusted CA root (LE production); the wildcard SAN covers internal-only hostnames too. spec: + # -- R-ENV-02: the hostname suffix is READ FROM THE CLUSTER UNDER TEST (flux-system/cluster-vars + # — the ConfigMap Flux substitutes ${BASE_DOMAIN} from), never authored per environment. + bindings: + - name: baseDomain + value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN) steps: - name: then-gateway-cert-chains-to-trusted-root try: @@ -54,13 +64,16 @@ spec: containers: - name: probe image: curlimages/curl:8.11.1 + env: + - name: BASE_DOMAIN + value: ($baseDomain) command: - sh - -c - | set -eu # Probe TWO wildcard hostnames — one "public", one "internal-only". Both - # match *.home.keithjackson.dev, so a clean verify against the SYSTEM CA + # match *.${BASE_DOMAIN}, so a clean verify against the SYSTEM CA # bundle for BOTH proves R-TLS-01 (chains to a publicly-trusted root) AND # R-TLS-03 (the same trusted wildcard covers internal-only names). Names # resolve via in-cluster split-horizon DNS to the Gateway VIP. curl (no @@ -68,7 +81,7 @@ spec: # the gateway's 404 once TLS+hostname verify; it exits 60 if the chain or # hostname fails. TLS terminates at the gateway before HTTP routing, so no # backend is needed. - for host in conformance.home.keithjackson.dev internal-only.home.keithjackson.dev; do + for host in "conformance.${BASE_DOMAIN}" "internal-only.${BASE_DOMAIN}"; do curl -sS --retry 5 --retry-connrefused --retry-delay 3 --max-time 20 \ -o /dev/null "https://${host}/" echo "TLS_TRUSTED ${host}" diff --git a/conformance/tests/l3-fault-injection/dr-restore/chainsaw-test.yaml b/conformance/tests/l3-fault-injection/dr-restore/chainsaw-test.yaml index 815ccce..76b205d 100644 --- a/conformance/tests/l3-fault-injection/dr-restore/chainsaw-test.yaml +++ b/conformance/tests/l3-fault-injection/dr-restore/chainsaw-test.yaml @@ -40,10 +40,17 @@ # alternative source for the data. # # KNOWN, DELIBERATE GAPS (asserted nowhere — do not read this test as claiming them): -# * OFF-HOST is NOT proven, only OFF-CLUSTER. On the dev estate the k3s nodes are -# VMs on the same Proxmox host that runs the versitygw S3 target (192.168.1.9), -# so this drill does not survive loss of that hypervisor. Off-site/off-host -# replication is an operator concern outside R-STATE-03's wording. +# * OFF-HOST is NOT proven, only OFF-CLUSTER — TRUE AS RUN, and being fixed. On the +# dev estate the k3s nodes are VMs on the same Proxmox host that runs the +# versitygw/MinIO S3 target (192.168.1.9), so this drill does not survive loss of +# that hypervisor: one machine takes the cluster AND its backups. +# That is exactly why the operator moved off-cluster DR to Cloudflare R2 on +# 2026-09-05 (commit 32d6766 — S3_BACKUP_ENDPOINT is now an R2 endpoint). Once that +# merges and this drill is RE-RUN against R2, off-host stops being a gap and becomes +# proven, because the target no longer shares a failure domain with the cluster. +# UNTIL THEN THIS TEST'S PASS IS AGAINST MinIO. R-STATE-03 is currently proven only +# for the same-host target; re-proving it against R2 is a prerequisite for claiming +# off-host DR, and the ledger row that proves R-STATE-03 predates the move. # * The test does NOT read the backup objects out of the bucket with an S3 client. # Doing so would require handing the conformance runner the backup credential, # widening a privileged test identity for weaker evidence than (1)-(4) above.