Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 6 Actions Packages Projects Releases Wiki Activity

feat(conformance): L2 tests derive their environment from the cluster, not a literal

Browse Source 
Six L2 behavioural tests embedded `home.keithjackson.dev`. They are test INPUTS,
not reconciled manifests, so postBuild substitution never touched them — correct
while dev held the apex, wrong the moment prod came up on
prod.home.keithjackson.dev, and blocking §9's "suite passes green against prod".

Each test now asks the cluster what it is, via a test-level chainsaw binding over
flux-system/cluster-vars — the SAME ConfigMap Flux renders the platform from, so
a test cannot disagree with the cluster it runs on:

  bindings:
    - name: baseDomain
      value: (x_k8s_get($client,'v1','ConfigMap','flux-system','cluster-vars').data.BASE_DOMAIN)

Chosen over a flag or env var because there is nothing to forget at run time, and
test-level scope covers every step (step outputs/bindings are step-scoped).

THE APEX WAS NOT THE ONLY LEAK. dns-exposure and dns-internal-resolution also
hardcoded DEV'S Cilium LB range 192.168.1.53-.60; prod's is 192.168.2.20-.30, so
both were wrong for prod regardless of hostname. Replaced with assertions on the
OBSERVED value — the DNS answer must equal that cluster's GATEWAY_VIP, and the
public record must equal the LB IP the Service was actually assigned — which is
strictly STRONGER than the old range check as well as environment-independent.
sso-forward-auth held the literal only in prose (it talks over Service DNS).

PROVEN, NOT ASSERTED — and the positive control is the important half:
  * expressions extracted verbatim from the tracked files resolve to the correct,
    DIFFERENT values per cluster (dev home.keithjackson.dev / 192.168.1.53,
    prod prod.home.keithjackson.dev / 192.168.2.20);
  * the same assert with the HARDCODED apex passes on dev and FAILS on prod
    ("Invalid value: *.prod.home... Expected: *.home..."), so the harness is
    provably not blind — that is exactly the drift removed;
  * chainsaw lint VALID on all six; run.sh --lint clean over the whole tree.

END-TO-END L2 RUN AGAINST LIVE DEV: `dns-exposure` PASSED (42.6s) — the binding
produced conformance-dns.home.keithjackson.dev, external-dns published it, and the
probe Job asserted the record against the assigned LB IP. This is the first time
one of these tests has been executed since parameterization, and it exercises the
full path rather than just the binding.

FINDING FROM THAT RUN, worth recording: the test's teardown is ASYNCHRONOUS. It
reports PASS as soon as chainsaw deletes the namespace, but the Cloudflare record
is reaped later by external-dns's own --interval=1m sync. A zone query immediately
after PASS still showed conformance-dns.home.keithjackson.dev plus its owner=default
TXT registry entry. Harmless here, but it means a green dns-exposure does NOT prove
cleanup happened, and if external-dns were suspended the record would persist
silently. Verified separately that it clears.

conformance/incluster/rbac.yaml: its cluster-wide `get configmaps` is now
LOAD-BEARING for these bindings — narrowing it to the test namespace breaks five
tests. Commented so it is not "tidied" later.

l3-fault-injection/dr-restore: its KNOWN-GAPS block said off-host is unproven
because the MinIO shares a Proxmox host with dev's VMs. Still true as run, so the
text now says both — that the R2 move (32d6766) is what closes it, that this test's
pass is against MinIO, and that the ledger row proving R-STATE-03 PREDATES the move,
so off-host DR cannot be claimed until the drill is re-run against R2.

NOT VERIFIED, plainly: no full L2 suite run against prod, so "green against prod"
remains unproven — what is proven is that the tests now derive the CORRECT
expectation for whichever cluster they are pointed at. Whether prod SATISFIES them
(split-horizon, LE wildcard, Grafana's issuer) is untested.

VERIFIED: full CI gate PASS. flux-postbuild-substitution 260 assertions,
render-golden 10/10.

Refs R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04, R-ENV-02, R-DNS-01/02/03.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-05 12:05:05 -05:00
1 parent 765ad032db
commit 59ffbaf11a
10 files changed
+196 -39
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

conformance/incluster/rbac.yaml
+7
View File
@@ -57,6 +57,13 @@ rules:
verbs: [create, delete, get, list, watch, patch]
- apiGroups: [""]
resources: [pods, services, configmaps, secrets, persistentvolumeclaims]
# LOAD-BEARING `get` ON CONFIGMAPS, BEYOND FIXTURE CREATION (R-ENV-02): the environment-
# agnostic L2 tests (dns-exposure, dns-internal-resolution, gateway-ingress-reachable,
# oidc-sso-grafana, trusted-tls) resolve the cluster's own BASE_DOMAIN / GATEWAY_VIP at run
# time from `flux-system/cluster-vars` — the same ConfigMap Flux substitutes from — via
# chainsaw `x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars')` bindings.
# If this rule is ever narrowed to the test's own namespace, those bindings fail and the
# tests error out. Narrow it only alongside a namespace-scoped read of that ConfigMap.
verbs: [create, delete, get, list, watch, patch]
- apiGroups: [""]
resources: [pods/exec]
conformance/tests/README.md
+21
View File
@@ -93,6 +93,27 @@ and coverage extraction exact and robust.
break the thing and assert fail-closed / no-data-loss, with `finally` restoring state.
- **Never weaken a test to go green (R-TEST-06).** If the spec can't be met, escalate to
the PM — do not relax the assertion or the requirement.
- **Never write a per-environment value into a test (R-ENV-02).** No hostname suffix, no
Gateway VIP, no LB-pool range. One authored suite must run green against dev *and* prod,
so a test **asks the cluster it is pointed at** what it is. The mechanism is a chainsaw
binding over the same ConfigMap Flux substitutes from:
```yaml
spec:
bindings:
- name: baseDomain
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
```
Declare it at **test level** (step `outputs`/`bindings` are step-scoped), then consume it:
`($baseDomain)` for a pod `env:` value, `(join('.', ['myhost', $baseDomain]))` where a
full hostname is needed — including inside a `fixtures/*.yaml` applied with `apply: file:`,
which chainsaw templates with the same bindings. Prefer reading the *actual* observed value
over restating a range (e.g. assert the DNS answer equals `GATEWAY_VIP`, or equals the LB IP
the Service was assigned) — it is both stricter and environment-independent. Never take the
environment from a CLI flag or an exported variable: a flag someone forgets is how a suite
ends up silently asserting the other cluster's expectations. L1 goldens are the one
exception: they are rendering fixtures and carry the literal `${BASE_DOMAIN}` token.
Scaffolds in this tree carry `TODO(T-x.y)` markers where a fixture/assertion lands as a
later roadmap phase delivers the capability under test. `lib/status.py` treats such a
conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml
+50 -8
View File
@@ -3,11 +3,18 @@
# Catalog: docs/14 "DNS/exposure (R-DNS-*)"
# Spec: docs/09 (ExternalDNS auto-manages records; split-horizon foundation)
#
# ENVIRONMENT-AGNOSTIC (R-ENV-02): the hostname is built from the ${BASE_DOMAIN} of the CLUSTER
# UNDER TEST (read from flux-system/cluster-vars via a test-level chainsaw binding), and the
# expected answer is the LB IP that THIS Service was actually assigned (read back from its own
# status) — so neither the name nor the address is authored per environment, and no run-time flag
# can be forgotten. Requesting a record under the cluster's own suffix is also what keeps the two
# environments from ever describing the same public name (the 2026-09-04 incident precondition).
#
# GIVEN a LoadBalancer Service annotated with an external-dns hostname,
# WHEN ExternalDNS reconciles (source=service, provider=cloudflare, policy=sync),
# THEN a PUBLIC DNS record for that hostname is auto-created, pointing at the
# Service's Cilium-assigned LB IP (R-DNS-02). The record resolves to the
# cluster LB IP (Cilium L2 `shared` pool 192.168.1.53-.60), i.e. clients reach the cluster endpoint.
# Service's Cilium-assigned LB IP (R-DNS-02) — asserted as an EXACT match against that
# Service's own `status.loadBalancer.ingress[0].ip`, i.e. clients reach the cluster endpoint.
#
# CLEANUP: chainsaw deletes the per-test namespace (Service included) at teardown;
# ExternalDNS policy=sync then REMOVES the record from Cloudflare on its next pass.
@@ -38,6 +45,13 @@ spec:
apply: 90s
assert: 5m
exec: 6m
# -- R-ENV-02: the environment's hostname suffix is READ FROM THE CLUSTER UNDER TEST.
# flux-system/cluster-vars is the ConfigMap Flux substitutes ${BASE_DOMAIN} from, so this
# binding is the same source of truth the platform is rendered from — dev asks dev, prod
# asks prod, and there is no per-environment fork and no flag to forget.
bindings:
- name: baseDomain
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
steps:
- name: given-a-service-requesting-a-dns-name
try:
@@ -48,7 +62,8 @@ spec:
metadata:
name: dns-probe-svc
annotations:
external-dns.alpha.kubernetes.io/hostname: conformance-dns.home.keithjackson.dev
# conformance-dns.<this cluster's BASE_DOMAIN> — never a literal apex.
external-dns.alpha.kubernetes.io/hostname: (join('.', ['conformance-dns', $baseDomain]))
spec:
type: LoadBalancer
# No real backend needed — Cilium LB-IPAM assigns a VIP regardless, and
@@ -59,7 +74,27 @@ spec:
- name: http
port: 80
targetPort: 80
# Wait for Cilium LB-IPAM to actually assign the VIP: it is the value ExternalDNS will
# publish and the value the probe below asserts EXACTLY. `|| ` + empty-array literal
# keeps the expression nil-safe while the address is still absent, so the assert POLLS
# (fails and retries) instead of erroring out on the first evaluation.
- assert:
resource:
apiVersion: v1
kind: Service
metadata:
name: dns-probe-svc
status:
(length(loadBalancer.ingress || `[]`)): 1
- name: then-externaldns-creates-the-public-record
# Step-scoped binding: read back the LB IP THIS Service was assigned (the previous step's
# assert guarantees it exists by now — step bindings are evaluated when the step runs).
# Expecting that exact address is both stricter than the old hardcoded 192.168.1.53-.60
# range and correct in every environment: dev's pool is 192.168.1.53-.60, prod's is
# 192.168.2.20-.30, and neither is written down here.
bindings:
- name: lbIp
value: (x_k8s_get($client, 'v1', 'Service', $namespace, 'dns-probe-svc').status.loadBalancer.ingress[0].ip)
try:
- apply:
resource:
@@ -76,20 +111,27 @@ spec:
containers:
- name: probe
image: busybox:1.36.1
# Both values come from the cluster under test: the suffix from
# flux-system/cluster-vars, the address from this Service's own status.
env:
- name: BASE_DOMAIN
value: ($baseDomain)
- name: LB_IP
value: ($lbIp)
command:
- sh
- -c
- |
set -u
host="conformance-dns.home.keithjackson.dev"
host="conformance-dns.${BASE_DOMAIN}"
# Poll PUBLIC DNS (1.1.1.1) until the auto-created A record appears and
# points into the cluster LB range (Cilium L2 `shared` pool 192.168.1.53-.60). Proves ExternalDNS created it.
# points at THIS Service's assigned LB IP. Proves ExternalDNS created it.
for i in $(seq 1 28); do
ip="$(nslookup "$host" 1.1.1.1 2>/dev/null | grep -oE '192\.168\.1\.(5[3-9]|60)' | head -1)"
ip="$(nslookup "$host" 1.1.1.1 2>/dev/null | awk '/^Address: /{print $2}' | grep -Fx "${LB_IP}" | head -1)"
if [ -n "$ip" ]; then echo "RECORD_FOUND $host -> $ip"; exit 0; fi
echo "poll $i: record not yet published"; sleep 10
echo "poll $i: record not yet published as ${LB_IP}"; sleep 10
done
echo "FAIL: ExternalDNS did not publish the record within timeout"; exit 1
echo "FAIL: ExternalDNS did not publish ${host} -> ${LB_IP} within timeout"; exit 1
- assert:
resource:
apiVersion: batch/v1
conformance/tests/l2-behavioral/dns-internal-resolution/chainsaw-test.yaml
+40 -14
View File
@@ -2,17 +2,24 @@
# L2 BEHAVIORAL — split-horizon INTERNAL resolution (R-DNS-01 / R-DNS-03) [RUNNABLE]
# Catalog: docs/14 "DNS/exposure (R-DNS-*)"
# Spec: docs/09 "Split-horizon resolution (R-DNS-03)" — internal clients resolve
# <svc>.home.keithjackson.dev to the INTERNAL Gateway VIP, not the public hairpin.
# <svc>.${BASE_DOMAIN} to the INTERNAL Gateway VIP, not the public hairpin.
# Impl: platform/coredns/coredns-custom.yaml (k3s CoreDNS `import *.server`; template plugin
# answers the whole home.keithjackson.dev zone with ${GATEWAY_VIP}).
# answers the whole ${BASE_DOMAIN} zone with ${GATEWAY_VIP}).
#
# ENVIRONMENT-AGNOSTIC (R-ENV-02): this test asks the CLUSTER IT IS POINTED AT for its own
# ${BASE_DOMAIN} and ${GATEWAY_VIP} — the same flux-system/cluster-vars ConfigMap Flux
# substitutes from — via test-level chainsaw bindings (see `spec.bindings`). Nothing here is
# per-environment and no flag is passed at run time, so the suite cannot be run against the
# wrong expectation: dev asserts dev's zone/VIP, prod asserts prod's.
#
# GIVEN the split-horizon coredns-custom ConfigMap is reconciled into k3s CoreDNS,
# WHEN an in-cluster pod resolves a platform hostname over its PUBLIC name
# (keycloak.home.keithjackson.dev) using the default cluster resolver (CoreDNS 10.43.0.10),
# THEN the answer is the in-cluster Gateway VIP (the Cilium L2 `shared` gateway pool 192.168.1.53-.60) — NOT a
# public/off-cluster IP. This is what lets pod->public-hostname traffic hairpin through the
# Gateway with the correct SNI (*.home.keithjackson.dev) so the wildcard cert validates
# (R-DNS-01 "services reachable at <svc>.home..."; R-DNS-03 split-horizon).
# (keycloak.${BASE_DOMAIN}) using the default cluster resolver (CoreDNS 10.43.0.10),
# THEN the answer is EXACTLY that cluster's ${GATEWAY_VIP} (the address coredns-custom is
# templated with, and the `shared` Gateway's own status address) — NOT a public/off-cluster
# IP. This is what lets pod->public-hostname traffic hairpin through the
# Gateway with the correct SNI (*.${BASE_DOMAIN}) so the wildcard cert validates
# (R-DNS-01 "services reachable at <svc>.${BASE_DOMAIN}"; R-DNS-03 split-horizon).
#
# Complements dns-exposure (R-DNS-02, the EXTERNAL/public-record side). Together they cover the
# split-horizon pair: external answer = public record (R-DNS-02); internal answer = Gateway VIP
@@ -34,13 +41,23 @@ metadata:
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
An in-cluster pod resolving keycloak.home.keithjackson.dev via cluster DNS gets the internal
Gateway VIP (192.168.1.53-.60), not the public hairpin — split-horizon in-cluster resolution.
An in-cluster pod resolving keycloak.${BASE_DOMAIN} via cluster DNS gets that cluster's
internal Gateway VIP, not the public hairpin — split-horizon in-cluster resolution.
spec:
timeouts:
apply: 60s
assert: 2m
exec: 2m
# -- R-ENV-02: the environment's identity is READ FROM THE CLUSTER UNDER TEST, never authored
# here. flux-system/cluster-vars is the ConfigMap Flux itself substitutes from
# (clusters/base/cluster-vars.yaml + the dev/prod overlay patch), so these bindings are the
# same source of truth the platform is rendered from. Test-level => in scope for every step
# (unlike step `outputs`, which are step-scoped).
bindings:
- name: baseDomain
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
- name: gatewayVip
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.GATEWAY_VIP)
steps:
- name: then-cluster-dns-answers-with-the-internal-gateway-vip
try:
@@ -59,23 +76,32 @@ spec:
containers:
- name: probe
image: busybox:1.36.1
# The cluster's OWN identity, resolved from flux-system/cluster-vars by the
# test-level bindings — never a literal (R-ENV-02).
env:
- name: BASE_DOMAIN
value: ($baseDomain)
- name: GATEWAY_VIP
value: ($gatewayVip)
command:
- sh
- -c
- |
set -u
host="keycloak.home.keithjackson.dev"
host="keycloak.${BASE_DOMAIN}"
# Resolve via the DEFAULT cluster resolver (CoreDNS, /etc/resolv.conf ->
# 10.43.0.10). Split-horizon must answer with the in-cluster Gateway VIP
# (Cilium L2 `shared` pool 192.168.1.53-.60), proving pods hairpin through the Gateway.
# 10.43.0.10). Split-horizon must answer with THIS cluster's Gateway VIP
# (the address coredns-custom is templated with), proving pods hairpin
# through the Gateway. Exact match on ${GATEWAY_VIP}: stricter than the
# old hardcoded 192.168.1.53-.60 range AND correct in both environments.
out="$(nslookup "$host" 2>/dev/null)"
echo "$out"
ip="$(echo "$out" | awk '/^Address: /{print $2}' | grep -E '^192\.168\.1\.(5[3-9]|60)$' | head -1)"
ip="$(echo "$out" | awk '/^Address: /{print $2}' | grep -Fx "${GATEWAY_VIP}" | head -1)"
if [ -n "$ip" ]; then
echo "INTERNAL_RESOLUTION_OK $host -> $ip (in-cluster Gateway VIP)"
exit 0
fi
echo "FAIL: $host did not resolve to the in-cluster Gateway VIP (192.168.1.53-.60);"
echo "FAIL: $host did not resolve to this cluster's Gateway VIP (${GATEWAY_VIP});"
echo " split-horizon coredns-custom not in effect (got a public/off-cluster answer)."
exit 1
- assert:
conformance/tests/l2-behavioral/gateway-ingress-reachable/chainsaw-test.yaml
+15 -2
View File
@@ -68,6 +68,15 @@ spec:
timeouts:
apply: 60s
assert: 180s
# -- R-ENV-02: the fixture's route hostnames MUST fall under the Gateway's wildcard listener
# (*.${BASE_DOMAIN}) or they never attach — so the suffix is READ FROM THE CLUSTER UNDER TEST
# (flux-system/cluster-vars, the ConfigMap Flux substitutes ${BASE_DOMAIN} from) rather than
# authored here. Test-level => in scope for the fixture apply AND every probe step below
# (step `outputs`, by contrast, are step-scoped — see the gwvip re-capture comments).
# fixtures/app-and-probe.yaml consumes it via chainsaw resource templating.
bindings:
- name: baseDomain
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
steps:
# -- GIVEN: the exposed-app fixture (default-deny + gateway-ingress allow + backend +
# Service + declared/undeclared HTTPRoutes) and a probe namespace. Assert the shared
@@ -139,12 +148,14 @@ spec:
env:
- name: GWVIP
value: ($gwvip)
- name: BASE_DOMAIN
value: ($baseDomain)
command:
- sh
- -c
- |
set -u
host="gw-declared-conformance.home.keithjackson.dev"
host="gw-declared-conformance.${BASE_DOMAIN}"
last=000
i=0
while [ "$i" -lt 30 ]; do
@@ -207,12 +218,14 @@ spec:
env:
- name: GWVIP
value: ($gwvip)
- name: BASE_DOMAIN
value: ($baseDomain)
command:
- sh
- -c
- |
set -u
host="gw-undeclared-conformance.home.keithjackson.dev"
host="gw-undeclared-conformance.${BASE_DOMAIN}"
reached_gw=no
i=0
while [ "$i" -lt 8 ]; do
conformance/tests/l2-behavioral/gateway-ingress-reachable/fixtures/app-and-probe.yaml
+11 -2
View File
@@ -10,6 +10,13 @@
# The namespace opts into the shared Gateway (gateway-access: shared) exactly like a
# rendered app namespace, so its HTTPRoutes attach to the real platform Gateway and the
# probe exercises the true north-south path (Gateway Envoy -> backend), not a stand-in.
#
# ENVIRONMENT-AGNOSTIC HOSTNAMES (R-ENV-02): the two HTTPRoute hostnames are NOT literals. They
# are chainsaw-templated from the `baseDomain` binding declared in ../chainsaw-test.yaml, which
# reads BASE_DOMAIN out of the flux-system/cluster-vars ConfigMap of the cluster under test — the
# same ConfigMap Flux substitutes ${BASE_DOMAIN} from. They MUST fall under the shared Gateway's
# wildcard listener (*.${BASE_DOMAIN}) or the routes never attach, which is precisely why a
# hardcoded suffix silently broke this test on any cluster that did not hold the apex.
---
apiVersion: v1
kind: Namespace
@@ -134,7 +141,8 @@ spec:
namespace: gateway
sectionName: https
hostnames:
- gw-declared-conformance.home.keithjackson.dev
# gw-declared-conformance.<this cluster's BASE_DOMAIN> — see the header.
- (join('.', ['gw-declared-conformance', $baseDomain]))
rules:
- matches:
- path:
@@ -158,7 +166,8 @@ spec:
namespace: gateway
sectionName: https
hostnames:
- gw-undeclared-conformance.home.keithjackson.dev
# gw-undeclared-conformance.<this cluster's BASE_DOMAIN> — see the header.
- (join('.', ['gw-undeclared-conformance', $baseDomain]))
rules:
- matches:
- path:
conformance/tests/l2-behavioral/oidc-sso-grafana/chainsaw-test.yaml
+19 -5
View File
@@ -3,10 +3,15 @@
# Spec: docs/15 identity-and-access; ADR-0014. Impl: Grafana as a Keycloak OIDC client
# (platform/monitoring/kube-prometheus-stack.yaml auth.generic_oauth) — the self-contained SSO proof.
#
# ENVIRONMENT-AGNOSTIC (R-ENV-02): the expected IdP host is built from the ${BASE_DOMAIN} of the
# CLUSTER UNDER TEST, read from flux-system/cluster-vars (the ConfigMap Flux substitutes from) by
# a test-level chainsaw binding — so this asserts each cluster redirects to ITS OWN Keycloak, which
# is exactly what would catch a cross-environment OIDC issuer left pointing at the other cluster.
#
# GIVEN Grafana is deployed and configured as an OIDC client of the Keycloak `homelab` realm,
# WHEN an unauthenticated browser starts the OIDC login (GET /login/generic_oauth)
# -> THEN Grafana redirects (302) to the CENTRAL IdP's authorize endpoint on
# keycloak.home.keithjackson.dev/realms/homelab — proving SSO is driven by the one IdP
# keycloak.${BASE_DOMAIN}/realms/homelab — proving SSO is driven by the one IdP
# (R-IDP-01: central Identity Provider as the single user directory).
#
# Goes GREEN once the grafana-oidc client-secret hand-off is filled and Keycloak + Grafana are up.
@@ -30,6 +35,11 @@ spec:
apply: 60s
assert: 3m
exec: 3m
# -- R-ENV-02: the IdP hostname suffix is READ FROM THE CLUSTER UNDER TEST
# (flux-system/cluster-vars — the ConfigMap Flux substitutes ${BASE_DOMAIN} from).
bindings:
- name: baseDomain
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
steps:
- name: given-grafana-is-an-oidc-client
try:
@@ -59,6 +69,9 @@ spec:
containers:
- name: probe
image: busybox:1.36.1
env:
- name: BASE_DOMAIN
value: ($baseDomain)
command:
- sh
- -c
@@ -68,13 +81,14 @@ spec:
# busybox wget has no --max-redirect; -S dumps every response header
# (incl. the initial 302 Location to the IdP) before following, so the
# grep below still matches the Keycloak authorize URL. (Grafana verified
# live: 302 -> keycloak.home.keithjackson.dev/realms/homelab/.../auth.)
# live: 302 -> keycloak.${BASE_DOMAIN}/realms/homelab/.../auth.)
idp="keycloak.${BASE_DOMAIN}/realms/homelab"
out="$(wget -S -O /dev/null "$svc/login/generic_oauth" 2>&1 || true)"
echo "$out"
if echo "$out" | grep -qi 'keycloak.home.keithjackson.dev/realms/homelab'; then
echo GRAFANA_OIDC_REDIRECT_OK
if echo "$out" | grep -qiF "$idp"; then
echo "GRAFANA_OIDC_REDIRECT_OK -> ${idp}"
else
echo "FAIL: Grafana did not redirect to the Keycloak homelab realm"; exit 1
echo "FAIL: Grafana did not redirect to ${idp} (this cluster's Keycloak homelab realm)"; exit 1
fi
- assert:
resource:
conformance/tests/l2-behavioral/sso-forward-auth/chainsaw-test.yaml
+6 -1
View File
@@ -3,8 +3,13 @@
# Catalog: docs/14 onboarding (auth wiring) + docs/15 identity-and-access.
# Impl: platform/oauth2-proxy (forward-auth against the Keycloak `homelab` realm), ADR-0014.
#
# ENVIRONMENT-AGNOSTIC (R-ENV-02): this test names NO hostname. It talks to oauth2-proxy over its
# in-cluster Service DNS and asserts HTTP behaviour, so the same authored test is correct on dev and
# on prod. The issuer host below is written as ${BASE_DOMAIN} because that is what the platform
# substitutes; it is prose, not an input — nothing here needs parameterizing.
#
# GIVEN oauth2-proxy is deployed and has validated the Keycloak OIDC issuer (it only becomes Available
# after successfully discovering https://keycloak.home.keithjackson.dev/realms/homelab — this is
# after successfully discovering https://keycloak.${BASE_DOMAIN}/realms/homelab — this is
# the proxy->Keycloak flow working),
# WHEN an UNAUTHENTICATED request hits the forward-auth decision endpoint (/oauth2/auth)
# -> THEN it is challenged: HTTP 401 (no valid Keycloak session => access denied), and a normal
conformance/tests/l2-behavioral/trusted-tls/chainsaw-test.yaml
+16 -3
View File
@@ -3,8 +3,13 @@
# Catalog: docs/14 "Trusted TLS (R-TLS-01/02)"
# Spec: docs/09 (Gateway API + DNS-01 wildcard), ADR-0006 (LE issuers)
#
# ENVIRONMENT-AGNOSTIC (R-ENV-02): the hostnames are built from the ${BASE_DOMAIN} of the CLUSTER
# UNDER TEST, read from flux-system/cluster-vars (the ConfigMap Flux substitutes from) by a
# test-level chainsaw binding. The same authored test therefore verifies dev's wildcard on dev and
# prod's on prod, with nothing to pass at run time.
#
# GIVEN the shared Gateway terminating TLS with the cert-manager wildcard cert,
# WHEN an in-cluster probe dials it by hostname (SNI *.home.keithjackson.dev) and
# WHEN an in-cluster probe dials it by hostname (SNI *.${BASE_DOMAIN}) and
# verifies the chain against the SYSTEM trust store,
# THEN the served cert is non-self-signed, unexpired, covers the wildcard SAN, and
# chains to a PUBLICLY-TRUSTED CA root (R-TLS-01). The wildcard SAN also covers
@@ -36,6 +41,11 @@ metadata:
that chains to a publicly-trusted CA root (LE production); the wildcard SAN
covers internal-only hostnames too.
spec:
# -- R-ENV-02: the hostname suffix is READ FROM THE CLUSTER UNDER TEST (flux-system/cluster-vars
# — the ConfigMap Flux substitutes ${BASE_DOMAIN} from), never authored per environment.
bindings:
- name: baseDomain
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
steps:
- name: then-gateway-cert-chains-to-trusted-root
try:
@@ -54,13 +64,16 @@ spec:
containers:
- name: probe
image: curlimages/curl:8.11.1
env:
- name: BASE_DOMAIN
value: ($baseDomain)
command:
- sh
- -c
- |
set -eu
# Probe TWO wildcard hostnames — one "public", one "internal-only". Both
# match *.home.keithjackson.dev, so a clean verify against the SYSTEM CA
# match *.${BASE_DOMAIN}, so a clean verify against the SYSTEM CA
# bundle for BOTH proves R-TLS-01 (chains to a publicly-trusted root) AND
# R-TLS-03 (the same trusted wildcard covers internal-only names). Names
# resolve via in-cluster split-horizon DNS to the Gateway VIP. curl (no
@@ -68,7 +81,7 @@ spec:
# the gateway's 404 once TLS+hostname verify; it exits 60 if the chain or
# hostname fails. TLS terminates at the gateway before HTTP routing, so no
# backend is needed.
for host in conformance.home.keithjackson.dev internal-only.home.keithjackson.dev; do
for host in "conformance.${BASE_DOMAIN}" "internal-only.${BASE_DOMAIN}"; do
curl -sS --retry 5 --retry-connrefused --retry-delay 3 --max-time 20 \
-o /dev/null "https://${host}/"
echo "TLS_TRUSTED ${host}"
conformance/tests/l3-fault-injection/dr-restore/chainsaw-test.yaml
+11 -4
View File
@@ -40,10 +40,17 @@
# alternative source for the data.
#
# KNOWN, DELIBERATE GAPS (asserted nowhere — do not read this test as claiming them):
# * OFF-HOST is NOT proven, only OFF-CLUSTER. On the dev estate the k3s nodes are
# VMs on the same Proxmox host that runs the versitygw S3 target (192.168.1.9),
# so this drill does not survive loss of that hypervisor. Off-site/off-host
# replication is an operator concern outside R-STATE-03's wording.
# * OFF-HOST is NOT proven, only OFF-CLUSTER — TRUE AS RUN, and being fixed. On the
# dev estate the k3s nodes are VMs on the same Proxmox host that runs the
# versitygw/MinIO S3 target (192.168.1.9), so this drill does not survive loss of
# that hypervisor: one machine takes the cluster AND its backups.
# That is exactly why the operator moved off-cluster DR to Cloudflare R2 on
# 2026-09-05 (commit 32d6766 — S3_BACKUP_ENDPOINT is now an R2 endpoint). Once that
# merges and this drill is RE-RUN against R2, off-host stops being a gap and becomes
# proven, because the target no longer shares a failure domain with the cluster.
# UNTIL THEN THIS TEST'S PASS IS AGAINST MinIO. R-STATE-03 is currently proven only
# for the same-host target; re-proving it against R2 is a prerequisite for claiming
# off-host DR, and the ledger row that proves R-STATE-03 predates the move.
# * The test does NOT read the backup objects out of the bucket with an S3 client.
# Doing so would require handing the conformance runner the backup credential,
# widening a privileged test identity for weaker evidence than (1)-(4) above.