feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
d1a6628a97
commit
6a8953e288
46 files changed
+7898
-71
No files matched your search
@@ -0,0 +1,391 @@
|
||||
# =============================================================================
|
||||
# L2 BEHAVIORAL — certificate lifecycle is automatic (R-TLS-02)
|
||||
# [RUNNABLE / NOT YET EXECUTED — see STATUS]
|
||||
# Catalog: docs/14 "Trusted TLS (R-TLS-01/02)"
|
||||
# Spec: docs/00 R-TLS-02 — "Certificates MUST be automatically **issued, renewed,
|
||||
# and revoked** without manual steps."
|
||||
# docs/08 §"Renewal & revocation (R-TLS-02)" — "cert-manager renews well
|
||||
# before expiry automatically; no human action." / "Revocation: removing/
|
||||
# rotating a certificate is a declarative change; cert-manager handles
|
||||
# issuance of replacements."
|
||||
# docs/08 acceptance gate — "Certificates renew automatically ahead of expiry
|
||||
# with no manual step (**observed across a renewal**)."
|
||||
# Impl: platform/cert-manager/**, platform/gateway/wildcard-certificate.yaml.
|
||||
# Sibling: tests/l0-static/tls-lifecycle-automated proves the "without manual steps"
|
||||
# clause statically (in the merge gate). This test observes the three VERBS.
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC (R-ENV-02): NOTHING about either environment is written down. The
|
||||
# serving certificate is DISCOVERED — find the Gateway whose listener terminates TLS, follow
|
||||
# its certificateRefs Secret, and read that Secret's `cert-manager.io/certificate-name`
|
||||
# annotation to reach the Certificate. No hostname, no namespace list, no issuer name, no
|
||||
# address: dev proves dev's cert and prod proves prod's from the same authored assertions.
|
||||
#
|
||||
# GIVEN cert-manager reconciled by Flux and a Gateway terminating TLS,
|
||||
# WHEN the serving Certificate's provenance and renewal schedule are read, and a NEW
|
||||
# Certificate is then requested, left to renew itself, and removed,
|
||||
# THEN 1. the serving credential was produced by the CONTROLLER, not a human: its
|
||||
# Secret carries cert-manager's certificate-name annotation, its
|
||||
# CertificateRequest was submitted by cert-manager's own ServiceAccount and
|
||||
# was APPROVED automatically, and its issuer is a Ready ACME ClusterIssuer;
|
||||
# 2. an unattended RENEWAL IS SCHEDULED: status.renewalTime is in the future and
|
||||
# strictly before status.notAfter, so no human is in the renewal path;
|
||||
# 3. ISSUED — a newly declared Certificate becomes Ready and its Secret appears
|
||||
# with certificate material, with no step between declaring and having it;
|
||||
# 4. RENEWED — that certificate, given a renewal window that opens one minute
|
||||
# after issuance, REISSUES ITSELF at least twice (status.revision >= 3) and
|
||||
# the serving bytes actually change, with no human action. This is docs/08's
|
||||
# "observed across a renewal", fast-forwarded;
|
||||
# 5. REMOVED/REVOKED — deleting the Certificate withdraws its serving material.
|
||||
#
|
||||
# WHY A HERMETIC SELF-SIGNED ISSUER FOR STEPS 3-5, AND WHAT THAT COSTS
|
||||
# -------------------------------------------------------------------
|
||||
# Steps 3-5 need a certificate lifecycle that completes in minutes and repeats on every
|
||||
# run. Driving that through the real ACME issuer would fire a public Let's Encrypt order
|
||||
# (and a Cloudflare _acme-challenge write) per conformance run, burning issuance rate
|
||||
# limits for evidence that is not about ACME: cert-manager's renewal scheduler and
|
||||
# ownership logic are issuer-independent. So steps 3-5 use a `selfSigned` Issuer created
|
||||
# in this test's OWN namespace, and the ACME/DNS-01 path is proven elsewhere — step 1-2
|
||||
# read the REAL wildcard certificate the Gateway serves, and l2-behavioral/trusted-tls
|
||||
# proves a publicly-trusted LE certificate is actually served (R-TLS-01/03).
|
||||
# STATED PLAINLY: steps 3-5 do NOT prove the ACME order path renews; they prove the
|
||||
# CONTROLLER's issue/renew/withdraw automation runs unattended in this cluster, while
|
||||
# steps 1-2 prove the real serving certificate is on exactly that automated path.
|
||||
#
|
||||
# STATUS: RUNNABLE, NOT YET EXECUTED. Steps 1-2 are read-only. Steps 3-5 CREATE an
|
||||
# Issuer/Certificate (and DELETE the Certificate they created) inside chainsaw's
|
||||
# per-test namespace — the same class of mutation every other L2 test here performs,
|
||||
# self-cleaning via namespace teardown, touching no platform object. WS-TEST authored
|
||||
# them against live dev/prod read-only inspection under an explicit instruction not to
|
||||
# mutate either cluster, so THE MUTATING STEPS HAVE NOT BEEN RUN. Nothing about the
|
||||
# suite gates on them: ci/conformance.sh only runs L2/L3 when CLUSTER_AVAILABLE=true.
|
||||
#
|
||||
# EXPECTED FAILURE, DELIBERATELY NOT SOFTENED (R-TEST-06)
|
||||
# ------------------------------------------------------
|
||||
# Step 5 WILL FAIL as the platform is authored today. cert-manager only garbage-collects
|
||||
# a Certificate's Secret when the controller runs with `enableCertificateOwnerRef: true`;
|
||||
# platform/cert-manager/helmrelease.yaml does not set it (chart default false), and the
|
||||
# live wildcard Secret on both clusters has NO ownerReferences (verified read-only,
|
||||
# 2026-09-05). So a removed Certificate leaves its credential in place, and "revoked
|
||||
# without manual steps" does not hold. That is a finding about the platform, not a
|
||||
# defect in this test: the assertion stays as the requirement reads and is escalated to
|
||||
# the PM. (ACME revocation proper is not implemented by cert-manager at all.)
|
||||
#
|
||||
# KNOWN, DELIBERATE GAPS (this test claims none of these):
|
||||
# * Not proven: that the CA would refuse a revoked certificate — there is no
|
||||
# revocation request to make, so nothing observable exists to assert.
|
||||
# * Not proven: renewal of the REAL wildcard cert (its window opens 30 days before a
|
||||
# 90-day expiry; a conformance run cannot wait, and forcing it would rotate the
|
||||
# platform's serving credential). Step 2 asserts the schedule; step 4 asserts the
|
||||
# mechanism that will act on it.
|
||||
# * Not proven: certificate expiry ALERTING — docs/08 delegates that to R-OBS-04
|
||||
# (l3-fault-injection/alerting).
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: cert-lifecycle-automated
|
||||
labels:
|
||||
conformance.home.k3s/R-TLS-02: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-TLS-02"
|
||||
conformance.home.k3s/layer: "L2"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Given cert-manager and a TLS-terminating Gateway, when the serving certificate's
|
||||
provenance and renewal schedule are read and a new certificate is requested, left
|
||||
to renew, and removed, then issuance, renewal and withdrawal all happen with no
|
||||
human step — the serving cert was controller-issued and auto-approved, its
|
||||
renewal is scheduled ahead of expiry, and a declared certificate issues and
|
||||
reissues itself unattended.
|
||||
spec:
|
||||
timeouts:
|
||||
apply: 60s
|
||||
assert: 3m
|
||||
exec: 10m
|
||||
steps:
|
||||
# ---------------------------------------------------------------- READ-ONLY HALF
|
||||
- name: given-cert-manager-is-reconciled-and-healthy
|
||||
description: >-
|
||||
The controller that issues and renews must be running, and must have been
|
||||
installed by Flux rather than by hand — otherwise "automatic" has no engine.
|
||||
try:
|
||||
- script:
|
||||
content: |
|
||||
set -u
|
||||
# Discover cert-manager's Deployments by chart label, in whatever namespace
|
||||
# the release lives (nothing about the environment is assumed).
|
||||
sel='app.kubernetes.io/instance=cert-manager'
|
||||
rows="$(kubectl get deploy -A -l "$sel" \
|
||||
-o custom-columns='NS:.metadata.namespace,N:.metadata.name,RDY:.status.readyReplicas' \
|
||||
--no-headers 2>/dev/null || true)"
|
||||
echo "$rows"
|
||||
[ -n "$rows" ] || { echo "FAIL: no cert-manager Deployment found ($sel)"; exit 1; }
|
||||
n=0
|
||||
echo "$rows" | while read -r ns name rdy; do
|
||||
[ "${rdy:-0}" -ge 1 ] || { echo "FAIL: $ns/$name has no ready replica"; exit 1; }
|
||||
done || exit 1
|
||||
n="$(echo "$rows" | wc -l)"
|
||||
# controller + webhook + cainjector: without the webhook a Certificate is
|
||||
# rejected, without cainjector the webhook has no CA — all three or nothing.
|
||||
[ "$n" -ge 3 ] || { echo "FAIL: expected >=3 cert-manager Deployments, saw $n"; exit 1; }
|
||||
# GitOps-managed, not hand-installed: Flux labels what it applies.
|
||||
ns="$(echo "$rows" | head -1 | awk '{print $1}')"
|
||||
lbl="$(kubectl get deploy -n "$ns" -l "$sel" \
|
||||
-o jsonpath='{.items[0].metadata.labels.helm\.toolkit\.fluxcd\.io/name}' 2>/dev/null || true)"
|
||||
[ -n "$lbl" ] || { echo "FAIL: cert-manager is not owned by a Flux HelmRelease"; exit 1; }
|
||||
echo "CERT_MANAGER_HEALTHY_OK ($n deployments, flux HelmRelease $lbl)"
|
||||
check:
|
||||
(contains($stdout, 'CERT_MANAGER_HEALTHY_OK')): true
|
||||
|
||||
- name: then-the-serving-certificate-was-issued-by-the-controller
|
||||
description: >-
|
||||
Follow the Gateway's terminating listener to the Secret it serves, and prove
|
||||
that Secret's provenance is cert-manager — annotation, CertificateRequest
|
||||
requester, automatic approval, and a Ready ACME issuer.
|
||||
try:
|
||||
- script:
|
||||
content: |
|
||||
set -u
|
||||
# --- discover the serving Secret from the Gateway itself -------------------
|
||||
gw="$(kubectl get gateway -A \
|
||||
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}' \
|
||||
2>/dev/null | head -1)"
|
||||
[ -n "$gw" ] || { echo "FAIL: no Gateway found — nothing terminates TLS"; exit 1; }
|
||||
gns="${gw% *}"; gname="${gw#* }"
|
||||
sec="$(kubectl -n "$gns" get gateway "$gname" \
|
||||
-o jsonpath='{.spec.listeners[?(@.tls.mode=="Terminate")].tls.certificateRefs[0].name}' \
|
||||
2>/dev/null | awk '{print $1}')"
|
||||
[ -n "$sec" ] || { echo "FAIL: Gateway $gns/$gname terminates no TLS listener"; exit 1; }
|
||||
echo "gateway=$gns/$gname serving-secret=$sec"
|
||||
|
||||
# --- provenance (1): only the controller writes this annotation ------------
|
||||
cert="$(kubectl -n "$gns" get secret "$sec" \
|
||||
-o jsonpath='{.metadata.annotations.cert-manager\.io/certificate-name}' 2>/dev/null || true)"
|
||||
[ -n "$cert" ] || { echo "FAIL: Secret $gns/$sec carries no cert-manager.io/certificate-name annotation — it was not produced by cert-manager"; exit 1; }
|
||||
crt="$(kubectl -n "$gns" get secret "$sec" -o jsonpath='{.data.tls\.crt}' 2>/dev/null || true)"
|
||||
[ -n "$crt" ] || { echo "FAIL: Secret $gns/$sec holds no tls.crt"; exit 1; }
|
||||
|
||||
# --- provenance (2): the Certificate is Ready and has a revision -----------
|
||||
ready="$(kubectl -n "$gns" get certificate "$cert" \
|
||||
-o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)"
|
||||
rev="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.status.revision}' 2>/dev/null || true)"
|
||||
echo "certificate=$gns/$cert ready=$ready revision=$rev"
|
||||
[ "$ready" = "True" ] || { echo "FAIL: Certificate $cert is not Ready"; exit 1; }
|
||||
[ -n "$rev" ] && [ "$rev" -ge 1 ] || { echo "FAIL: Certificate $cert has no issued revision"; exit 1; }
|
||||
|
||||
# --- provenance (3): the request was made BY THE CONTROLLER and auto-approved
|
||||
# spec.username is the identity that created the CertificateRequest. A human
|
||||
# operator would appear as a user/OIDC identity, not a ServiceAccount.
|
||||
cr="$(kubectl -n "$gns" get certificaterequest \
|
||||
-o jsonpath="{range .items[?(@.metadata.annotations.cert-manager\.io/certificate-name=='${cert}')]}{.metadata.name}{' '}{.spec.username}{'\n'}{end}" \
|
||||
2>/dev/null | tail -1)"
|
||||
[ -n "$cr" ] || { echo "FAIL: no CertificateRequest for $cert — cannot show who requested it"; exit 1; }
|
||||
crname="${cr% *}"; requester="${cr#* }"
|
||||
echo "certificaterequest=$crname requester=$requester"
|
||||
case "$requester" in
|
||||
system:serviceaccount:*) : ;;
|
||||
*) echo "FAIL: CertificateRequest $crname was submitted by '$requester', not a controller ServiceAccount"; exit 1 ;;
|
||||
esac
|
||||
appr="$(kubectl -n "$gns" get certificaterequest "$crname" \
|
||||
-o jsonpath='{.status.conditions[?(@.type=="Approved")].status}' 2>/dev/null || true)"
|
||||
[ "$appr" = "True" ] || { echo "FAIL: CertificateRequest $crname is not Approved=True (approval would be a manual step)"; exit 1; }
|
||||
|
||||
# --- provenance (4): an automated ACME issuer, and it is Ready -------------
|
||||
ikind="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.spec.issuerRef.kind}')"
|
||||
iname="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.spec.issuerRef.name}')"
|
||||
echo "issuer=$ikind/$iname"
|
||||
if [ "$ikind" = "ClusterIssuer" ]; then
|
||||
iready="$(kubectl get clusterissuer "$iname" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)"
|
||||
iacme="$(kubectl get clusterissuer "$iname" -o jsonpath='{.spec.acme.server}' 2>/dev/null || true)"
|
||||
else
|
||||
iready="$(kubectl -n "$gns" get issuer "$iname" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)"
|
||||
iacme="$(kubectl -n "$gns" get issuer "$iname" -o jsonpath='{.spec.acme.server}' 2>/dev/null || true)"
|
||||
fi
|
||||
[ "$iready" = "True" ] || { echo "FAIL: issuer $iname is not Ready — renewal would need a human to fix it"; exit 1; }
|
||||
[ -n "$iacme" ] || { echo "FAIL: issuer $iname is not an ACME issuer"; exit 1; }
|
||||
echo "SERVING_CERT_CONTROLLER_ISSUED_OK"
|
||||
check:
|
||||
(contains($stdout, 'SERVING_CERT_CONTROLLER_ISSUED_OK')): true
|
||||
|
||||
- name: then-an-unattended-renewal-is-already-scheduled
|
||||
description: >-
|
||||
cert-manager writes status.renewalTime only after computing when it will renew.
|
||||
Asserting it is in the future and before expiry proves the renewal is scheduled
|
||||
and that nothing is waiting on a human (docs/08 "renews well before expiry").
|
||||
try:
|
||||
- script:
|
||||
content: |
|
||||
set -u
|
||||
gw="$(kubectl get gateway -A \
|
||||
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}' | head -1)"
|
||||
gns="${gw% *}"; gname="${gw#* }"
|
||||
sec="$(kubectl -n "$gns" get gateway "$gname" \
|
||||
-o jsonpath='{.spec.listeners[?(@.tls.mode=="Terminate")].tls.certificateRefs[0].name}' | awk '{print $1}')"
|
||||
cert="$(kubectl -n "$gns" get secret "$sec" \
|
||||
-o jsonpath='{.metadata.annotations.cert-manager\.io/certificate-name}')"
|
||||
renew="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.status.renewalTime}')"
|
||||
after="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.status.notAfter}')"
|
||||
[ -n "$renew" ] || { echo "FAIL: Certificate $cert has no status.renewalTime — no renewal is scheduled"; exit 1; }
|
||||
[ -n "$after" ] || { echo "FAIL: Certificate $cert has no status.notAfter"; exit 1; }
|
||||
now_s="$(date -u +%s)"
|
||||
r_s="$(date -u -d "$renew" +%s)"
|
||||
a_s="$(date -u -d "$after" +%s)"
|
||||
echo "now=$(date -u -d "@$now_s" +%FT%TZ) renewalTime=$renew notAfter=$after"
|
||||
echo "renewal is in $(( (r_s - now_s) / 86400 ))d; expiry in $(( (a_s - now_s) / 86400 ))d"
|
||||
[ "$r_s" -gt "$now_s" ] || { echo "FAIL: renewalTime is in the PAST — the scheduled renewal did not happen"; exit 1; }
|
||||
[ "$r_s" -lt "$a_s" ] || { echo "FAIL: renewalTime is not before notAfter — renewal would come too late"; exit 1; }
|
||||
echo "RENEWAL_SCHEDULED_OK"
|
||||
check:
|
||||
(contains($stdout, 'RENEWAL_SCHEDULED_OK')): true
|
||||
|
||||
# ------------------------------------------------- MUTATING HALF (hermetic namespace)
|
||||
- name: when-a-certificate-is-declared-then-it-is-issued-with-no-human-step
|
||||
description: >-
|
||||
Declare an Issuer and a Certificate in this test's own namespace, with a renewal
|
||||
window that opens one minute after issuance. Nothing else is done to them for the
|
||||
rest of the test — every later transition is the controller acting on its own.
|
||||
try:
|
||||
- apply:
|
||||
resource:
|
||||
# selfSigned: no external CA, no rate limit, no DNS write. See the header for
|
||||
# what this proves (controller automation) and what it does not (the ACME path).
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Issuer
|
||||
metadata:
|
||||
name: conformance-lifecycle
|
||||
spec:
|
||||
selfSigned: {}
|
||||
- apply:
|
||||
resource:
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: conformance-lifecycle
|
||||
spec:
|
||||
secretName: conformance-lifecycle-tls
|
||||
# 1h is cert-manager's minimum duration; renewing 59m before expiry puts the
|
||||
# renewal instant ~1 minute after each issuance, so an unattended renewal is
|
||||
# observable inside a test run instead of 30 days from now.
|
||||
duration: 1h
|
||||
renewBefore: 3540s
|
||||
commonName: conformance-lifecycle.invalid
|
||||
dnsNames:
|
||||
- conformance-lifecycle.invalid
|
||||
privateKey:
|
||||
rotationPolicy: Always
|
||||
issuerRef:
|
||||
kind: Issuer
|
||||
name: conformance-lifecycle
|
||||
- assert:
|
||||
resource:
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: conformance-lifecycle
|
||||
status:
|
||||
conditions:
|
||||
- type: Ready
|
||||
status: "True"
|
||||
revision: 1
|
||||
- script:
|
||||
content: |
|
||||
set -u
|
||||
crt="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \
|
||||
-o jsonpath='{.data.tls\.crt}' 2>/dev/null || true)"
|
||||
key="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \
|
||||
-o jsonpath='{.data.tls\.key}' 2>/dev/null || true)"
|
||||
[ -n "$crt" ] && [ -n "$key" ] || { echo "FAIL: no certificate material was issued into conformance-lifecycle-tls"; exit 1; }
|
||||
# Record the serving bytes so the renewal step can prove they CHANGED.
|
||||
echo "$crt" | cksum > /tmp/conformance-cert-r1.cksum
|
||||
echo "revision-1 cert fingerprint: $(cat /tmp/conformance-cert-r1.cksum)"
|
||||
echo "CERT_ISSUED_OK"
|
||||
check:
|
||||
(contains($stdout, 'CERT_ISSUED_OK')): true
|
||||
|
||||
- name: then-it-renews-itself-before-expiry-with-no-human-step
|
||||
description: >-
|
||||
Nothing touches the Certificate here. Poll until the controller has reissued it
|
||||
at least twice on its own and the serving bytes differ from revision 1 — docs/08's
|
||||
"observed across a renewal", and every request auto-approved.
|
||||
try:
|
||||
- script:
|
||||
content: |
|
||||
set -u
|
||||
want=3 # revision 1 = issuance; >=3 means TWO unattended renewals
|
||||
for i in $(seq 1 30); do
|
||||
rev="$(kubectl -n "$NAMESPACE" get certificate conformance-lifecycle \
|
||||
-o jsonpath='{.status.revision}' 2>/dev/null || echo 0)"
|
||||
echo "poll $i: revision=${rev:-0}"
|
||||
[ "${rev:-0}" -ge "$want" ] && break
|
||||
sleep 20
|
||||
done
|
||||
rev="$(kubectl -n "$NAMESPACE" get certificate conformance-lifecycle \
|
||||
-o jsonpath='{.status.revision}' 2>/dev/null || echo 0)"
|
||||
[ "${rev:-0}" -ge "$want" ] || { echo "FAIL: revision reached only ${rev:-0} — cert-manager did not renew unattended"; exit 1; }
|
||||
|
||||
# The serving material must actually have rotated, not just the revision count.
|
||||
now="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \
|
||||
-o jsonpath='{.data.tls\.crt}' | cksum)"
|
||||
was="$(cat /tmp/conformance-cert-r1.cksum)"
|
||||
echo "r1=$was now=$now"
|
||||
[ "$now" != "$was" ] || { echo "FAIL: the Secret still holds the revision-1 certificate — nothing was actually reissued"; exit 1; }
|
||||
|
||||
# Every renewal request was approved by the controller, never by a person.
|
||||
bad=0
|
||||
for cr in $(kubectl -n "$NAMESPACE" get certificaterequest -o name 2>/dev/null); do
|
||||
st="$(kubectl -n "$NAMESPACE" get "$cr" \
|
||||
-o jsonpath='{.status.conditions[?(@.type=="Approved")].status}' 2>/dev/null || true)"
|
||||
who="$(kubectl -n "$NAMESPACE" get "$cr" -o jsonpath='{.spec.username}' 2>/dev/null || true)"
|
||||
echo " $cr approved=$st requester=$who"
|
||||
[ "$st" = "True" ] || bad=1
|
||||
case "$who" in system:serviceaccount:*) : ;; *) bad=1 ;; esac
|
||||
done
|
||||
[ "$bad" = 0 ] || { echo "FAIL: a CertificateRequest was not auto-approved, or was not submitted by a controller"; exit 1; }
|
||||
echo "CERT_RENEWED_UNATTENDED_OK (revision=$rev)"
|
||||
check:
|
||||
(contains($stdout, 'CERT_RENEWED_UNATTENDED_OK')): true
|
||||
|
||||
- name: then-removing-the-certificate-withdraws-the-serving-material
|
||||
description: >-
|
||||
docs/08 reads revocation as "removing/rotating a certificate is a declarative
|
||||
change". The declarative change is the deletion; withdrawal of the credential
|
||||
must follow with no manual cleanup. EXPECTED TO FAIL TODAY — cert-manager needs
|
||||
`enableCertificateOwnerRef: true` for the Secret to be garbage-collected, and the
|
||||
tracked HelmRelease does not set it. Kept as the requirement reads (R-TEST-06).
|
||||
try:
|
||||
- delete:
|
||||
ref:
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
name: conformance-lifecycle
|
||||
- script:
|
||||
content: |
|
||||
set -u
|
||||
for i in $(seq 1 15); do
|
||||
if ! kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls >/dev/null 2>&1; then
|
||||
echo "CERT_WITHDRAWN_OK (serving material gone after $((i * 4))s)"
|
||||
exit 0
|
||||
fi
|
||||
sleep 4
|
||||
done
|
||||
owners="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \
|
||||
-o jsonpath='{.metadata.ownerReferences}' 2>/dev/null || true)"
|
||||
echo "FAIL: the Certificate was deleted but Secret conformance-lifecycle-tls still"
|
||||
echo " holds usable certificate material (ownerReferences=${owners:-none})."
|
||||
echo " Removing a certificate therefore does NOT withdraw the credential, so"
|
||||
echo " R-TLS-02's 'revoked ... without manual steps' does not hold. FIX:"
|
||||
echo " set enableCertificateOwnerRef: true in platform/cert-manager values"
|
||||
echo " (owner decision — WS-TEST does not weaken the assertion instead)."
|
||||
exit 1
|
||||
check:
|
||||
(contains($stdout, 'CERT_WITHDRAWN_OK')): true
|
||||
finally:
|
||||
# Namespace teardown removes both objects anyway; explicit so a leftover Secret
|
||||
# from the expected failure above is never left holding key material.
|
||||
- script:
|
||||
content: |
|
||||
kubectl -n "$NAMESPACE" delete secret conformance-lifecycle-tls --ignore-not-found --wait=false || true
|
||||
kubectl -n "$NAMESPACE" delete certificate conformance-lifecycle --ignore-not-found --wait=false || true
|
||||
kubectl -n "$NAMESPACE" delete issuer conformance-lifecycle --ignore-not-found --wait=false || true
|
||||
rm -f /tmp/conformance-cert-r1.cksum || true
|
||||
@@ -153,3 +153,20 @@ spec:
|
||||
finally:
|
||||
- script:
|
||||
content: 'echo "cleanup handled by chainsaw namespace teardown"'
|
||||
# -----------------------------------------------------------------------------------
|
||||
# PROOF-OF-PASS SENTINEL (R-SEC-03). Every step above is an assert/Job-status gate, so
|
||||
# chainsaw already fails the test before reaching this step if any of them fails — but a
|
||||
# green Chainsaw JUnit carries NO stdout, and the L2/L3 tests in this suite can self-skip
|
||||
# GREEN, so lib/record_run.py refuses to treat a bare `passed` as evidence and records
|
||||
# `unknown-no-sentinel-declared` instead. Without a declared token this test could pass
|
||||
# forever and NEVER become a recorded proof of R-NET-01 — the isolation half of docs/07's audit proof list
|
||||
# — which is precisely what docs/07 asks for ("These proofs are part of the platform
|
||||
# acceptance suite, not one-time checks"). This step exists only to emit that token, and it
|
||||
# is reached only when everything above has passed. Asserted by
|
||||
# tests/l0-static/auditability-proofs-producible.
|
||||
- name: then-emit-the-proof-of-pass-sentinel
|
||||
try:
|
||||
- script:
|
||||
content: echo DEFAULT_DENY_PROVEN_OK
|
||||
check:
|
||||
(contains($stdout, 'DEFAULT_DENY_PROVEN_OK')): true
|
||||
@@ -3,11 +3,18 @@
|
||||
# Catalog: docs/14 "DNS/exposure (R-DNS-*)"
|
||||
# Spec: docs/09 (ExternalDNS auto-manages records; split-horizon foundation)
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC (R-ENV-02): the hostname is built from the ${BASE_DOMAIN} of the CLUSTER
|
||||
# UNDER TEST (read from flux-system/cluster-vars via a test-level chainsaw binding), and the
|
||||
# expected answer is the LB IP that THIS Service was actually assigned (read back from its own
|
||||
# status) — so neither the name nor the address is authored per environment, and no run-time flag
|
||||
# can be forgotten. Requesting a record under the cluster's own suffix is also what keeps the two
|
||||
# environments from ever describing the same public name (the 2026-09-04 incident precondition).
|
||||
#
|
||||
# GIVEN a LoadBalancer Service annotated with an external-dns hostname,
|
||||
# WHEN ExternalDNS reconciles (source=service, provider=cloudflare, policy=sync),
|
||||
# THEN a PUBLIC DNS record for that hostname is auto-created, pointing at the
|
||||
# Service's Cilium-assigned LB IP (R-DNS-02). The record resolves to the
|
||||
# cluster LB IP (Cilium L2 `shared` pool 192.168.1.53-.60), i.e. clients reach the cluster endpoint.
|
||||
# Service's Cilium-assigned LB IP (R-DNS-02) — asserted as an EXACT match against that
|
||||
# Service's own `status.loadBalancer.ingress[0].ip`, i.e. clients reach the cluster endpoint.
|
||||
#
|
||||
# CLEANUP: chainsaw deletes the per-test namespace (Service included) at teardown;
|
||||
# ExternalDNS policy=sync then REMOVES the record from Cloudflare on its next pass.
|
||||
@@ -38,6 +45,13 @@ spec:
|
||||
apply: 90s
|
||||
assert: 5m
|
||||
exec: 6m
|
||||
# -- R-ENV-02: the environment's hostname suffix is READ FROM THE CLUSTER UNDER TEST.
|
||||
# flux-system/cluster-vars is the ConfigMap Flux substitutes ${BASE_DOMAIN} from, so this
|
||||
# binding is the same source of truth the platform is rendered from — dev asks dev, prod
|
||||
# asks prod, and there is no per-environment fork and no flag to forget.
|
||||
bindings:
|
||||
- name: baseDomain
|
||||
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
|
||||
steps:
|
||||
- name: given-a-service-requesting-a-dns-name
|
||||
try:
|
||||
@@ -48,7 +62,8 @@ spec:
|
||||
metadata:
|
||||
name: dns-probe-svc
|
||||
annotations:
|
||||
external-dns.alpha.kubernetes.io/hostname: conformance-dns.home.keithjackson.dev
|
||||
# conformance-dns.<this cluster's BASE_DOMAIN> — never a literal apex.
|
||||
external-dns.alpha.kubernetes.io/hostname: (join('.', ['conformance-dns', $baseDomain]))
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
# No real backend needed — Cilium LB-IPAM assigns a VIP regardless, and
|
||||
@@ -59,7 +74,27 @@ spec:
|
||||
- name: http
|
||||
port: 80
|
||||
targetPort: 80
|
||||
# Wait for Cilium LB-IPAM to actually assign the VIP: it is the value ExternalDNS will
|
||||
# publish and the value the probe below asserts EXACTLY. `|| ` + empty-array literal
|
||||
# keeps the expression nil-safe while the address is still absent, so the assert POLLS
|
||||
# (fails and retries) instead of erroring out on the first evaluation.
|
||||
- assert:
|
||||
resource:
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: dns-probe-svc
|
||||
status:
|
||||
(length(loadBalancer.ingress || `[]`)): 1
|
||||
- name: then-externaldns-creates-the-public-record
|
||||
# Step-scoped binding: read back the LB IP THIS Service was assigned (the previous step's
|
||||
# assert guarantees it exists by now — step bindings are evaluated when the step runs).
|
||||
# Expecting that exact address is both stricter than the old hardcoded 192.168.1.53-.60
|
||||
# range and correct in every environment: dev's pool is 192.168.1.53-.60, prod's is
|
||||
# 192.168.2.20-.30, and neither is written down here.
|
||||
bindings:
|
||||
- name: lbIp
|
||||
value: (x_k8s_get($client, 'v1', 'Service', $namespace, 'dns-probe-svc').status.loadBalancer.ingress[0].ip)
|
||||
try:
|
||||
- apply:
|
||||
resource:
|
||||
@@ -76,20 +111,27 @@ spec:
|
||||
containers:
|
||||
- name: probe
|
||||
image: busybox:1.36.1
|
||||
# Both values come from the cluster under test: the suffix from
|
||||
# flux-system/cluster-vars, the address from this Service's own status.
|
||||
env:
|
||||
- name: BASE_DOMAIN
|
||||
value: ($baseDomain)
|
||||
- name: LB_IP
|
||||
value: ($lbIp)
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- |
|
||||
set -u
|
||||
host="conformance-dns.home.keithjackson.dev"
|
||||
host="conformance-dns.${BASE_DOMAIN}"
|
||||
# Poll PUBLIC DNS (1.1.1.1) until the auto-created A record appears and
|
||||
# points into the cluster LB range (Cilium L2 `shared` pool 192.168.1.53-.60). Proves ExternalDNS created it.
|
||||
# points at THIS Service's assigned LB IP. Proves ExternalDNS created it.
|
||||
for i in $(seq 1 28); do
|
||||
ip="$(nslookup "$host" 1.1.1.1 2>/dev/null | grep -oE '192\.168\.1\.(5[3-9]|60)' | head -1)"
|
||||
ip="$(nslookup "$host" 1.1.1.1 2>/dev/null | awk '/^Address: /{print $2}' | grep -Fx "${LB_IP}" | head -1)"
|
||||
if [ -n "$ip" ]; then echo "RECORD_FOUND $host -> $ip"; exit 0; fi
|
||||
echo "poll $i: record not yet published"; sleep 10
|
||||
echo "poll $i: record not yet published as ${LB_IP}"; sleep 10
|
||||
done
|
||||
echo "FAIL: ExternalDNS did not publish the record within timeout"; exit 1
|
||||
echo "FAIL: ExternalDNS did not publish ${host} -> ${LB_IP} within timeout"; exit 1
|
||||
- assert:
|
||||
resource:
|
||||
apiVersion: batch/v1
|
||||
|
||||
@@ -2,17 +2,24 @@
|
||||
# L2 BEHAVIORAL — split-horizon INTERNAL resolution (R-DNS-01 / R-DNS-03) [RUNNABLE]
|
||||
# Catalog: docs/14 "DNS/exposure (R-DNS-*)"
|
||||
# Spec: docs/09 "Split-horizon resolution (R-DNS-03)" — internal clients resolve
|
||||
# <svc>.home.keithjackson.dev to the INTERNAL Gateway VIP, not the public hairpin.
|
||||
# <svc>.${BASE_DOMAIN} to the INTERNAL Gateway VIP, not the public hairpin.
|
||||
# Impl: platform/coredns/coredns-custom.yaml (k3s CoreDNS `import *.server`; template plugin
|
||||
# answers the whole home.keithjackson.dev zone with ${GATEWAY_VIP}).
|
||||
# answers the whole ${BASE_DOMAIN} zone with ${GATEWAY_VIP}).
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC (R-ENV-02): this test asks the CLUSTER IT IS POINTED AT for its own
|
||||
# ${BASE_DOMAIN} and ${GATEWAY_VIP} — the same flux-system/cluster-vars ConfigMap Flux
|
||||
# substitutes from — via test-level chainsaw bindings (see `spec.bindings`). Nothing here is
|
||||
# per-environment and no flag is passed at run time, so the suite cannot be run against the
|
||||
# wrong expectation: dev asserts dev's zone/VIP, prod asserts prod's.
|
||||
#
|
||||
# GIVEN the split-horizon coredns-custom ConfigMap is reconciled into k3s CoreDNS,
|
||||
# WHEN an in-cluster pod resolves a platform hostname over its PUBLIC name
|
||||
# (keycloak.home.keithjackson.dev) using the default cluster resolver (CoreDNS 10.43.0.10),
|
||||
# THEN the answer is the in-cluster Gateway VIP (the Cilium L2 `shared` gateway pool 192.168.1.53-.60) — NOT a
|
||||
# public/off-cluster IP. This is what lets pod->public-hostname traffic hairpin through the
|
||||
# Gateway with the correct SNI (*.home.keithjackson.dev) so the wildcard cert validates
|
||||
# (R-DNS-01 "services reachable at <svc>.home..."; R-DNS-03 split-horizon).
|
||||
# (keycloak.${BASE_DOMAIN}) using the default cluster resolver (CoreDNS 10.43.0.10),
|
||||
# THEN the answer is EXACTLY that cluster's ${GATEWAY_VIP} (the address coredns-custom is
|
||||
# templated with, and the `shared` Gateway's own status address) — NOT a public/off-cluster
|
||||
# IP. This is what lets pod->public-hostname traffic hairpin through the
|
||||
# Gateway with the correct SNI (*.${BASE_DOMAIN}) so the wildcard cert validates
|
||||
# (R-DNS-01 "services reachable at <svc>.${BASE_DOMAIN}"; R-DNS-03 split-horizon).
|
||||
#
|
||||
# Complements dns-exposure (R-DNS-02, the EXTERNAL/public-record side). Together they cover the
|
||||
# split-horizon pair: external answer = public record (R-DNS-02); internal answer = Gateway VIP
|
||||
@@ -34,13 +41,23 @@ metadata:
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
An in-cluster pod resolving keycloak.home.keithjackson.dev via cluster DNS gets the internal
|
||||
Gateway VIP (192.168.1.53-.60), not the public hairpin — split-horizon in-cluster resolution.
|
||||
An in-cluster pod resolving keycloak.${BASE_DOMAIN} via cluster DNS gets that cluster's
|
||||
internal Gateway VIP, not the public hairpin — split-horizon in-cluster resolution.
|
||||
spec:
|
||||
timeouts:
|
||||
apply: 60s
|
||||
assert: 2m
|
||||
exec: 2m
|
||||
# -- R-ENV-02: the environment's identity is READ FROM THE CLUSTER UNDER TEST, never authored
|
||||
# here. flux-system/cluster-vars is the ConfigMap Flux itself substitutes from
|
||||
# (clusters/base/cluster-vars.yaml + the dev/prod overlay patch), so these bindings are the
|
||||
# same source of truth the platform is rendered from. Test-level => in scope for every step
|
||||
# (unlike step `outputs`, which are step-scoped).
|
||||
bindings:
|
||||
- name: baseDomain
|
||||
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
|
||||
- name: gatewayVip
|
||||
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.GATEWAY_VIP)
|
||||
steps:
|
||||
- name: then-cluster-dns-answers-with-the-internal-gateway-vip
|
||||
try:
|
||||
@@ -59,23 +76,32 @@ spec:
|
||||
containers:
|
||||
- name: probe
|
||||
image: busybox:1.36.1
|
||||
# The cluster's OWN identity, resolved from flux-system/cluster-vars by the
|
||||
# test-level bindings — never a literal (R-ENV-02).
|
||||
env:
|
||||
- name: BASE_DOMAIN
|
||||
value: ($baseDomain)
|
||||
- name: GATEWAY_VIP
|
||||
value: ($gatewayVip)
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- |
|
||||
set -u
|
||||
host="keycloak.home.keithjackson.dev"
|
||||
host="keycloak.${BASE_DOMAIN}"
|
||||
# Resolve via the DEFAULT cluster resolver (CoreDNS, /etc/resolv.conf ->
|
||||
# 10.43.0.10). Split-horizon must answer with the in-cluster Gateway VIP
|
||||
# (Cilium L2 `shared` pool 192.168.1.53-.60), proving pods hairpin through the Gateway.
|
||||
# 10.43.0.10). Split-horizon must answer with THIS cluster's Gateway VIP
|
||||
# (the address coredns-custom is templated with), proving pods hairpin
|
||||
# through the Gateway. Exact match on ${GATEWAY_VIP}: stricter than the
|
||||
# old hardcoded 192.168.1.53-.60 range AND correct in both environments.
|
||||
out="$(nslookup "$host" 2>/dev/null)"
|
||||
echo "$out"
|
||||
ip="$(echo "$out" | awk '/^Address: /{print $2}' | grep -E '^192\.168\.1\.(5[3-9]|60)$' | head -1)"
|
||||
ip="$(echo "$out" | awk '/^Address: /{print $2}' | grep -Fx "${GATEWAY_VIP}" | head -1)"
|
||||
if [ -n "$ip" ]; then
|
||||
echo "INTERNAL_RESOLUTION_OK $host -> $ip (in-cluster Gateway VIP)"
|
||||
exit 0
|
||||
fi
|
||||
echo "FAIL: $host did not resolve to the in-cluster Gateway VIP (192.168.1.53-.60);"
|
||||
echo "FAIL: $host did not resolve to this cluster's Gateway VIP (${GATEWAY_VIP});"
|
||||
echo " split-horizon coredns-custom not in effect (got a public/off-cluster answer)."
|
||||
exit 1
|
||||
- assert:
|
||||
|
||||
@@ -0,0 +1,199 @@
|
||||
# =============================================================================
|
||||
# L2 BEHAVIORAL — the default egress path is the WAN; the VPN is opt-in (R-NET-04) [RUNNABLE]
|
||||
# Catalog: docs/14 L2 row ("real runtime behavior, probes against a cluster")
|
||||
# Spec: docs/00 R-NET-04 ("The platform MUST support **multiple egress paths** selected by
|
||||
# routing rules ... realized as two paths — the local `internet` path and the
|
||||
# in-cluster `vpn` path").
|
||||
# docs/03 §"Egress routing: two paths, one physical WAN": "the `internet` path is
|
||||
# simply the default route out".
|
||||
# docs/03 §"VPN egress model": "**Default is the WAN; the VPN is strictly opt-in.**
|
||||
# Every pod — and the nodes themselves — egress the local/private WAN by default and
|
||||
# **never** the VPN. Only services labeled `egress-class=vpn` are steered to the
|
||||
# tunnel"; and the tunnel "is **split-tunnel**: ... only what Cilium points at `tun0`
|
||||
# uses the VPN."
|
||||
#
|
||||
# WHAT THIS PROVES THAT tests/l2-behavioral/egress-class DOES NOT
|
||||
# --------------------------------------------------------------
|
||||
# That test (R-NET-05/07) compares two LABELLED pods — `internet` vs `vpn` — and shows their
|
||||
# public IPs differ, then flips a label. Both of its pods declared a class, so it says nothing
|
||||
# about the case R-NET-04 actually turns on: a pod that NO routing rule selects. This test adds
|
||||
# that pod, and asserts the two claims that make a rule-selected path set complete:
|
||||
#
|
||||
# TOTALITY — an UNLABELLED pod egresses the WAN, i.e. the same public IP as the
|
||||
# explicitly-`internet` pod. Absence of a matching rule IS a path, and it is
|
||||
# the local one. If the steering rule ever widened (an empty selector, a
|
||||
# matchExpression, a second rule), this pod's IP would move to the tunnel and
|
||||
# this assertion is what would notice.
|
||||
# COEXISTENCE — the WAN path and the VPN path are live AT THE SAME TIME, observed in one
|
||||
# pass: three pods, two distinct public IPs. "Multiple egress paths" is a
|
||||
# simultaneity claim, not a claim that one switch has two positions.
|
||||
# PER-DESTINATION SELECTION — the vpn-class pod still reaches an in-cluster/RFC1918
|
||||
# destination over the LOCAL path while its public traffic is tunnelled. The
|
||||
# tunnel is destination-scoped (excludedCIDRs), not a default route, so ONE pod
|
||||
# uses BOTH paths at once.
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC (R-ENV-02): no address, hostname or range appears. Every assertion is a
|
||||
# RELATION between observed values (equal / not-equal), so it holds against dev and prod alike
|
||||
# and survives the WAN or the VPN provider rotating addresses.
|
||||
#
|
||||
# GIVEN one unlabelled, one `internet`-class and one `vpn`-class pod — the same workload three
|
||||
# times, differing only in the class they declare (or do not),
|
||||
# WHEN each asks an external echo for the public IP it egressed from, and the vpn-class pod
|
||||
# also resolves a name through the in-cluster resolver,
|
||||
# THEN the unlabelled pod's public IP EQUALS the internet-class pod's (the default route out
|
||||
# is the WAN, and the VPN is strictly opt-in);
|
||||
# AND the vpn-class pod's public IP DIFFERS from both (a second, simultaneously-live path);
|
||||
# AND the vpn-class pod still resolves in-cluster (its private-destination traffic was not
|
||||
# swallowed by the tunnel), so the same pod is using both paths at once.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them):
|
||||
# - THAT NO OTHER RULE EXISTS. Three pods cannot enumerate a rule set. That exactly one
|
||||
# steering rule is authored, that it selects only the vpn class, that no `ip rule` is
|
||||
# unscoped and no nftables DROP unqualified is the L0 half, in
|
||||
# tests/l0-static/egress-paths-selected-by-rules (same requirement label).
|
||||
# - WHICH INTERFACE the packets left by. The observation is the public IP an external echo
|
||||
# reports; it distinguishes the paths without naming them. Asserting "out tun0" would need
|
||||
# host-level datapath inspection (Hubble/tcpdump on the gateway node), which no test in
|
||||
# this suite does.
|
||||
# - THE KILL-SWITCH. Tunnel-down behaviour is R-NET-06's, at L3 (vpn-killswitch): `vpn`
|
||||
# fails closed. Its mirror — that `internet` is never blackholed — is asserted statically
|
||||
# by the L0 half, not here.
|
||||
# - NODE traffic. docs/03 also says "the nodes themselves" egress the WAN by default. That
|
||||
# is a host-level fact this test cannot see from a pod.
|
||||
# - EXTERNAL DEPENDENCY, stated plainly: the public-IP echo (api.ipify.org) is off-cluster,
|
||||
# so a WAN outage or the echo being down fails this test. That is the same dependency
|
||||
# tests/l2-behavioral/egress-class already carries; the alternative — trusting the
|
||||
# cluster's own account of which path it used — would not be a behavioural test at all.
|
||||
#
|
||||
# CLEANUP: chainsaw deletes the per-test namespace (all three pods with it). Nothing outside
|
||||
# that namespace is created or changed — no label on a platform object, no policy edit.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: egress-default-path
|
||||
labels:
|
||||
conformance.home.k3s/R-NET-04: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-NET-04"
|
||||
conformance.home.k3s/layer: "L2"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Given an unlabelled, an internet-class and a vpn-class pod, when each observes the public
|
||||
IP it egressed from, then the unlabelled pod's IP equals the internet-class pod's (the
|
||||
default route out is the WAN, the VPN is opt-in) while the vpn-class pod's differs — two
|
||||
paths live at once — and the vpn-class pod still reaches in-cluster destinations over the
|
||||
local path, so one pod uses both paths simultaneously.
|
||||
spec:
|
||||
timeouts:
|
||||
apply: 60s
|
||||
assert: 2m
|
||||
exec: 3m
|
||||
steps:
|
||||
# -----------------------------------------------------------------------------------
|
||||
- name: given-three-pods-differing-only-in-their-declared-egress-class
|
||||
try:
|
||||
- apply:
|
||||
file: fixtures/three-egress-pods.yaml
|
||||
# Bare pods (restartPolicy: Never) — Ready == Running here.
|
||||
- assert:
|
||||
resource:
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata: { name: egress-unlabelled }
|
||||
status: { phase: Running }
|
||||
- assert:
|
||||
resource:
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata: { name: egress-internet }
|
||||
status: { phase: Running }
|
||||
- assert:
|
||||
resource:
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata: { name: egress-vpn }
|
||||
status: { phase: Running }
|
||||
# -----------------------------------------------------------------------------------
|
||||
- name: then-the-unlabelled-pod-takes-the-wan-and-the-vpn-path-is-live-too
|
||||
try:
|
||||
- script:
|
||||
content: |
|
||||
set -eu
|
||||
ns="$NAMESPACE"
|
||||
get_ip() {
|
||||
# Echo the pod's observed public IP, or empty on failure.
|
||||
kubectl -n "$ns" exec "$1" -c curl -- \
|
||||
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true
|
||||
}
|
||||
|
||||
# Poll until all three answer AND the vpn class has diverged from the WAN:
|
||||
# Cilium may still be programming the egress datapath just after the pods go
|
||||
# Ready. Bounded by the loop, never a bare sleep-and-hope.
|
||||
none="" inet="" vpn=""
|
||||
for i in $(seq 1 12); do
|
||||
[ -n "$none" ] || none="$(get_ip egress-unlabelled)"
|
||||
[ -n "$inet" ] || inet="$(get_ip egress-internet)"
|
||||
[ -n "$vpn" ] || vpn="$(get_ip egress-vpn)"
|
||||
if [ -n "$none" ] && [ -n "$inet" ] && [ -n "$vpn" ]; then
|
||||
[ "$vpn" != "$inet" ] && break
|
||||
# Steering may not have converged; re-read the vpn pod only.
|
||||
vpn=""
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
echo "unlabelled_ip=$none internet_ip=$inet vpn_ip=$vpn"
|
||||
|
||||
[ -n "$none" ] || { echo "FAIL: the UNLABELLED pod never returned a public IP — a pod that declares no egress class must still reach the internet over the default WAN path (docs/03: 'the internet path is simply the default route out'); no path at all means the default was blackholed or steered into a dead tunnel" >&2; exit 1; }
|
||||
[ -n "$inet" ] || { echo "FAIL: the internet-class pod never returned a public IP — the declared default path is down, so no comparison is possible" >&2; exit 1; }
|
||||
[ -n "$vpn" ] || { echo "FAIL: the vpn-class pod never returned a public IP — the second egress path is not carrying traffic, so the platform is not offering MULTIPLE paths (R-NET-04)" >&2; exit 1; }
|
||||
|
||||
# TOTALITY: no rule selects the unlabelled pod, so it must land on the WAN —
|
||||
# the same WAN the explicitly-internet pod uses.
|
||||
if [ "$none" != "$inet" ]; then
|
||||
echo "FAIL: the UNLABELLED pod egressed $none but the internet-class pod egressed $inet. They must be the same path: docs/03 — 'Default is the WAN; the VPN is strictly opt-in. Every pod ... egress the local/private WAN by default and never the VPN.' A different IP means some rule is selecting a pod that never opted in." >&2
|
||||
exit 1
|
||||
fi
|
||||
# COEXISTENCE: and the vpn path is a genuinely different path, live right now.
|
||||
if [ "$vpn" = "$inet" ]; then
|
||||
echo "FAIL: the vpn-class pod egressed the same public IP as the WAN ($vpn) — there is only ONE live egress path, so 'multiple egress paths selected by routing rules' (R-NET-04) does not hold" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "EGRESS_DEFAULT_PATH_IS_WAN_OK"
|
||||
check:
|
||||
(contains($stdout, 'EGRESS_DEFAULT_PATH_IS_WAN_OK')): true
|
||||
# -----------------------------------------------------------------------------------
|
||||
- name: then-the-vpn-class-pod-still-reaches-private-destinations-locally
|
||||
try:
|
||||
- script:
|
||||
# PER-DESTINATION SELECTION. The steering rule tunnels 0.0.0.0/0 MINUS all
|
||||
# RFC1918 space, so a vpn-class pod's private-destination traffic must stay on the
|
||||
# local datapath. The cluster resolver is an in-cluster (RFC1918) address reached
|
||||
# from the pod's own /etc/resolv.conf — no literal is written here — so a
|
||||
# successful lookup from the vpn-class pod proves the tunnel did not swallow its
|
||||
# local traffic, i.e. ONE pod is using BOTH paths at the same time.
|
||||
content: |
|
||||
set -eu
|
||||
ns="$NAMESPACE"
|
||||
resolver="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
|
||||
sh -c 'awk "/^nameserver/{print \$2; exit}" /etc/resolv.conf' 2>/dev/null || true)"
|
||||
[ -n "$resolver" ] || { echo "FAIL: the vpn-class pod has no nameserver in /etc/resolv.conf, so its local-path reachability cannot be observed" >&2; exit 1; }
|
||||
echo "vpn_pod_resolver=$resolver (in-cluster, RFC1918 — excluded from the tunnel)"
|
||||
|
||||
# A cluster-internal name, resolvable only by the in-cluster resolver: reaching
|
||||
# it proves the private-destination path works from a vpn-class pod.
|
||||
out=""
|
||||
for i in $(seq 1 6); do
|
||||
out="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
|
||||
nslookup kubernetes.default.svc.cluster.local 2>/dev/null || true)"
|
||||
case "$out" in *"Address"*) break ;; esac
|
||||
sleep 5
|
||||
done
|
||||
case "$out" in
|
||||
*Address*) : ;;
|
||||
*) echo "FAIL: the vpn-class pod could not resolve kubernetes.default.svc.cluster.local via its in-cluster resolver ($resolver). Its RFC1918 traffic is being steered into the tunnel (or blackholed) instead of taking the local path — the two paths are not coexisting per destination, and the tunnel is behaving as a default route (docs/03: split-tunnel, 'only what Cilium points at tun0 uses the VPN')" >&2; exit 1 ;;
|
||||
esac
|
||||
echo "EGRESS_PATHS_COEXIST_PER_DESTINATION_OK"
|
||||
check:
|
||||
(contains($stdout, 'EGRESS_PATHS_COEXIST_PER_DESTINATION_OK')): true
|
||||
@@ -0,0 +1,67 @@
|
||||
# Three pods for the R-NET-04 default-path test — the SAME workload three times, differing
|
||||
# ONLY in how (or whether) they declare an egress class:
|
||||
#
|
||||
# egress-unlabelled NO egress-class label at all -> selected by NO routing rule
|
||||
# egress-internet egress-class: internet -> the declared default path
|
||||
# egress-vpn egress-class: vpn -> the only class any rule steers
|
||||
#
|
||||
# WHY AN UNLABELLED POD IS THE POINT. docs/03: "Every pod — and the nodes themselves — egress
|
||||
# the local/private WAN by default and **never** the VPN." The unlabelled pod is the case no
|
||||
# other test covers: nothing selects it, so the DATAPATH's default must carry it, and it must
|
||||
# come out the same WAN as the explicitly-`internet` pod. That is the totality half of
|
||||
# R-NET-04's "selected by routing rules" — absence of a rule is itself a path.
|
||||
#
|
||||
# The unlabelled pod is admissible here only because chainsaw's per-test namespace is not an
|
||||
# app namespace: Kyverno's require-egress-class-label (R-NET-05) rejects an unlabelled pod in
|
||||
# a namespace labelled homelab.keithjackson.dev/app-namespace=true. That admission guarantee
|
||||
# is R-NET-05's, and it is exactly why this test must run outside an app namespace to observe
|
||||
# the datapath's own default at all.
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: egress-unlabelled
|
||||
labels:
|
||||
app: egress-default-path-test
|
||||
spec:
|
||||
containers:
|
||||
- name: curl
|
||||
image: curlimages/curl:8.11.1
|
||||
command: ["sleep", "3600"]
|
||||
resources:
|
||||
requests: { cpu: 10m, memory: 16Mi }
|
||||
limits: { memory: 32Mi }
|
||||
restartPolicy: Never
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: egress-internet
|
||||
labels:
|
||||
app: egress-default-path-test
|
||||
homelab.keithjackson.dev/egress-class: internet
|
||||
spec:
|
||||
containers:
|
||||
- name: curl
|
||||
image: curlimages/curl:8.11.1
|
||||
command: ["sleep", "3600"]
|
||||
resources:
|
||||
requests: { cpu: 10m, memory: 16Mi }
|
||||
limits: { memory: 32Mi }
|
||||
restartPolicy: Never
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: egress-vpn
|
||||
labels:
|
||||
app: egress-default-path-test
|
||||
homelab.keithjackson.dev/egress-class: vpn
|
||||
spec:
|
||||
containers:
|
||||
- name: curl
|
||||
image: curlimages/curl:8.11.1
|
||||
command: ["sleep", "3600"]
|
||||
resources:
|
||||
requests: { cpu: 10m, memory: 16Mi }
|
||||
limits: { memory: 32Mi }
|
||||
restartPolicy: Never
|
||||
@@ -68,6 +68,15 @@ spec:
|
||||
timeouts:
|
||||
apply: 60s
|
||||
assert: 180s
|
||||
# -- R-ENV-02: the fixture's route hostnames MUST fall under the Gateway's wildcard listener
|
||||
# (*.${BASE_DOMAIN}) or they never attach — so the suffix is READ FROM THE CLUSTER UNDER TEST
|
||||
# (flux-system/cluster-vars, the ConfigMap Flux substitutes ${BASE_DOMAIN} from) rather than
|
||||
# authored here. Test-level => in scope for the fixture apply AND every probe step below
|
||||
# (step `outputs`, by contrast, are step-scoped — see the gwvip re-capture comments).
|
||||
# fixtures/app-and-probe.yaml consumes it via chainsaw resource templating.
|
||||
bindings:
|
||||
- name: baseDomain
|
||||
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
|
||||
steps:
|
||||
# -- GIVEN: the exposed-app fixture (default-deny + gateway-ingress allow + backend +
|
||||
# Service + declared/undeclared HTTPRoutes) and a probe namespace. Assert the shared
|
||||
@@ -139,12 +148,14 @@ spec:
|
||||
env:
|
||||
- name: GWVIP
|
||||
value: ($gwvip)
|
||||
- name: BASE_DOMAIN
|
||||
value: ($baseDomain)
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- |
|
||||
set -u
|
||||
host="gw-declared-conformance.home.keithjackson.dev"
|
||||
host="gw-declared-conformance.${BASE_DOMAIN}"
|
||||
last=000
|
||||
i=0
|
||||
while [ "$i" -lt 30 ]; do
|
||||
@@ -207,12 +218,14 @@ spec:
|
||||
env:
|
||||
- name: GWVIP
|
||||
value: ($gwvip)
|
||||
- name: BASE_DOMAIN
|
||||
value: ($baseDomain)
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- |
|
||||
set -u
|
||||
host="gw-undeclared-conformance.home.keithjackson.dev"
|
||||
host="gw-undeclared-conformance.${BASE_DOMAIN}"
|
||||
reached_gw=no
|
||||
i=0
|
||||
while [ "$i" -lt 8 ]; do
|
||||
@@ -300,3 +313,20 @@ spec:
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
name: conformance-gw-probe
|
||||
# -----------------------------------------------------------------------------------
|
||||
# PROOF-OF-PASS SENTINEL (R-SEC-03). Every step above is an assert/Job-status gate, so
|
||||
# chainsaw already fails the test before reaching this step if any of them fails — but a
|
||||
# green Chainsaw JUnit carries NO stdout, and the L2/L3 tests in this suite can self-skip
|
||||
# GREEN, so lib/record_run.py refuses to treat a bare `passed` as evidence and records
|
||||
# `unknown-no-sentinel-declared` instead. Without a declared token this test could pass
|
||||
# forever and NEVER become a recorded proof of R-NET-01/02/03 — north-south reachability under default-deny, part of the isolation proof
|
||||
# — which is precisely what docs/07 asks for ("These proofs are part of the platform
|
||||
# acceptance suite, not one-time checks"). This step exists only to emit that token, and it
|
||||
# is reached only when everything above has passed. Asserted by
|
||||
# tests/l0-static/auditability-proofs-producible.
|
||||
- name: then-emit-the-proof-of-pass-sentinel
|
||||
try:
|
||||
- script:
|
||||
content: echo GATEWAY_INGRESS_ISOLATION_PROVEN_OK
|
||||
check:
|
||||
(contains($stdout, 'GATEWAY_INGRESS_ISOLATION_PROVEN_OK')): true
|
||||
+11
-2
@@ -10,6 +10,13 @@
|
||||
# The namespace opts into the shared Gateway (gateway-access: shared) exactly like a
|
||||
# rendered app namespace, so its HTTPRoutes attach to the real platform Gateway and the
|
||||
# probe exercises the true north-south path (Gateway Envoy -> backend), not a stand-in.
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC HOSTNAMES (R-ENV-02): the two HTTPRoute hostnames are NOT literals. They
|
||||
# are chainsaw-templated from the `baseDomain` binding declared in ../chainsaw-test.yaml, which
|
||||
# reads BASE_DOMAIN out of the flux-system/cluster-vars ConfigMap of the cluster under test — the
|
||||
# same ConfigMap Flux substitutes ${BASE_DOMAIN} from. They MUST fall under the shared Gateway's
|
||||
# wildcard listener (*.${BASE_DOMAIN}) or the routes never attach, which is precisely why a
|
||||
# hardcoded suffix silently broke this test on any cluster that did not hold the apex.
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
@@ -134,7 +141,8 @@ spec:
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- gw-declared-conformance.home.keithjackson.dev
|
||||
# gw-declared-conformance.<this cluster's BASE_DOMAIN> — see the header.
|
||||
- (join('.', ['gw-declared-conformance', $baseDomain]))
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
@@ -158,7 +166,8 @@ spec:
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- gw-undeclared-conformance.home.keithjackson.dev
|
||||
# gw-undeclared-conformance.<this cluster's BASE_DOMAIN> — see the header.
|
||||
- (join('.', ['gw-undeclared-conformance', $baseDomain]))
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
|
||||
@@ -0,0 +1,379 @@
|
||||
# =============================================================================
|
||||
# L2 BEHAVIORAL — the cluster under test IS an HA k3s control plane (R-PLAT-01) [RUNNABLE]
|
||||
# Catalog: docs/14 L2 row ("real runtime behavior, probes against a cluster")
|
||||
# Spec: docs/00 R-PLAT-01 ("The platform MUST run on k3s across 3 nodes with
|
||||
# production-grade HA control plane in prod; dev/test MAY run fewer nodes but MUST
|
||||
# exercise the same multi-node topology.")
|
||||
# docs/01 "Kubernetes distro | **k3s** (HA, embedded etcd) | R-PLAT-01/02"
|
||||
# docs/02 table: "Control plane | HA embedded etcd (k3s server on all 3) | HA
|
||||
# embedded etcd (k3s server on all 3)"
|
||||
# "Node count | 3 VMs (mirror prod topology) | 3 physical nodes"
|
||||
# docs/02 §"k3s topology decisions": "HA control plane with **embedded etcd** across
|
||||
# all 3 nodes (no external datastore)"; "Nodes are both control-plane and workload
|
||||
# (3-node converged cluster)".
|
||||
#
|
||||
# WHY L2 — AND WHAT L0 ALREADY DID
|
||||
# --------------------------------
|
||||
# tests/l0-static/ha-control-plane-as-code (same requirement label) proves the AUTHORED
|
||||
# half offline: the repo builds a 3-server embedded-etcd control plane in both environments,
|
||||
# one founder, no external datastore, a leader-elected control-plane-only VIP in the serving
|
||||
# cert's SANs. It cannot prove that any of it is TRUE OF A RUNNING CLUSTER — an inventory
|
||||
# entry is not a node, and a templated DaemonSet is not a floating VIP. That is this test:
|
||||
# every assertion below is read from the cluster the suite is pointed at.
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC (R-ENV-02): no address, hostname, node count or range is written
|
||||
# here. The expected node count comes from the cluster's OWN declared identity
|
||||
# (flux-system/cluster-vars CLUSTER_ENV, via a test-level chainsaw binding — the same
|
||||
# ConfigMap Flux substitutes from); the control-plane VIP is read out of the kube-vip
|
||||
# DaemonSet the cluster is actually running; the server addresses come from the Node
|
||||
# objects; and the LoadBalancer pool the VIP must stay outside of is read from cluster-vars
|
||||
# too. R-PLAT-01 itself distinguishes prod from dev/test ("3 nodes ... in prod; dev/test MAY
|
||||
# run fewer"), so the assertion is necessarily environment-AWARE — but it takes the
|
||||
# environment from the cluster, never from a flag or a literal, which is precisely the
|
||||
# mechanism the tests/README.md convention prescribes.
|
||||
#
|
||||
# GIVEN the cluster this suite is pointed at,
|
||||
# WHEN its Nodes, its kube-vip DaemonSet and its kube-apiserver endpoints are inspected,
|
||||
# THEN DISTRO: every node's kubelet is a k3s build (`+k3s`) — the distro really is k3s;
|
||||
# SERVERS: every Ready node is BOTH control-plane AND etcd (the converged 3-node
|
||||
# topology docs/02 fixes — no agent-only node, no control-plane-only node),
|
||||
# the count is ODD and quorum-capable (tolerates >= 1 loss), and it is
|
||||
# EXACTLY three when the cluster declares itself prod;
|
||||
# ETCD: every one of those nodes is a REGISTERED embedded-etcd member (k3s stamps
|
||||
# `etcd.k3s.cattle.io/node-name` on a node that joined the etcd cluster), so
|
||||
# the datastore is embedded and replicated across all of them;
|
||||
# VIP: kube-vip is Ready on EVERY control-plane node, is control-plane-only
|
||||
# (svc_enable=false — it must not contend with Cilium LB-IPAM for ARP), and
|
||||
# the address it floats is neither a node address, nor the ingress Gateway
|
||||
# VIP, nor inside the Cilium LoadBalancer pool;
|
||||
# ENDPOINT: the kube-apiserver answers `/readyz` THROUGH THAT VIP over TLS that
|
||||
# validates against the cluster CA for the VIP itself — i.e. the VIP really
|
||||
# fronts the API and the serving cert really carries it;
|
||||
# EVERY SERVER SERVES: the apiserver also answers `/readyz` on EVERY server's own
|
||||
# address, again with a CA-valid cert for that address. This is the assertion
|
||||
# that separates a real HA control plane from one server plus two joiners:
|
||||
# if only the founder ran an apiserver, the other addresses would refuse.
|
||||
#
|
||||
# THE PROBE RUNS IN-CLUSTER, DELIBERATELY. An earlier draft dialled the VIP and the node
|
||||
# addresses from the RUNNER. That works from a host on the node LAN (verified against prod)
|
||||
# and FAILS against dev, whose nodes sit behind a NAT the runner reaches only via a
|
||||
# forwarded port — i.e. it would have made the test a property of where it was launched
|
||||
# from. A Job on the cluster is on the node network in both environments, so the probe is
|
||||
# the same probe everywhere. The Job is created imperatively (kubectl heredoc) rather than
|
||||
# as a fixture file because its address list is DISCOVERED from the cluster at run time —
|
||||
# writing it into a fixture is exactly the per-environment literal R-ENV-02 forbids.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them):
|
||||
# - FAILOVER / SURVIVAL. Nothing here removes a node. That etcd keeps quorum, that the VIP
|
||||
# migrates, and that the API stays reachable when a server dies is an L3 fault-injection
|
||||
# claim and is proven NOWHERE in this suite yet (tests/l3-fault-injection/node-loss
|
||||
# drains a node for R-STATE-02's data-availability claim and asserts nothing about the
|
||||
# control plane). This is the largest remaining R-PLAT-01 gap.
|
||||
# - THE NEXT environment. That the repo would build an HA control plane again is the L0
|
||||
# half (tests/l0-static/ha-control-plane-as-code).
|
||||
# - "PRODUCTION-GRADE" as a capacity/performance judgement (operator's, docs/02 + §2 of
|
||||
# infra/PROD-PROVISIONING.md).
|
||||
# - THAT THE RUNNING cilium-agents ACTUALLY DIAL THIS VIP. As of 2026-09-06 the repository
|
||||
# authors `K3S_API_HOST` = the control-plane VIP in both environments (asserted at L0 by
|
||||
# tests/l0-static/ha-control-plane-as-code, on both sides of the [parity] contract), but
|
||||
# this test does not read the agents' live `KUBERNETES_SERVICE_HOST`. Until the operator
|
||||
# applies that change the LIVE agents still carry the PRIMARY NODE's address — measured
|
||||
# 2026-09-06: all three prod agents on 192.168.2.10 while kube-vip held 192.168.2.9. An
|
||||
# in-cluster assertion on the agents' endpoint belongs here and is NOT yet written; it is
|
||||
# the obvious follow-up once the change is applied and would catch drift the L0 check
|
||||
# cannot see (a hand-edited DaemonSet, a Helm value overridden out of band).
|
||||
# - etcd health beyond membership registration (raft leader, DB size, alarms) — k3s exposes
|
||||
# it only on the node-local etcd metrics port, which is not reachable through the
|
||||
# Kubernetes API this test is allowed to use.
|
||||
#
|
||||
# CLEANUP: the probe Job lives in chainsaw's per-test namespace and is deleted with it
|
||||
# (config/chainsaw-config.yaml skipDelete: false). Everything else is read-only: this test
|
||||
# creates nothing outside its own namespace and mutates no platform object.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: ha-control-plane
|
||||
labels:
|
||||
conformance.home.k3s/R-PLAT-01: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-PLAT-01"
|
||||
conformance.home.k3s/layer: "L2"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Given the cluster under test, when its Nodes, kube-vip DaemonSet and apiserver
|
||||
endpoints are probed, then every node is a k3s server that is both control-plane and a
|
||||
registered embedded-etcd member, the count is odd and quorum-capable (exactly three
|
||||
when the cluster declares itself prod), kube-vip floats a control-plane-only VIP that
|
||||
collides with no allocator, and the apiserver answers /readyz through that VIP AND on
|
||||
every server's own address with a CA-valid cert.
|
||||
spec:
|
||||
timeouts:
|
||||
apply: 60s
|
||||
assert: 2m
|
||||
exec: 3m
|
||||
# -- R-ENV-02: the environment's identity and its address allocations are READ FROM THE
|
||||
# CLUSTER UNDER TEST. flux-system/cluster-vars is the ConfigMap Flux itself substitutes
|
||||
# from, so these are the same values the platform was rendered with. Test-level, so
|
||||
# they are in scope for every step (step outputs/bindings are step-scoped).
|
||||
bindings:
|
||||
- name: clusterEnv
|
||||
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.CLUSTER_ENV)
|
||||
- name: gatewayVip
|
||||
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.GATEWAY_VIP)
|
||||
- name: lbPoolStart
|
||||
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.CILIUM_LB_POOL_START)
|
||||
- name: lbPoolStop
|
||||
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.CILIUM_LB_POOL_STOP)
|
||||
steps:
|
||||
# -----------------------------------------------------------------------------------
|
||||
- name: then-the-node-set-is-a-quorum-capable-converged-k3s-control-plane
|
||||
description: >-
|
||||
Read the Node objects: k3s on every node, every node both control-plane and a
|
||||
registered embedded-etcd member, an odd quorum-capable count, exactly three in prod.
|
||||
try:
|
||||
- script:
|
||||
env:
|
||||
- name: CLUSTER_ENV
|
||||
value: ($clusterEnv)
|
||||
content: |
|
||||
set -eu
|
||||
fail() { echo "FAIL: $*" >&2; exit 1; }
|
||||
|
||||
total="$(kubectl get nodes -o name | grep -c . || true)"
|
||||
[ "$total" -ge 1 ] || fail "the cluster reports NO nodes at all"
|
||||
|
||||
# --- k3s, on every node (R-PLAT-01 names the distro; docs/01 ADR-0001) -----
|
||||
# A non-k3s kubelet here would mean the platform is not running on k3s.
|
||||
notk3s="$(kubectl get nodes \
|
||||
-o jsonpath='{range .items[*]}{.metadata.name}{"="}{.status.nodeInfo.kubeletVersion}{"\n"}{end}' \
|
||||
| grep -v '+k3s' | grep -c . || true)"
|
||||
[ "$notk3s" = "0" ] \
|
||||
|| fail "$notk3s node(s) do not run a k3s kubelet — R-PLAT-01 requires the platform to run on k3s"
|
||||
|
||||
# --- every node is BOTH control-plane AND etcd (docs/02 converged topology) -
|
||||
# Counted by label EXISTENCE, so it does not matter whether k3s writes "true"
|
||||
# or "". A control-plane node without the etcd role would be a server with an
|
||||
# external/absent datastore; an etcd node without control-plane would be an
|
||||
# isolated datastore member. Both are excluded by requiring all three counts
|
||||
# to be equal.
|
||||
cp="$(kubectl get nodes -l node-role.kubernetes.io/control-plane -o name | grep -c . || true)"
|
||||
etcd="$(kubectl get nodes -l node-role.kubernetes.io/etcd -o name | grep -c . || true)"
|
||||
echo "nodes_total=$total control_plane=$cp etcd=$etcd cluster_env=${CLUSTER_ENV:-<unset>}"
|
||||
[ "$cp" = "$total" ] \
|
||||
|| fail "$cp of $total nodes carry node-role.kubernetes.io/control-plane — docs/02 fixes a converged cluster where every node is a control-plane member ('k3s server on all 3')"
|
||||
[ "$etcd" = "$total" ] \
|
||||
|| fail "$etcd of $total nodes carry node-role.kubernetes.io/etcd — docs/02 requires embedded etcd 'across all 3 nodes (no external datastore)'"
|
||||
|
||||
# --- every node is a REGISTERED embedded-etcd member -----------------------
|
||||
# k3s stamps etcd.k3s.cattle.io/node-name on a node that actually joined the
|
||||
# embedded-etcd cluster. The role label alone is a label; this annotation is
|
||||
# the datastore's own record of membership.
|
||||
members="$(kubectl get nodes \
|
||||
-o jsonpath='{range .items[*]}{.metadata.annotations.etcd\.k3s\.cattle\.io/node-name}{"\n"}{end}' \
|
||||
| grep -c . || true)"
|
||||
[ "$members" = "$total" ] \
|
||||
|| fail "$members of $total nodes are registered embedded-etcd members (etcd.k3s.cattle.io/node-name) — the datastore is not replicated across every server"
|
||||
|
||||
# --- Ready ---------------------------------------------------------------
|
||||
notready="$(kubectl get nodes \
|
||||
-o jsonpath='{range .items[*]}{.metadata.name}{"="}{range .status.conditions[?(@.type=="Ready")]}{.status}{end}{"\n"}{end}' \
|
||||
| grep -v '=True$' | grep -c . || true)"
|
||||
[ "$notready" = "0" ] \
|
||||
|| fail "$notready node(s) are not Ready — a control plane missing a member is not the HA topology under test"
|
||||
|
||||
# --- quorum arithmetic ---------------------------------------------------
|
||||
# Embedded etcd needs floor(N/2)+1 members to commit. Fault tolerance is
|
||||
# N - quorum; 'production-grade HA' must survive at least one loss, which
|
||||
# rules out 1 and 2 members. An EVEN count buys no extra tolerance and adds
|
||||
# a tie, so it is also rejected.
|
||||
quorum=$(( total / 2 + 1 ))
|
||||
tolerance=$(( total - quorum ))
|
||||
echo "quorum=$quorum tolerance=$tolerance"
|
||||
[ "$tolerance" -ge 1 ] \
|
||||
|| fail "$total etcd member(s) give quorum $quorum and tolerate $tolerance loss(es) — an HA control plane must survive losing a server"
|
||||
[ $(( total % 2 )) -eq 1 ] \
|
||||
|| fail "$total etcd members is EVEN: quorum is the same as for $(( total - 1 )) members, so the extra member adds a tie instead of tolerance"
|
||||
|
||||
# --- the count R-PLAT-01 itself fixes, per environment --------------------
|
||||
# The environment is the cluster's OWN CLUSTER_ENV (cluster-vars), never a
|
||||
# flag: 'k3s across 3 nodes ... in prod; dev/test MAY run fewer nodes but MUST
|
||||
# exercise the same multi-node topology'.
|
||||
case "${CLUSTER_ENV:-}" in
|
||||
prod)
|
||||
[ "$total" = "3" ] \
|
||||
|| fail "this cluster declares CLUSTER_ENV=prod and has $total nodes — R-PLAT-01 requires 'k3s across 3 nodes ... in prod' (docs/02: '3 physical nodes')" ;;
|
||||
"")
|
||||
fail "flux-system/cluster-vars has no CLUSTER_ENV — the cluster does not declare which environment it is, so R-PLAT-01's prod clause has no referent (and no literal may be substituted for it)" ;;
|
||||
*)
|
||||
[ "$total" -ge 2 ] \
|
||||
|| fail "this cluster declares CLUSTER_ENV=${CLUSTER_ENV} and has $total node(s) — dev/test may run fewer nodes than prod but MUST 'exercise the same multi-node topology'" ;;
|
||||
esac
|
||||
echo "HA_TOPOLOGY_OK"
|
||||
check:
|
||||
(contains($stdout, 'HA_TOPOLOGY_OK')): true
|
||||
# -----------------------------------------------------------------------------------
|
||||
- name: then-a-control-plane-only-vip-is-floated-on-every-server
|
||||
description: >-
|
||||
Read the running kube-vip DaemonSet: Ready on every control-plane node,
|
||||
control-plane-only, and floating an address that collides with no other allocator.
|
||||
try:
|
||||
- script:
|
||||
env:
|
||||
- name: GATEWAY_VIP
|
||||
value: ($gatewayVip)
|
||||
- name: LB_POOL_START
|
||||
value: ($lbPoolStart)
|
||||
- name: LB_POOL_STOP
|
||||
value: ($lbPoolStop)
|
||||
content: |
|
||||
set -eu
|
||||
fail() { echo "FAIL: $*" >&2; exit 1; }
|
||||
# dotted quad -> integer, for range comparison without ipcalc/python.
|
||||
n() { echo "$1" | awk -F. '{printf "%d", ($1*16777216)+($2*65536)+($3*256)+$4}'; }
|
||||
|
||||
# The DaemonSet is found by the label roles/kube_vip stamps on it, not by a
|
||||
# hardcoded name in a namespace this test guessed.
|
||||
ds="$(kubectl -n kube-system get ds -l app.kubernetes.io/name=kube-vip-ds -o name | head -1)"
|
||||
[ -n "$ds" ] \
|
||||
|| fail "no kube-vip DaemonSet is running in kube-system — the kube-apiserver has no floating endpoint, so losing the founder takes the API endpoint with it"
|
||||
|
||||
desired="$(kubectl -n kube-system get "$ds" -o jsonpath='{.status.desiredNumberScheduled}')"
|
||||
ready="$(kubectl -n kube-system get "$ds" -o jsonpath='{.status.numberReady}')"
|
||||
cp="$(kubectl get nodes -l node-role.kubernetes.io/control-plane -o name | grep -c . || true)"
|
||||
vip="$(kubectl -n kube-system get "$ds" \
|
||||
-o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="address")].value}')"
|
||||
cp_enable="$(kubectl -n kube-system get "$ds" \
|
||||
-o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="cp_enable")].value}')"
|
||||
svc_enable="$(kubectl -n kube-system get "$ds" \
|
||||
-o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="svc_enable")].value}')"
|
||||
echo "kube_vip=$ds ready=$ready/$desired control_plane_nodes=$cp vip=$vip cp_enable=$cp_enable svc_enable=$svc_enable"
|
||||
|
||||
[ -n "$vip" ] \
|
||||
|| fail "the kube-vip DaemonSet announces no address — nothing is floated"
|
||||
[ "$cp_enable" = "true" ] \
|
||||
|| fail "kube-vip runs with cp_enable=$cp_enable — control-plane mode is off, so it floats no API VIP"
|
||||
[ "$svc_enable" = "false" ] \
|
||||
|| fail "kube-vip runs with svc_enable=$svc_enable — it would also serve Service LoadBalancers and contend with Cilium LB-IPAM for ARP on the same addresses (ADR-0002 gives Service LB to Cilium)"
|
||||
[ "$desired" = "$cp" ] \
|
||||
|| fail "kube-vip is scheduled on $desired node(s) but there are $cp control-plane nodes — the VIP must be able to move to ANY server, or its failover set is smaller than the control plane"
|
||||
[ "$ready" = "$desired" ] \
|
||||
|| fail "kube-vip is Ready on $ready of $desired nodes — a VIP holder that is not running cannot take the address over"
|
||||
|
||||
# --- the VIP collides with nothing --------------------------------------
|
||||
node_ips="$(kubectl get nodes \
|
||||
-o jsonpath='{range .items[*]}{.status.addresses[?(@.type=="InternalIP")].address}{"\n"}{end}' \
|
||||
| grep -c "^${vip}$" || true)"
|
||||
[ "$node_ips" = "0" ] \
|
||||
|| fail "the floated VIP $vip is ALSO a node's own InternalIP — a VIP that is a node's address cannot float off that node"
|
||||
[ "$vip" != "${GATEWAY_VIP:-}" ] \
|
||||
|| fail "the control-plane VIP and the ingress Gateway VIP are the same address ($vip) — kube-vip and Cilium would both ARP for it"
|
||||
if [ -n "${LB_POOL_START:-}" ] && [ -n "${LB_POOL_STOP:-}" ]; then
|
||||
v="$(n "$vip")"; lo="$(n "$LB_POOL_START")"; hi="$(n "$LB_POOL_STOP")"
|
||||
[ "$lo" -le "$hi" ] \
|
||||
|| fail "the cluster's Cilium LB pool ${LB_POOL_START}-${LB_POOL_STOP} is inverted, so the VIP cannot be proven to be outside it"
|
||||
if [ "$v" -ge "$lo" ] && [ "$v" -le "$hi" ]; then
|
||||
fail "the control-plane VIP $vip lies INSIDE this cluster's Cilium LoadBalancer pool ${LB_POOL_START}-${LB_POOL_STOP} — LB-IPAM could hand the kube-API's VIP to a Service"
|
||||
fi
|
||||
else
|
||||
fail "flux-system/cluster-vars declares no CILIUM_LB_POOL_START/STOP — the separation between the control-plane VIP and the Service-LB allocator cannot be checked, and no literal range may be substituted for it (R-ENV-02)"
|
||||
fi
|
||||
echo "HA_VIP_CONTROL_PLANE_ONLY_OK"
|
||||
check:
|
||||
(contains($stdout, 'HA_VIP_CONTROL_PLANE_ONLY_OK')): true
|
||||
# -----------------------------------------------------------------------------------
|
||||
- name: then-the-apiserver-answers-through-the-vip-and-on-every-server
|
||||
description: >-
|
||||
From inside the cluster, probe /readyz on the floated VIP and on every server's own
|
||||
address, validating TLS against the cluster CA for the address dialled.
|
||||
try:
|
||||
- script:
|
||||
# The address list is DISCOVERED here (Nodes + the kube-vip DaemonSet), so the
|
||||
# Job cannot be a static fixture — writing the addresses down is the
|
||||
# per-environment literal R-ENV-02 forbids. Hence the imperative heredoc, the
|
||||
# same pattern tests/l3-fault-injection/dr-restore uses for run-time-shaped
|
||||
# fixtures. The Job lives in chainsaw's per-test namespace and is torn down with
|
||||
# it; nothing outside that namespace is created or changed.
|
||||
content: |
|
||||
set -eu
|
||||
ns="$NAMESPACE"
|
||||
fail() { echo "FAIL: $*" >&2; exit 1; }
|
||||
|
||||
ds="$(kubectl -n kube-system get ds -l app.kubernetes.io/name=kube-vip-ds -o name | head -1)"
|
||||
vip="$(kubectl -n kube-system get "$ds" \
|
||||
-o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="address")].value}')"
|
||||
servers="$(kubectl get nodes -l node-role.kubernetes.io/control-plane \
|
||||
-o jsonpath='{range .items[*]}{.status.addresses[?(@.type=="InternalIP")].address}{" "}{end}')"
|
||||
[ -n "$vip" ] || fail "no floated VIP to probe"
|
||||
[ -n "$servers" ] || fail "no control-plane node addresses to probe"
|
||||
echo "probing vip=$vip servers=$servers"
|
||||
|
||||
# The probe pod authenticates with its own ServiceAccount token and validates
|
||||
# TLS against the cluster CA, both from the projected SA volume. /readyz is
|
||||
# authorized for any authenticated identity via the built-in
|
||||
# system:public-info-viewer role — anonymous access is NOT relied on (verified:
|
||||
# an unauthenticated /readyz on this platform answers 401).
|
||||
kubectl -n "$ns" apply -f - <<EOF
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: ha-api-endpoint-probe
|
||||
spec:
|
||||
backoffLimit: 1
|
||||
activeDeadlineSeconds: 150
|
||||
template:
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: probe
|
||||
image: curlimages/curl:8.11.1
|
||||
env:
|
||||
- name: VIP
|
||||
value: "${vip}"
|
||||
- name: SERVERS
|
||||
value: "${servers}"
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- |
|
||||
set -u
|
||||
SA=/var/run/secrets/kubernetes.io/serviceaccount
|
||||
probe() { # \$1 = address to dial
|
||||
curl -sS --max-time 15 \\
|
||||
--cacert "\$SA/ca.crt" \\
|
||||
-H "Authorization: Bearer \$(cat \$SA/token)" \\
|
||||
"https://\$1:6443/readyz"
|
||||
}
|
||||
rc=0
|
||||
out="\$(probe "\$VIP" 2>&1)" || rc=1
|
||||
echo "vip \$VIP -> [\$out]"
|
||||
if [ "\$rc" != "0" ] || [ "\$out" != "ok" ]; then
|
||||
echo "FAIL: the kube-apiserver did not answer /readyz through the floated VIP \$VIP with a CA-valid cert for that address — the VIP does not front the API (or it is missing from the serving cert's SANs)"
|
||||
exit 1
|
||||
fi
|
||||
for s in \$SERVERS; do
|
||||
rc=0
|
||||
out="\$(probe "\$s" 2>&1)" || rc=1
|
||||
echo "server \$s -> [\$out]"
|
||||
if [ "\$rc" != "0" ] || [ "\$out" != "ok" ]; then
|
||||
echo "FAIL: server \$s does not answer /readyz with a CA-valid cert for its own address — it is not running a kube-apiserver, so the control plane is not HA across every node"
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "HA_API_ENDPOINT_OK"
|
||||
EOF
|
||||
|
||||
# Bounded wait; on failure surface the probe's own output before failing.
|
||||
if ! kubectl -n "$ns" wait --for=condition=complete --timeout=140s job/ha-api-endpoint-probe; then
|
||||
kubectl -n "$ns" logs job/ha-api-endpoint-probe --tail=40 || true
|
||||
fail "the in-cluster API-endpoint probe did not complete — see its log above"
|
||||
fi
|
||||
logs="$(kubectl -n "$ns" logs job/ha-api-endpoint-probe --tail=40)"
|
||||
echo "$logs"
|
||||
case "$logs" in
|
||||
*HA_API_ENDPOINT_OK*) echo "HA_API_ENDPOINT_REACHABLE_OK" ;;
|
||||
*) fail "the probe completed without reporting success" ;;
|
||||
esac
|
||||
check:
|
||||
(contains($stdout, 'HA_API_ENDPOINT_REACHABLE_OK')): true
|
||||
@@ -0,0 +1,212 @@
|
||||
# =============================================================================
|
||||
# L2 BEHAVIORAL — every OIDC client's STORED secret equals the one git declares
|
||||
# (R-IDP-02, R-IDP-04) [RUNNABLE]
|
||||
# Spec: docs/15 identity-and-access; ADR-0014.
|
||||
# Impl: platform/keycloak/client-secret-reconciler.yaml (the CronJob + its script).
|
||||
#
|
||||
# WHY THIS TEST EXISTS
|
||||
# --------------------
|
||||
# The KeycloakRealmImport resolves each confidential client's secret from a `(env:NAME)`
|
||||
# placeholder mapped to a key of the SOPS-encrypted keycloak-realm-clients Secret. When
|
||||
# the Keycloak Operator cannot resolve a placeholder it stores the LITERAL placeholder
|
||||
# text as the client secret AND THE IMPORT JOB STILL REPORTS SUCCESS; the CR then rests
|
||||
# at Done=True / Started=False and never re-runs. Measured 2026-09-06: prod had all 12
|
||||
# clients holding the literal placeholder — every OIDC login on the cluster failed with
|
||||
# "Invalid client or Invalid client credentials" — and dev had one (oauth2-proxy), whose
|
||||
# Secret key was added after dev's import had already completed.
|
||||
#
|
||||
# Every SIGNAL the cluster offered was green: the import Job succeeded, the CR read
|
||||
# Done=True, the Keycloak pods were healthy, and the existing keycloak-config-as-code
|
||||
# test (which asserts Done=True) PASSED throughout. The only observable was a human
|
||||
# failing to log in. This test is that observable, automated: it compares STORED against
|
||||
# DECLARED for every client and fails on any difference, so the class cannot sit
|
||||
# undetected for months again.
|
||||
#
|
||||
# GIVEN the `homelab` realm and the SOPS-encrypted keycloak-realm-clients Secret,
|
||||
# WHEN the reconciler's script is run in --audit mode (READ-ONLY: it cannot write, and
|
||||
# its endpoint allow-list makes users/groups/roles unreachable),
|
||||
# -> THEN every declared client's stored secret matches git, and none holds an
|
||||
# unresolved import placeholder. Emits KEYCLOAK_CLIENT_SECRETS_AUDIT_OK.
|
||||
#
|
||||
# WHY IT RUNS THE SHIPPED SCRIPT rather than re-implementing the comparison: a detector
|
||||
# that duplicates the healer's logic drifts from it. This audits with the same code, in
|
||||
# the same mode the L0 gate selftests, so a passing audit is evidence about the healer
|
||||
# too. It runs as a Job IN the keycloak namespace (not in the ephemeral chainsaw
|
||||
# namespace) because the desired state is the mounted keycloak-realm-clients Secret and
|
||||
# the admin credential is keycloak-admin-credentials — both namespace-local, which is
|
||||
# also why neither this Job nor the CronJob needs any Kubernetes API access.
|
||||
#
|
||||
# chainsaw v0.2.12: Job + status assert; the leftover-Job delete keeps a re-run from
|
||||
# colliding with an immutable Job left by a previous failure.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: keycloak-client-secrets-match-git
|
||||
labels:
|
||||
conformance.home.k3s/R-IDP-02: "true"
|
||||
conformance.home.k3s/R-IDP-04: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-IDP-02, R-IDP-04"
|
||||
conformance.home.k3s/layer: "L2"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Every OIDC client's stored secret equals the SOPS-declared one and none holds an
|
||||
unresolved import placeholder — the observable that was missing when all 12 prod
|
||||
clients were broken behind a green realm import.
|
||||
spec:
|
||||
timeouts:
|
||||
apply: 60s
|
||||
assert: 5m
|
||||
exec: 5m
|
||||
steps:
|
||||
- name: given-the-reconciler-is-reconciled-onto-the-cluster
|
||||
description: >-
|
||||
The audit Job runs the CronJob's own script from its ConfigMap; if Flux has not
|
||||
applied the reconciler, there is nothing to audit WITH and that is the failure.
|
||||
try:
|
||||
- assert:
|
||||
resource:
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: keycloak-client-secret-reconciler
|
||||
namespace: keycloak
|
||||
- assert:
|
||||
resource:
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: keycloak-client-secret-reconciler
|
||||
namespace: keycloak
|
||||
spec:
|
||||
concurrencyPolicy: Forbid
|
||||
- name: then-every-stored-client-secret-matches-git
|
||||
try:
|
||||
- script:
|
||||
content: |
|
||||
kubectl -n keycloak delete job keycloak-client-secret-audit \
|
||||
--ignore-not-found --wait=true
|
||||
- apply:
|
||||
resource:
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: keycloak-client-secret-audit
|
||||
namespace: keycloak
|
||||
labels:
|
||||
app.kubernetes.io/name: keycloak-client-secret-audit
|
||||
app.kubernetes.io/part-of: conformance
|
||||
spec:
|
||||
backoffLimit: 0
|
||||
activeDeadlineSeconds: 240
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: keycloak-client-secret-audit
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
automountServiceAccountToken: false
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 65532
|
||||
runAsGroup: 65532
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: audit
|
||||
image: python:3.13.15-alpine3.23
|
||||
command:
|
||||
- python3
|
||||
- /opt/reconciler/reconcile-client-secrets.py
|
||||
# READ-ONLY. --audit never issues a write on any path; the
|
||||
# script's --selftest proves that (L0 gate).
|
||||
- --audit
|
||||
env:
|
||||
- name: KEYCLOAK_BASE_URL
|
||||
value: http://keycloak-service.keycloak.svc.cluster.local:8080
|
||||
- name: KEYCLOAK_REALM
|
||||
value: homelab
|
||||
- name: PYTHONDONTWRITEBYTECODE
|
||||
value: "1"
|
||||
- name: PYTHONUNBUFFERED
|
||||
value: "1"
|
||||
- name: KEYCLOAK_ADMIN_USER
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: keycloak-admin-credentials
|
||||
key: admin-user
|
||||
- name: KEYCLOAK_ADMIN_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: keycloak-admin-credentials
|
||||
key: admin-password
|
||||
volumeMounts:
|
||||
- name: script
|
||||
mountPath: /opt/reconciler
|
||||
readOnly: true
|
||||
- name: client-secrets
|
||||
mountPath: /etc/keycloak-realm-clients
|
||||
readOnly: true
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 48Mi
|
||||
limits:
|
||||
memory: 128Mi
|
||||
volumes:
|
||||
- name: script
|
||||
configMap:
|
||||
name: keycloak-client-secret-reconciler
|
||||
defaultMode: 0444
|
||||
- name: client-secrets
|
||||
secret:
|
||||
secretName: keycloak-realm-clients
|
||||
defaultMode: 0400
|
||||
- assert:
|
||||
resource:
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: keycloak-client-secret-audit
|
||||
namespace: keycloak
|
||||
status:
|
||||
succeeded: 1
|
||||
# The Job's exit status already encodes the verdict, but the sentinel is what the
|
||||
# run LEDGER records (conformance/runs/*/run.log): a JUnit alone cannot tell a real
|
||||
# audit from a Job that succeeded without auditing anything.
|
||||
- script:
|
||||
content: |
|
||||
set -u
|
||||
out="$(kubectl -n keycloak logs job/keycloak-client-secret-audit \
|
||||
--tail=-1 2>&1 || true)"
|
||||
echo "$out"
|
||||
echo "$out" | grep -q 'KEYCLOAK_CLIENT_SECRETS_AUDIT_OK' \
|
||||
|| { echo "FAIL: the audit did not report every client secret matching git"; exit 1; }
|
||||
echo "$out" | grep -q 'UNRESOLVED IMPORT PLACEHOLDER' \
|
||||
&& { echo "FAIL: a client still holds the literal import placeholder"; exit 1; }
|
||||
echo KEYCLOAK_CLIENT_SECRET_DRIFT_NONE_OK
|
||||
check:
|
||||
(contains($stdout, 'KEYCLOAK_CLIENT_SECRET_DRIFT_NONE_OK')): true
|
||||
catch:
|
||||
- describe:
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
namespace: keycloak
|
||||
name: keycloak-client-secret-audit
|
||||
- podLogs:
|
||||
namespace: keycloak
|
||||
selector: app.kubernetes.io/name=keycloak-client-secret-audit
|
||||
tail: 60
|
||||
finally:
|
||||
- script:
|
||||
content: |
|
||||
kubectl -n keycloak delete job keycloak-client-secret-audit \
|
||||
--ignore-not-found --wait=false
|
||||
@@ -11,6 +11,18 @@
|
||||
#
|
||||
# Goes GREEN once Keycloak is up + the realm client-secret hand-off (keycloak-realm-clients) is filled
|
||||
# so the placeholder resolves and the import completes. chainsaw v0.2.12: script + block contains check.
|
||||
#
|
||||
# NOT PROVEN HERE — READ THIS BEFORE TRUSTING A GREEN RUN (added 2026-09-06).
|
||||
# Done=True means the import JOB EXITED 0. It does NOT mean the placeholders resolved: when the
|
||||
# operator cannot resolve one it stores the LITERAL `(env:...)` text as the client secret and the
|
||||
# import still reports success, then rests at Done=True / Started=False and never re-runs. THIS
|
||||
# TEST PASSED THROUGHOUT the 2026-09-06 outage in which all 12 prod clients held the literal
|
||||
# placeholder and every OIDC login on the cluster failed. Done=True is a necessary, badly
|
||||
# insufficient condition.
|
||||
# The sufficient check is tests/l2-behavioral/keycloak-client-secrets-match-git, which compares
|
||||
# each client's STORED secret against the SOPS-declared one; the static half is
|
||||
# tests/l0-static/keycloak-client-secret-reconciler. Do not "simplify" by folding those into this
|
||||
# test: the whole lesson is that the import's own success signal is not evidence.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
|
||||
@@ -3,10 +3,15 @@
|
||||
# Spec: docs/15 identity-and-access; ADR-0014. Impl: Grafana as a Keycloak OIDC client
|
||||
# (platform/monitoring/kube-prometheus-stack.yaml auth.generic_oauth) — the self-contained SSO proof.
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC (R-ENV-02): the expected IdP host is built from the ${BASE_DOMAIN} of the
|
||||
# CLUSTER UNDER TEST, read from flux-system/cluster-vars (the ConfigMap Flux substitutes from) by
|
||||
# a test-level chainsaw binding — so this asserts each cluster redirects to ITS OWN Keycloak, which
|
||||
# is exactly what would catch a cross-environment OIDC issuer left pointing at the other cluster.
|
||||
#
|
||||
# GIVEN Grafana is deployed and configured as an OIDC client of the Keycloak `homelab` realm,
|
||||
# WHEN an unauthenticated browser starts the OIDC login (GET /login/generic_oauth)
|
||||
# -> THEN Grafana redirects (302) to the CENTRAL IdP's authorize endpoint on
|
||||
# keycloak.home.keithjackson.dev/realms/homelab — proving SSO is driven by the one IdP
|
||||
# keycloak.${BASE_DOMAIN}/realms/homelab — proving SSO is driven by the one IdP
|
||||
# (R-IDP-01: central Identity Provider as the single user directory).
|
||||
#
|
||||
# Goes GREEN once the grafana-oidc client-secret hand-off is filled and Keycloak + Grafana are up.
|
||||
@@ -30,6 +35,11 @@ spec:
|
||||
apply: 60s
|
||||
assert: 3m
|
||||
exec: 3m
|
||||
# -- R-ENV-02: the IdP hostname suffix is READ FROM THE CLUSTER UNDER TEST
|
||||
# (flux-system/cluster-vars — the ConfigMap Flux substitutes ${BASE_DOMAIN} from).
|
||||
bindings:
|
||||
- name: baseDomain
|
||||
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
|
||||
steps:
|
||||
- name: given-grafana-is-an-oidc-client
|
||||
try:
|
||||
@@ -59,6 +69,9 @@ spec:
|
||||
containers:
|
||||
- name: probe
|
||||
image: busybox:1.36.1
|
||||
env:
|
||||
- name: BASE_DOMAIN
|
||||
value: ($baseDomain)
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
@@ -68,13 +81,14 @@ spec:
|
||||
# busybox wget has no --max-redirect; -S dumps every response header
|
||||
# (incl. the initial 302 Location to the IdP) before following, so the
|
||||
# grep below still matches the Keycloak authorize URL. (Grafana verified
|
||||
# live: 302 -> keycloak.home.keithjackson.dev/realms/homelab/.../auth.)
|
||||
# live: 302 -> keycloak.${BASE_DOMAIN}/realms/homelab/.../auth.)
|
||||
idp="keycloak.${BASE_DOMAIN}/realms/homelab"
|
||||
out="$(wget -S -O /dev/null "$svc/login/generic_oauth" 2>&1 || true)"
|
||||
echo "$out"
|
||||
if echo "$out" | grep -qi 'keycloak.home.keithjackson.dev/realms/homelab'; then
|
||||
echo GRAFANA_OIDC_REDIRECT_OK
|
||||
if echo "$out" | grep -qiF "$idp"; then
|
||||
echo "GRAFANA_OIDC_REDIRECT_OK -> ${idp}"
|
||||
else
|
||||
echo "FAIL: Grafana did not redirect to the Keycloak homelab realm"; exit 1
|
||||
echo "FAIL: Grafana did not redirect to ${idp} (this cluster's Keycloak homelab realm)"; exit 1
|
||||
fi
|
||||
- assert:
|
||||
resource:
|
||||
|
||||
@@ -3,8 +3,13 @@
|
||||
# Catalog: docs/14 onboarding (auth wiring) + docs/15 identity-and-access.
|
||||
# Impl: platform/oauth2-proxy (forward-auth against the Keycloak `homelab` realm), ADR-0014.
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC (R-ENV-02): this test names NO hostname. It talks to oauth2-proxy over its
|
||||
# in-cluster Service DNS and asserts HTTP behaviour, so the same authored test is correct on dev and
|
||||
# on prod. The issuer host below is written as ${BASE_DOMAIN} because that is what the platform
|
||||
# substitutes; it is prose, not an input — nothing here needs parameterizing.
|
||||
#
|
||||
# GIVEN oauth2-proxy is deployed and has validated the Keycloak OIDC issuer (it only becomes Available
|
||||
# after successfully discovering https://keycloak.home.keithjackson.dev/realms/homelab — this is
|
||||
# after successfully discovering https://keycloak.${BASE_DOMAIN}/realms/homelab — this is
|
||||
# the proxy->Keycloak flow working),
|
||||
# WHEN an UNAUTHENTICATED request hits the forward-auth decision endpoint (/oauth2/auth)
|
||||
# -> THEN it is challenged: HTTP 401 (no valid Keycloak session => access denied), and a normal
|
||||
|
||||
@@ -3,8 +3,13 @@
|
||||
# Catalog: docs/14 "Trusted TLS (R-TLS-01/02)"
|
||||
# Spec: docs/09 (Gateway API + DNS-01 wildcard), ADR-0006 (LE issuers)
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC (R-ENV-02): the hostnames are built from the ${BASE_DOMAIN} of the CLUSTER
|
||||
# UNDER TEST, read from flux-system/cluster-vars (the ConfigMap Flux substitutes from) by a
|
||||
# test-level chainsaw binding. The same authored test therefore verifies dev's wildcard on dev and
|
||||
# prod's on prod, with nothing to pass at run time.
|
||||
#
|
||||
# GIVEN the shared Gateway terminating TLS with the cert-manager wildcard cert,
|
||||
# WHEN an in-cluster probe dials it by hostname (SNI *.home.keithjackson.dev) and
|
||||
# WHEN an in-cluster probe dials it by hostname (SNI *.${BASE_DOMAIN}) and
|
||||
# verifies the chain against the SYSTEM trust store,
|
||||
# THEN the served cert is non-self-signed, unexpired, covers the wildcard SAN, and
|
||||
# chains to a PUBLICLY-TRUSTED CA root (R-TLS-01). The wildcard SAN also covers
|
||||
@@ -36,6 +41,11 @@ metadata:
|
||||
that chains to a publicly-trusted CA root (LE production); the wildcard SAN
|
||||
covers internal-only hostnames too.
|
||||
spec:
|
||||
# -- R-ENV-02: the hostname suffix is READ FROM THE CLUSTER UNDER TEST (flux-system/cluster-vars
|
||||
# — the ConfigMap Flux substitutes ${BASE_DOMAIN} from), never authored per environment.
|
||||
bindings:
|
||||
- name: baseDomain
|
||||
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
|
||||
steps:
|
||||
- name: then-gateway-cert-chains-to-trusted-root
|
||||
try:
|
||||
@@ -54,13 +64,16 @@ spec:
|
||||
containers:
|
||||
- name: probe
|
||||
image: curlimages/curl:8.11.1
|
||||
env:
|
||||
- name: BASE_DOMAIN
|
||||
value: ($baseDomain)
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- |
|
||||
set -eu
|
||||
# Probe TWO wildcard hostnames — one "public", one "internal-only". Both
|
||||
# match *.home.keithjackson.dev, so a clean verify against the SYSTEM CA
|
||||
# match *.${BASE_DOMAIN}, so a clean verify against the SYSTEM CA
|
||||
# bundle for BOTH proves R-TLS-01 (chains to a publicly-trusted root) AND
|
||||
# R-TLS-03 (the same trusted wildcard covers internal-only names). Names
|
||||
# resolve via in-cluster split-horizon DNS to the Gateway VIP. curl (no
|
||||
@@ -68,7 +81,7 @@ spec:
|
||||
# the gateway's 404 once TLS+hostname verify; it exits 60 if the chain or
|
||||
# hostname fails. TLS terminates at the gateway before HTTP routing, so no
|
||||
# backend is needed.
|
||||
for host in conformance.home.keithjackson.dev internal-only.home.keithjackson.dev; do
|
||||
for host in "conformance.${BASE_DOMAIN}" "internal-only.${BASE_DOMAIN}"; do
|
||||
curl -sS --retry 5 --retry-connrefused --retry-delay 3 --max-time 20 \
|
||||
-o /dev/null "https://${host}/"
|
||||
echo "TLS_TRUSTED ${host}"
|
||||
@@ -80,3 +93,20 @@ spec:
|
||||
kind: Job
|
||||
metadata: { name: tls-probe }
|
||||
status: { succeeded: 1 }
|
||||
# -----------------------------------------------------------------------------------
|
||||
# PROOF-OF-PASS SENTINEL (R-SEC-03). Every step above is an assert/Job-status gate, so
|
||||
# chainsaw already fails the test before reaching this step if any of them fails — but a
|
||||
# green Chainsaw JUnit carries NO stdout, and the L2/L3 tests in this suite can self-skip
|
||||
# GREEN, so lib/record_run.py refuses to treat a bare `passed` as evidence and records
|
||||
# `unknown-no-sentinel-declared` instead. Without a declared token this test could pass
|
||||
# forever and NEVER become a recorded proof of R-TLS-01/03 — the TLS half of docs/07's audit proof list
|
||||
# — which is precisely what docs/07 asks for ("These proofs are part of the platform
|
||||
# acceptance suite, not one-time checks"). This step exists only to emit that token, and it
|
||||
# is reached only when everything above has passed. Asserted by
|
||||
# tests/l0-static/auditability-proofs-producible.
|
||||
- name: then-emit-the-proof-of-pass-sentinel
|
||||
try:
|
||||
- script:
|
||||
content: echo TLS_TRUSTED_PROVEN_OK
|
||||
check:
|
||||
(contains($stdout, 'TLS_TRUSTED_PROVEN_OK')): true
|
||||
Reference in new issue
Block a user