jakman85andClaude Opus 5
6a8953e288
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
392 lines
24 KiB
YAML
392 lines
24 KiB
YAML
# =============================================================================
|
|
# L2 BEHAVIORAL — certificate lifecycle is automatic (R-TLS-02)
|
|
# [RUNNABLE / NOT YET EXECUTED — see STATUS]
|
|
# Catalog: docs/14 "Trusted TLS (R-TLS-01/02)"
|
|
# Spec: docs/00 R-TLS-02 — "Certificates MUST be automatically **issued, renewed,
|
|
# and revoked** without manual steps."
|
|
# docs/08 §"Renewal & revocation (R-TLS-02)" — "cert-manager renews well
|
|
# before expiry automatically; no human action." / "Revocation: removing/
|
|
# rotating a certificate is a declarative change; cert-manager handles
|
|
# issuance of replacements."
|
|
# docs/08 acceptance gate — "Certificates renew automatically ahead of expiry
|
|
# with no manual step (**observed across a renewal**)."
|
|
# Impl: platform/cert-manager/**, platform/gateway/wildcard-certificate.yaml.
|
|
# Sibling: tests/l0-static/tls-lifecycle-automated proves the "without manual steps"
|
|
# clause statically (in the merge gate). This test observes the three VERBS.
|
|
#
|
|
# ENVIRONMENT-AGNOSTIC (R-ENV-02): NOTHING about either environment is written down. The
|
|
# serving certificate is DISCOVERED — find the Gateway whose listener terminates TLS, follow
|
|
# its certificateRefs Secret, and read that Secret's `cert-manager.io/certificate-name`
|
|
# annotation to reach the Certificate. No hostname, no namespace list, no issuer name, no
|
|
# address: dev proves dev's cert and prod proves prod's from the same authored assertions.
|
|
#
|
|
# GIVEN cert-manager reconciled by Flux and a Gateway terminating TLS,
|
|
# WHEN the serving Certificate's provenance and renewal schedule are read, and a NEW
|
|
# Certificate is then requested, left to renew itself, and removed,
|
|
# THEN 1. the serving credential was produced by the CONTROLLER, not a human: its
|
|
# Secret carries cert-manager's certificate-name annotation, its
|
|
# CertificateRequest was submitted by cert-manager's own ServiceAccount and
|
|
# was APPROVED automatically, and its issuer is a Ready ACME ClusterIssuer;
|
|
# 2. an unattended RENEWAL IS SCHEDULED: status.renewalTime is in the future and
|
|
# strictly before status.notAfter, so no human is in the renewal path;
|
|
# 3. ISSUED — a newly declared Certificate becomes Ready and its Secret appears
|
|
# with certificate material, with no step between declaring and having it;
|
|
# 4. RENEWED — that certificate, given a renewal window that opens one minute
|
|
# after issuance, REISSUES ITSELF at least twice (status.revision >= 3) and
|
|
# the serving bytes actually change, with no human action. This is docs/08's
|
|
# "observed across a renewal", fast-forwarded;
|
|
# 5. REMOVED/REVOKED — deleting the Certificate withdraws its serving material.
|
|
#
|
|
# WHY A HERMETIC SELF-SIGNED ISSUER FOR STEPS 3-5, AND WHAT THAT COSTS
|
|
# -------------------------------------------------------------------
|
|
# Steps 3-5 need a certificate lifecycle that completes in minutes and repeats on every
|
|
# run. Driving that through the real ACME issuer would fire a public Let's Encrypt order
|
|
# (and a Cloudflare _acme-challenge write) per conformance run, burning issuance rate
|
|
# limits for evidence that is not about ACME: cert-manager's renewal scheduler and
|
|
# ownership logic are issuer-independent. So steps 3-5 use a `selfSigned` Issuer created
|
|
# in this test's OWN namespace, and the ACME/DNS-01 path is proven elsewhere — step 1-2
|
|
# read the REAL wildcard certificate the Gateway serves, and l2-behavioral/trusted-tls
|
|
# proves a publicly-trusted LE certificate is actually served (R-TLS-01/03).
|
|
# STATED PLAINLY: steps 3-5 do NOT prove the ACME order path renews; they prove the
|
|
# CONTROLLER's issue/renew/withdraw automation runs unattended in this cluster, while
|
|
# steps 1-2 prove the real serving certificate is on exactly that automated path.
|
|
#
|
|
# STATUS: RUNNABLE, NOT YET EXECUTED. Steps 1-2 are read-only. Steps 3-5 CREATE an
|
|
# Issuer/Certificate (and DELETE the Certificate they created) inside chainsaw's
|
|
# per-test namespace — the same class of mutation every other L2 test here performs,
|
|
# self-cleaning via namespace teardown, touching no platform object. WS-TEST authored
|
|
# them against live dev/prod read-only inspection under an explicit instruction not to
|
|
# mutate either cluster, so THE MUTATING STEPS HAVE NOT BEEN RUN. Nothing about the
|
|
# suite gates on them: ci/conformance.sh only runs L2/L3 when CLUSTER_AVAILABLE=true.
|
|
#
|
|
# EXPECTED FAILURE, DELIBERATELY NOT SOFTENED (R-TEST-06)
|
|
# ------------------------------------------------------
|
|
# Step 5 WILL FAIL as the platform is authored today. cert-manager only garbage-collects
|
|
# a Certificate's Secret when the controller runs with `enableCertificateOwnerRef: true`;
|
|
# platform/cert-manager/helmrelease.yaml does not set it (chart default false), and the
|
|
# live wildcard Secret on both clusters has NO ownerReferences (verified read-only,
|
|
# 2026-09-05). So a removed Certificate leaves its credential in place, and "revoked
|
|
# without manual steps" does not hold. That is a finding about the platform, not a
|
|
# defect in this test: the assertion stays as the requirement reads and is escalated to
|
|
# the PM. (ACME revocation proper is not implemented by cert-manager at all.)
|
|
#
|
|
# KNOWN, DELIBERATE GAPS (this test claims none of these):
|
|
# * Not proven: that the CA would refuse a revoked certificate — there is no
|
|
# revocation request to make, so nothing observable exists to assert.
|
|
# * Not proven: renewal of the REAL wildcard cert (its window opens 30 days before a
|
|
# 90-day expiry; a conformance run cannot wait, and forcing it would rotate the
|
|
# platform's serving credential). Step 2 asserts the schedule; step 4 asserts the
|
|
# mechanism that will act on it.
|
|
# * Not proven: certificate expiry ALERTING — docs/08 delegates that to R-OBS-04
|
|
# (l3-fault-injection/alerting).
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: cert-lifecycle-automated
|
|
labels:
|
|
conformance.home.k3s/R-TLS-02: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-TLS-02"
|
|
conformance.home.k3s/layer: "L2"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
Given cert-manager and a TLS-terminating Gateway, when the serving certificate's
|
|
provenance and renewal schedule are read and a new certificate is requested, left
|
|
to renew, and removed, then issuance, renewal and withdrawal all happen with no
|
|
human step — the serving cert was controller-issued and auto-approved, its
|
|
renewal is scheduled ahead of expiry, and a declared certificate issues and
|
|
reissues itself unattended.
|
|
spec:
|
|
timeouts:
|
|
apply: 60s
|
|
assert: 3m
|
|
exec: 10m
|
|
steps:
|
|
# ---------------------------------------------------------------- READ-ONLY HALF
|
|
- name: given-cert-manager-is-reconciled-and-healthy
|
|
description: >-
|
|
The controller that issues and renews must be running, and must have been
|
|
installed by Flux rather than by hand — otherwise "automatic" has no engine.
|
|
try:
|
|
- script:
|
|
content: |
|
|
set -u
|
|
# Discover cert-manager's Deployments by chart label, in whatever namespace
|
|
# the release lives (nothing about the environment is assumed).
|
|
sel='app.kubernetes.io/instance=cert-manager'
|
|
rows="$(kubectl get deploy -A -l "$sel" \
|
|
-o custom-columns='NS:.metadata.namespace,N:.metadata.name,RDY:.status.readyReplicas' \
|
|
--no-headers 2>/dev/null || true)"
|
|
echo "$rows"
|
|
[ -n "$rows" ] || { echo "FAIL: no cert-manager Deployment found ($sel)"; exit 1; }
|
|
n=0
|
|
echo "$rows" | while read -r ns name rdy; do
|
|
[ "${rdy:-0}" -ge 1 ] || { echo "FAIL: $ns/$name has no ready replica"; exit 1; }
|
|
done || exit 1
|
|
n="$(echo "$rows" | wc -l)"
|
|
# controller + webhook + cainjector: without the webhook a Certificate is
|
|
# rejected, without cainjector the webhook has no CA — all three or nothing.
|
|
[ "$n" -ge 3 ] || { echo "FAIL: expected >=3 cert-manager Deployments, saw $n"; exit 1; }
|
|
# GitOps-managed, not hand-installed: Flux labels what it applies.
|
|
ns="$(echo "$rows" | head -1 | awk '{print $1}')"
|
|
lbl="$(kubectl get deploy -n "$ns" -l "$sel" \
|
|
-o jsonpath='{.items[0].metadata.labels.helm\.toolkit\.fluxcd\.io/name}' 2>/dev/null || true)"
|
|
[ -n "$lbl" ] || { echo "FAIL: cert-manager is not owned by a Flux HelmRelease"; exit 1; }
|
|
echo "CERT_MANAGER_HEALTHY_OK ($n deployments, flux HelmRelease $lbl)"
|
|
check:
|
|
(contains($stdout, 'CERT_MANAGER_HEALTHY_OK')): true
|
|
|
|
- name: then-the-serving-certificate-was-issued-by-the-controller
|
|
description: >-
|
|
Follow the Gateway's terminating listener to the Secret it serves, and prove
|
|
that Secret's provenance is cert-manager — annotation, CertificateRequest
|
|
requester, automatic approval, and a Ready ACME issuer.
|
|
try:
|
|
- script:
|
|
content: |
|
|
set -u
|
|
# --- discover the serving Secret from the Gateway itself -------------------
|
|
gw="$(kubectl get gateway -A \
|
|
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}' \
|
|
2>/dev/null | head -1)"
|
|
[ -n "$gw" ] || { echo "FAIL: no Gateway found — nothing terminates TLS"; exit 1; }
|
|
gns="${gw% *}"; gname="${gw#* }"
|
|
sec="$(kubectl -n "$gns" get gateway "$gname" \
|
|
-o jsonpath='{.spec.listeners[?(@.tls.mode=="Terminate")].tls.certificateRefs[0].name}' \
|
|
2>/dev/null | awk '{print $1}')"
|
|
[ -n "$sec" ] || { echo "FAIL: Gateway $gns/$gname terminates no TLS listener"; exit 1; }
|
|
echo "gateway=$gns/$gname serving-secret=$sec"
|
|
|
|
# --- provenance (1): only the controller writes this annotation ------------
|
|
cert="$(kubectl -n "$gns" get secret "$sec" \
|
|
-o jsonpath='{.metadata.annotations.cert-manager\.io/certificate-name}' 2>/dev/null || true)"
|
|
[ -n "$cert" ] || { echo "FAIL: Secret $gns/$sec carries no cert-manager.io/certificate-name annotation — it was not produced by cert-manager"; exit 1; }
|
|
crt="$(kubectl -n "$gns" get secret "$sec" -o jsonpath='{.data.tls\.crt}' 2>/dev/null || true)"
|
|
[ -n "$crt" ] || { echo "FAIL: Secret $gns/$sec holds no tls.crt"; exit 1; }
|
|
|
|
# --- provenance (2): the Certificate is Ready and has a revision -----------
|
|
ready="$(kubectl -n "$gns" get certificate "$cert" \
|
|
-o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)"
|
|
rev="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.status.revision}' 2>/dev/null || true)"
|
|
echo "certificate=$gns/$cert ready=$ready revision=$rev"
|
|
[ "$ready" = "True" ] || { echo "FAIL: Certificate $cert is not Ready"; exit 1; }
|
|
[ -n "$rev" ] && [ "$rev" -ge 1 ] || { echo "FAIL: Certificate $cert has no issued revision"; exit 1; }
|
|
|
|
# --- provenance (3): the request was made BY THE CONTROLLER and auto-approved
|
|
# spec.username is the identity that created the CertificateRequest. A human
|
|
# operator would appear as a user/OIDC identity, not a ServiceAccount.
|
|
cr="$(kubectl -n "$gns" get certificaterequest \
|
|
-o jsonpath="{range .items[?(@.metadata.annotations.cert-manager\.io/certificate-name=='${cert}')]}{.metadata.name}{' '}{.spec.username}{'\n'}{end}" \
|
|
2>/dev/null | tail -1)"
|
|
[ -n "$cr" ] || { echo "FAIL: no CertificateRequest for $cert — cannot show who requested it"; exit 1; }
|
|
crname="${cr% *}"; requester="${cr#* }"
|
|
echo "certificaterequest=$crname requester=$requester"
|
|
case "$requester" in
|
|
system:serviceaccount:*) : ;;
|
|
*) echo "FAIL: CertificateRequest $crname was submitted by '$requester', not a controller ServiceAccount"; exit 1 ;;
|
|
esac
|
|
appr="$(kubectl -n "$gns" get certificaterequest "$crname" \
|
|
-o jsonpath='{.status.conditions[?(@.type=="Approved")].status}' 2>/dev/null || true)"
|
|
[ "$appr" = "True" ] || { echo "FAIL: CertificateRequest $crname is not Approved=True (approval would be a manual step)"; exit 1; }
|
|
|
|
# --- provenance (4): an automated ACME issuer, and it is Ready -------------
|
|
ikind="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.spec.issuerRef.kind}')"
|
|
iname="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.spec.issuerRef.name}')"
|
|
echo "issuer=$ikind/$iname"
|
|
if [ "$ikind" = "ClusterIssuer" ]; then
|
|
iready="$(kubectl get clusterissuer "$iname" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)"
|
|
iacme="$(kubectl get clusterissuer "$iname" -o jsonpath='{.spec.acme.server}' 2>/dev/null || true)"
|
|
else
|
|
iready="$(kubectl -n "$gns" get issuer "$iname" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)"
|
|
iacme="$(kubectl -n "$gns" get issuer "$iname" -o jsonpath='{.spec.acme.server}' 2>/dev/null || true)"
|
|
fi
|
|
[ "$iready" = "True" ] || { echo "FAIL: issuer $iname is not Ready — renewal would need a human to fix it"; exit 1; }
|
|
[ -n "$iacme" ] || { echo "FAIL: issuer $iname is not an ACME issuer"; exit 1; }
|
|
echo "SERVING_CERT_CONTROLLER_ISSUED_OK"
|
|
check:
|
|
(contains($stdout, 'SERVING_CERT_CONTROLLER_ISSUED_OK')): true
|
|
|
|
- name: then-an-unattended-renewal-is-already-scheduled
|
|
description: >-
|
|
cert-manager writes status.renewalTime only after computing when it will renew.
|
|
Asserting it is in the future and before expiry proves the renewal is scheduled
|
|
and that nothing is waiting on a human (docs/08 "renews well before expiry").
|
|
try:
|
|
- script:
|
|
content: |
|
|
set -u
|
|
gw="$(kubectl get gateway -A \
|
|
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}' | head -1)"
|
|
gns="${gw% *}"; gname="${gw#* }"
|
|
sec="$(kubectl -n "$gns" get gateway "$gname" \
|
|
-o jsonpath='{.spec.listeners[?(@.tls.mode=="Terminate")].tls.certificateRefs[0].name}' | awk '{print $1}')"
|
|
cert="$(kubectl -n "$gns" get secret "$sec" \
|
|
-o jsonpath='{.metadata.annotations.cert-manager\.io/certificate-name}')"
|
|
renew="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.status.renewalTime}')"
|
|
after="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.status.notAfter}')"
|
|
[ -n "$renew" ] || { echo "FAIL: Certificate $cert has no status.renewalTime — no renewal is scheduled"; exit 1; }
|
|
[ -n "$after" ] || { echo "FAIL: Certificate $cert has no status.notAfter"; exit 1; }
|
|
now_s="$(date -u +%s)"
|
|
r_s="$(date -u -d "$renew" +%s)"
|
|
a_s="$(date -u -d "$after" +%s)"
|
|
echo "now=$(date -u -d "@$now_s" +%FT%TZ) renewalTime=$renew notAfter=$after"
|
|
echo "renewal is in $(( (r_s - now_s) / 86400 ))d; expiry in $(( (a_s - now_s) / 86400 ))d"
|
|
[ "$r_s" -gt "$now_s" ] || { echo "FAIL: renewalTime is in the PAST — the scheduled renewal did not happen"; exit 1; }
|
|
[ "$r_s" -lt "$a_s" ] || { echo "FAIL: renewalTime is not before notAfter — renewal would come too late"; exit 1; }
|
|
echo "RENEWAL_SCHEDULED_OK"
|
|
check:
|
|
(contains($stdout, 'RENEWAL_SCHEDULED_OK')): true
|
|
|
|
# ------------------------------------------------- MUTATING HALF (hermetic namespace)
|
|
- name: when-a-certificate-is-declared-then-it-is-issued-with-no-human-step
|
|
description: >-
|
|
Declare an Issuer and a Certificate in this test's own namespace, with a renewal
|
|
window that opens one minute after issuance. Nothing else is done to them for the
|
|
rest of the test — every later transition is the controller acting on its own.
|
|
try:
|
|
- apply:
|
|
resource:
|
|
# selfSigned: no external CA, no rate limit, no DNS write. See the header for
|
|
# what this proves (controller automation) and what it does not (the ACME path).
|
|
apiVersion: cert-manager.io/v1
|
|
kind: Issuer
|
|
metadata:
|
|
name: conformance-lifecycle
|
|
spec:
|
|
selfSigned: {}
|
|
- apply:
|
|
resource:
|
|
apiVersion: cert-manager.io/v1
|
|
kind: Certificate
|
|
metadata:
|
|
name: conformance-lifecycle
|
|
spec:
|
|
secretName: conformance-lifecycle-tls
|
|
# 1h is cert-manager's minimum duration; renewing 59m before expiry puts the
|
|
# renewal instant ~1 minute after each issuance, so an unattended renewal is
|
|
# observable inside a test run instead of 30 days from now.
|
|
duration: 1h
|
|
renewBefore: 3540s
|
|
commonName: conformance-lifecycle.invalid
|
|
dnsNames:
|
|
- conformance-lifecycle.invalid
|
|
privateKey:
|
|
rotationPolicy: Always
|
|
issuerRef:
|
|
kind: Issuer
|
|
name: conformance-lifecycle
|
|
- assert:
|
|
resource:
|
|
apiVersion: cert-manager.io/v1
|
|
kind: Certificate
|
|
metadata:
|
|
name: conformance-lifecycle
|
|
status:
|
|
conditions:
|
|
- type: Ready
|
|
status: "True"
|
|
revision: 1
|
|
- script:
|
|
content: |
|
|
set -u
|
|
crt="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \
|
|
-o jsonpath='{.data.tls\.crt}' 2>/dev/null || true)"
|
|
key="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \
|
|
-o jsonpath='{.data.tls\.key}' 2>/dev/null || true)"
|
|
[ -n "$crt" ] && [ -n "$key" ] || { echo "FAIL: no certificate material was issued into conformance-lifecycle-tls"; exit 1; }
|
|
# Record the serving bytes so the renewal step can prove they CHANGED.
|
|
echo "$crt" | cksum > /tmp/conformance-cert-r1.cksum
|
|
echo "revision-1 cert fingerprint: $(cat /tmp/conformance-cert-r1.cksum)"
|
|
echo "CERT_ISSUED_OK"
|
|
check:
|
|
(contains($stdout, 'CERT_ISSUED_OK')): true
|
|
|
|
- name: then-it-renews-itself-before-expiry-with-no-human-step
|
|
description: >-
|
|
Nothing touches the Certificate here. Poll until the controller has reissued it
|
|
at least twice on its own and the serving bytes differ from revision 1 — docs/08's
|
|
"observed across a renewal", and every request auto-approved.
|
|
try:
|
|
- script:
|
|
content: |
|
|
set -u
|
|
want=3 # revision 1 = issuance; >=3 means TWO unattended renewals
|
|
for i in $(seq 1 30); do
|
|
rev="$(kubectl -n "$NAMESPACE" get certificate conformance-lifecycle \
|
|
-o jsonpath='{.status.revision}' 2>/dev/null || echo 0)"
|
|
echo "poll $i: revision=${rev:-0}"
|
|
[ "${rev:-0}" -ge "$want" ] && break
|
|
sleep 20
|
|
done
|
|
rev="$(kubectl -n "$NAMESPACE" get certificate conformance-lifecycle \
|
|
-o jsonpath='{.status.revision}' 2>/dev/null || echo 0)"
|
|
[ "${rev:-0}" -ge "$want" ] || { echo "FAIL: revision reached only ${rev:-0} — cert-manager did not renew unattended"; exit 1; }
|
|
|
|
# The serving material must actually have rotated, not just the revision count.
|
|
now="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \
|
|
-o jsonpath='{.data.tls\.crt}' | cksum)"
|
|
was="$(cat /tmp/conformance-cert-r1.cksum)"
|
|
echo "r1=$was now=$now"
|
|
[ "$now" != "$was" ] || { echo "FAIL: the Secret still holds the revision-1 certificate — nothing was actually reissued"; exit 1; }
|
|
|
|
# Every renewal request was approved by the controller, never by a person.
|
|
bad=0
|
|
for cr in $(kubectl -n "$NAMESPACE" get certificaterequest -o name 2>/dev/null); do
|
|
st="$(kubectl -n "$NAMESPACE" get "$cr" \
|
|
-o jsonpath='{.status.conditions[?(@.type=="Approved")].status}' 2>/dev/null || true)"
|
|
who="$(kubectl -n "$NAMESPACE" get "$cr" -o jsonpath='{.spec.username}' 2>/dev/null || true)"
|
|
echo " $cr approved=$st requester=$who"
|
|
[ "$st" = "True" ] || bad=1
|
|
case "$who" in system:serviceaccount:*) : ;; *) bad=1 ;; esac
|
|
done
|
|
[ "$bad" = 0 ] || { echo "FAIL: a CertificateRequest was not auto-approved, or was not submitted by a controller"; exit 1; }
|
|
echo "CERT_RENEWED_UNATTENDED_OK (revision=$rev)"
|
|
check:
|
|
(contains($stdout, 'CERT_RENEWED_UNATTENDED_OK')): true
|
|
|
|
- name: then-removing-the-certificate-withdraws-the-serving-material
|
|
description: >-
|
|
docs/08 reads revocation as "removing/rotating a certificate is a declarative
|
|
change". The declarative change is the deletion; withdrawal of the credential
|
|
must follow with no manual cleanup. EXPECTED TO FAIL TODAY — cert-manager needs
|
|
`enableCertificateOwnerRef: true` for the Secret to be garbage-collected, and the
|
|
tracked HelmRelease does not set it. Kept as the requirement reads (R-TEST-06).
|
|
try:
|
|
- delete:
|
|
ref:
|
|
apiVersion: cert-manager.io/v1
|
|
kind: Certificate
|
|
name: conformance-lifecycle
|
|
- script:
|
|
content: |
|
|
set -u
|
|
for i in $(seq 1 15); do
|
|
if ! kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls >/dev/null 2>&1; then
|
|
echo "CERT_WITHDRAWN_OK (serving material gone after $((i * 4))s)"
|
|
exit 0
|
|
fi
|
|
sleep 4
|
|
done
|
|
owners="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \
|
|
-o jsonpath='{.metadata.ownerReferences}' 2>/dev/null || true)"
|
|
echo "FAIL: the Certificate was deleted but Secret conformance-lifecycle-tls still"
|
|
echo " holds usable certificate material (ownerReferences=${owners:-none})."
|
|
echo " Removing a certificate therefore does NOT withdraw the credential, so"
|
|
echo " R-TLS-02's 'revoked ... without manual steps' does not hold. FIX:"
|
|
echo " set enableCertificateOwnerRef: true in platform/cert-manager values"
|
|
echo " (owner decision — WS-TEST does not weaken the assertion instead)."
|
|
exit 1
|
|
check:
|
|
(contains($stdout, 'CERT_WITHDRAWN_OK')): true
|
|
finally:
|
|
# Namespace teardown removes both objects anyway; explicit so a leftover Secret
|
|
# from the expected failure above is never left holding key material.
|
|
- script:
|
|
content: |
|
|
kubectl -n "$NAMESPACE" delete secret conformance-lifecycle-tls --ignore-not-found --wait=false || true
|
|
kubectl -n "$NAMESPACE" delete certificate conformance-lifecycle --ignore-not-found --wait=false || true
|
|
kubectl -n "$NAMESPACE" delete issuer conformance-lifecycle --ignore-not-found --wait=false || true
|
|
rm -f /tmp/conformance-cert-r1.cksum || true
|