feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
d1a6628a97
commit
6a8953e288
46 files changed
+7898
-71
No files matched your search
@@ -0,0 +1,391 @@
|
||||
# =============================================================================
|
||||
# L2 BEHAVIORAL — certificate lifecycle is automatic (R-TLS-02)
|
||||
# [RUNNABLE / NOT YET EXECUTED — see STATUS]
|
||||
# Catalog: docs/14 "Trusted TLS (R-TLS-01/02)"
|
||||
# Spec: docs/00 R-TLS-02 — "Certificates MUST be automatically **issued, renewed,
|
||||
# and revoked** without manual steps."
|
||||
# docs/08 §"Renewal & revocation (R-TLS-02)" — "cert-manager renews well
|
||||
# before expiry automatically; no human action." / "Revocation: removing/
|
||||
# rotating a certificate is a declarative change; cert-manager handles
|
||||
# issuance of replacements."
|
||||
# docs/08 acceptance gate — "Certificates renew automatically ahead of expiry
|
||||
# with no manual step (**observed across a renewal**)."
|
||||
# Impl: platform/cert-manager/**, platform/gateway/wildcard-certificate.yaml.
|
||||
# Sibling: tests/l0-static/tls-lifecycle-automated proves the "without manual steps"
|
||||
# clause statically (in the merge gate). This test observes the three VERBS.
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC (R-ENV-02): NOTHING about either environment is written down. The
|
||||
# serving certificate is DISCOVERED — find the Gateway whose listener terminates TLS, follow
|
||||
# its certificateRefs Secret, and read that Secret's `cert-manager.io/certificate-name`
|
||||
# annotation to reach the Certificate. No hostname, no namespace list, no issuer name, no
|
||||
# address: dev proves dev's cert and prod proves prod's from the same authored assertions.
|
||||
#
|
||||
# GIVEN cert-manager reconciled by Flux and a Gateway terminating TLS,
|
||||
# WHEN the serving Certificate's provenance and renewal schedule are read, and a NEW
|
||||
# Certificate is then requested, left to renew itself, and removed,
|
||||
# THEN 1. the serving credential was produced by the CONTROLLER, not a human: its
|
||||
# Secret carries cert-manager's certificate-name annotation, its
|
||||
# CertificateRequest was submitted by cert-manager's own ServiceAccount and
|
||||
# was APPROVED automatically, and its issuer is a Ready ACME ClusterIssuer;
|
||||
# 2. an unattended RENEWAL IS SCHEDULED: status.renewalTime is in the future and
|
||||
# strictly before status.notAfter, so no human is in the renewal path;
|
||||
# 3. ISSUED — a newly declared Certificate becomes Ready and its Secret appears
|
||||
# with certificate material, with no step between declaring and having it;
|
||||
# 4. RENEWED — that certificate, given a renewal window that opens one minute
|
||||
# after issuance, REISSUES ITSELF at least twice (status.revision >= 3) and
|
||||
# the serving bytes actually change, with no human action. This is docs/08's
|
||||
# "observed across a renewal", fast-forwarded;
|
||||
# 5. REMOVED/REVOKED — deleting the Certificate withdraws its serving material.
|
||||
#
|
||||
# WHY A HERMETIC SELF-SIGNED ISSUER FOR STEPS 3-5, AND WHAT THAT COSTS
|
||||
# -------------------------------------------------------------------
|
||||
# Steps 3-5 need a certificate lifecycle that completes in minutes and repeats on every
|
||||
# run. Driving that through the real ACME issuer would fire a public Let's Encrypt order
|
||||
# (and a Cloudflare _acme-challenge write) per conformance run, burning issuance rate
|
||||
# limits for evidence that is not about ACME: cert-manager's renewal scheduler and
|
||||
# ownership logic are issuer-independent. So steps 3-5 use a `selfSigned` Issuer created
|
||||
# in this test's OWN namespace, and the ACME/DNS-01 path is proven elsewhere — step 1-2
|
||||
# read the REAL wildcard certificate the Gateway serves, and l2-behavioral/trusted-tls
|
||||
# proves a publicly-trusted LE certificate is actually served (R-TLS-01/03).
|
||||
# STATED PLAINLY: steps 3-5 do NOT prove the ACME order path renews; they prove the
|
||||
# CONTROLLER's issue/renew/withdraw automation runs unattended in this cluster, while
|
||||
# steps 1-2 prove the real serving certificate is on exactly that automated path.
|
||||
#
|
||||
# STATUS: RUNNABLE, NOT YET EXECUTED. Steps 1-2 are read-only. Steps 3-5 CREATE an
|
||||
# Issuer/Certificate (and DELETE the Certificate they created) inside chainsaw's
|
||||
# per-test namespace — the same class of mutation every other L2 test here performs,
|
||||
# self-cleaning via namespace teardown, touching no platform object. WS-TEST authored
|
||||
# them against live dev/prod read-only inspection under an explicit instruction not to
|
||||
# mutate either cluster, so THE MUTATING STEPS HAVE NOT BEEN RUN. Nothing about the
|
||||
# suite gates on them: ci/conformance.sh only runs L2/L3 when CLUSTER_AVAILABLE=true.
|
||||
#
|
||||
# EXPECTED FAILURE, DELIBERATELY NOT SOFTENED (R-TEST-06)
|
||||
# ------------------------------------------------------
|
||||
# Step 5 WILL FAIL as the platform is authored today. cert-manager only garbage-collects
|
||||
# a Certificate's Secret when the controller runs with `enableCertificateOwnerRef: true`;
|
||||
# platform/cert-manager/helmrelease.yaml does not set it (chart default false), and the
|
||||
# live wildcard Secret on both clusters has NO ownerReferences (verified read-only,
|
||||
# 2026-09-05). So a removed Certificate leaves its credential in place, and "revoked
|
||||
# without manual steps" does not hold. That is a finding about the platform, not a
|
||||
# defect in this test: the assertion stays as the requirement reads and is escalated to
|
||||
# the PM. (ACME revocation proper is not implemented by cert-manager at all.)
|
||||
#
|
||||
# KNOWN, DELIBERATE GAPS (this test claims none of these):
|
||||
# * Not proven: that the CA would refuse a revoked certificate — there is no
|
||||
# revocation request to make, so nothing observable exists to assert.
|
||||
# * Not proven: renewal of the REAL wildcard cert (its window opens 30 days before a
|
||||
# 90-day expiry; a conformance run cannot wait, and forcing it would rotate the
|
||||
# platform's serving credential). Step 2 asserts the schedule; step 4 asserts the
|
||||
# mechanism that will act on it.
|
||||
# * Not proven: certificate expiry ALERTING — docs/08 delegates that to R-OBS-04
|
||||
# (l3-fault-injection/alerting).
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: cert-lifecycle-automated
|
||||
labels:
|
||||
conformance.home.k3s/R-TLS-02: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-TLS-02"
|
||||
conformance.home.k3s/layer: "L2"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Given cert-manager and a TLS-terminating Gateway, when the serving certificate's
|
||||
provenance and renewal schedule are read and a new certificate is requested, left
|
||||
to renew, and removed, then issuance, renewal and withdrawal all happen with no
|
||||
human step — the serving cert was controller-issued and auto-approved, its
|
||||
renewal is scheduled ahead of expiry, and a declared certificate issues and
|
||||
reissues itself unattended.
|
||||
spec:
|
||||
timeouts:
|
||||
apply: 60s
|
||||
assert: 3m
|
||||
exec: 10m
|
||||
steps:
|
||||
# ---------------------------------------------------------------- READ-ONLY HALF
|
||||
- name: given-cert-manager-is-reconciled-and-healthy
|
||||
description: >-
|
||||
The controller that issues and renews must be running, and must have been
|
||||
installed by Flux rather than by hand — otherwise "automatic" has no engine.
|
||||
try:
|
||||
- script:
|
||||
content: |
|
||||
set -u
|
||||
# Discover cert-manager's Deployments by chart label, in whatever namespace
|
||||
# the release lives (nothing about the environment is assumed).
|
||||
sel='app.kubernetes.io/instance=cert-manager'
|
||||
rows="$(kubectl get deploy -A -l "$sel" \
|
||||
-o custom-columns='NS:.metadata.namespace,N:.metadata.name,RDY:.status.readyReplicas' \
|
||||
--no-headers 2>/dev/null || true)"
|
||||
echo "$rows"
|
||||
[ -n "$rows" ] || { echo "FAIL: no cert-manager Deployment found ($sel)"; exit 1; }
|
||||
n=0
|
||||
echo "$rows" | while read -r ns name rdy; do
|
||||
[ "${rdy:-0}" -ge 1 ] || { echo "FAIL: $ns/$name has no ready replica"; exit 1; }
|
||||
done || exit 1
|
||||
n="$(echo "$rows" | wc -l)"
|
||||
# controller + webhook + cainjector: without the webhook a Certificate is
|
||||
# rejected, without cainjector the webhook has no CA — all three or nothing.
|
||||
[ "$n" -ge 3 ] || { echo "FAIL: expected >=3 cert-manager Deployments, saw $n"; exit 1; }
|
||||
# GitOps-managed, not hand-installed: Flux labels what it applies.
|
||||
ns="$(echo "$rows" | head -1 | awk '{print $1}')"
|
||||
lbl="$(kubectl get deploy -n "$ns" -l "$sel" \
|
||||
-o jsonpath='{.items[0].metadata.labels.helm\.toolkit\.fluxcd\.io/name}' 2>/dev/null || true)"
|
||||
[ -n "$lbl" ] || { echo "FAIL: cert-manager is not owned by a Flux HelmRelease"; exit 1; }
|
||||
echo "CERT_MANAGER_HEALTHY_OK ($n deployments, flux HelmRelease $lbl)"
|
||||
check:
|
||||
(contains($stdout, 'CERT_MANAGER_HEALTHY_OK')): true
|
||||
|
||||
- name: then-the-serving-certificate-was-issued-by-the-controller
|
||||
description: >-
|
||||
Follow the Gateway's terminating listener to the Secret it serves, and prove
|
||||
that Secret's provenance is cert-manager — annotation, CertificateRequest
|
||||
requester, automatic approval, and a Ready ACME issuer.
|
||||
try:
|
||||
- script:
|
||||
content: |
|
||||
set -u
|
||||
# --- discover the serving Secret from the Gateway itself -------------------
|
||||
gw="$(kubectl get gateway -A \
|
||||
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}' \
|
||||
2>/dev/null | head -1)"
|
||||
[ -n "$gw" ] || { echo "FAIL: no Gateway found — nothing terminates TLS"; exit 1; }
|
||||
gns="${gw% *}"; gname="${gw#* }"
|
||||
sec="$(kubectl -n "$gns" get gateway "$gname" \
|
||||
-o jsonpath='{.spec.listeners[?(@.tls.mode=="Terminate")].tls.certificateRefs[0].name}' \
|
||||
2>/dev/null | awk '{print $1}')"
|
||||
[ -n "$sec" ] || { echo "FAIL: Gateway $gns/$gname terminates no TLS listener"; exit 1; }
|
||||
echo "gateway=$gns/$gname serving-secret=$sec"
|
||||
|
||||
# --- provenance (1): only the controller writes this annotation ------------
|
||||
cert="$(kubectl -n "$gns" get secret "$sec" \
|
||||
-o jsonpath='{.metadata.annotations.cert-manager\.io/certificate-name}' 2>/dev/null || true)"
|
||||
[ -n "$cert" ] || { echo "FAIL: Secret $gns/$sec carries no cert-manager.io/certificate-name annotation — it was not produced by cert-manager"; exit 1; }
|
||||
crt="$(kubectl -n "$gns" get secret "$sec" -o jsonpath='{.data.tls\.crt}' 2>/dev/null || true)"
|
||||
[ -n "$crt" ] || { echo "FAIL: Secret $gns/$sec holds no tls.crt"; exit 1; }
|
||||
|
||||
# --- provenance (2): the Certificate is Ready and has a revision -----------
|
||||
ready="$(kubectl -n "$gns" get certificate "$cert" \
|
||||
-o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)"
|
||||
rev="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.status.revision}' 2>/dev/null || true)"
|
||||
echo "certificate=$gns/$cert ready=$ready revision=$rev"
|
||||
[ "$ready" = "True" ] || { echo "FAIL: Certificate $cert is not Ready"; exit 1; }
|
||||
[ -n "$rev" ] && [ "$rev" -ge 1 ] || { echo "FAIL: Certificate $cert has no issued revision"; exit 1; }
|
||||
|
||||
# --- provenance (3): the request was made BY THE CONTROLLER and auto-approved
|
||||
# spec.username is the identity that created the CertificateRequest. A human
|
||||
# operator would appear as a user/OIDC identity, not a ServiceAccount.
|
||||
cr="$(kubectl -n "$gns" get certificaterequest \
|
||||
-o jsonpath="{range .items[?(@.metadata.annotations.cert-manager\.io/certificate-name=='${cert}')]}{.metadata.name}{' '}{.spec.username}{'\n'}{end}" \
|
||||
2>/dev/null | tail -1)"
|
||||
[ -n "$cr" ] || { echo "FAIL: no CertificateRequest for $cert — cannot show who requested it"; exit 1; }
|
||||
crname="${cr% *}"; requester="${cr#* }"
|
||||
echo "certificaterequest=$crname requester=$requester"
|
||||
case "$requester" in
|
||||
system:serviceaccount:*) : ;;
|
||||
*) echo "FAIL: CertificateRequest $crname was submitted by '$requester', not a controller ServiceAccount"; exit 1 ;;
|
||||
esac
|
||||
appr="$(kubectl -n "$gns" get certificaterequest "$crname" \
|
||||
-o jsonpath='{.status.conditions[?(@.type=="Approved")].status}' 2>/dev/null || true)"
|
||||
[ "$appr" = "True" ] || { echo "FAIL: CertificateRequest $crname is not Approved=True (approval would be a manual step)"; exit 1; }
|
||||
|
||||
# --- provenance (4): an automated ACME issuer, and it is Ready -------------
|
||||
ikind="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.spec.issuerRef.kind}')"
|
||||
iname="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.spec.issuerRef.name}')"
|
||||
echo "issuer=$ikind/$iname"
|
||||
if [ "$ikind" = "ClusterIssuer" ]; then
|
||||
iready="$(kubectl get clusterissuer "$iname" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)"
|
||||
iacme="$(kubectl get clusterissuer "$iname" -o jsonpath='{.spec.acme.server}' 2>/dev/null || true)"
|
||||
else
|
||||
iready="$(kubectl -n "$gns" get issuer "$iname" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)"
|
||||
iacme="$(kubectl -n "$gns" get issuer "$iname" -o jsonpath='{.spec.acme.server}' 2>/dev/null || true)"
|
||||
fi
|
||||
[ "$iready" = "True" ] || { echo "FAIL: issuer $iname is not Ready — renewal would need a human to fix it"; exit 1; }
|
||||
[ -n "$iacme" ] || { echo "FAIL: issuer $iname is not an ACME issuer"; exit 1; }
|
||||
echo "SERVING_CERT_CONTROLLER_ISSUED_OK"
|
||||
check:
|
||||
(contains($stdout, 'SERVING_CERT_CONTROLLER_ISSUED_OK')): true
|
||||
|
||||
- name: then-an-unattended-renewal-is-already-scheduled
|
||||
description: >-
|
||||
cert-manager writes status.renewalTime only after computing when it will renew.
|
||||
Asserting it is in the future and before expiry proves the renewal is scheduled
|
||||
and that nothing is waiting on a human (docs/08 "renews well before expiry").
|
||||
try:
|
||||
- script:
|
||||
content: |
|
||||
set -u
|
||||
gw="$(kubectl get gateway -A \
|
||||
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}' | head -1)"
|
||||
gns="${gw% *}"; gname="${gw#* }"
|
||||
sec="$(kubectl -n "$gns" get gateway "$gname" \
|
||||
-o jsonpath='{.spec.listeners[?(@.tls.mode=="Terminate")].tls.certificateRefs[0].name}' | awk '{print $1}')"
|
||||
cert="$(kubectl -n "$gns" get secret "$sec" \
|
||||
-o jsonpath='{.metadata.annotations.cert-manager\.io/certificate-name}')"
|
||||
renew="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.status.renewalTime}')"
|
||||
after="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.status.notAfter}')"
|
||||
[ -n "$renew" ] || { echo "FAIL: Certificate $cert has no status.renewalTime — no renewal is scheduled"; exit 1; }
|
||||
[ -n "$after" ] || { echo "FAIL: Certificate $cert has no status.notAfter"; exit 1; }
|
||||
now_s="$(date -u +%s)"
|
||||
r_s="$(date -u -d "$renew" +%s)"
|
||||
a_s="$(date -u -d "$after" +%s)"
|
||||
echo "now=$(date -u -d "@$now_s" +%FT%TZ) renewalTime=$renew notAfter=$after"
|
||||
echo "renewal is in $(( (r_s - now_s) / 86400 ))d; expiry in $(( (a_s - now_s) / 86400 ))d"
|
||||
[ "$r_s" -gt "$now_s" ] || { echo "FAIL: renewalTime is in the PAST — the scheduled renewal did not happen"; exit 1; }
|
||||
[ "$r_s" -lt "$a_s" ] || { echo "FAIL: renewalTime is not before notAfter — renewal would come too late"; exit 1; }
|
||||
echo "RENEWAL_SCHEDULED_OK"
|
||||
check:
|
||||
(contains($stdout, 'RENEWAL_SCHEDULED_OK')): true
|
||||
|
||||
# ------------------------------------------------- MUTATING HALF (hermetic namespace)
|
||||
- name: when-a-certificate-is-declared-then-it-is-issued-with-no-human-step
|
||||
description: >-
|
||||
Declare an Issuer and a Certificate in this test's own namespace, with a renewal
|
||||
window that opens one minute after issuance. Nothing else is done to them for the
|
||||
rest of the test — every later transition is the controller acting on its own.
|
||||
try:
|
||||
- apply:
|
||||
resource:
|
||||
# selfSigned: no external CA, no rate limit, no DNS write. See the header for
|
||||
# what this proves (controller automation) and what it does not (the ACME path).
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Issuer
|
||||
metadata:
|
||||
name: conformance-lifecycle
|
||||
spec:
|
||||
selfSigned: {}
|
||||
- apply:
|
||||
resource:
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: conformance-lifecycle
|
||||
spec:
|
||||
secretName: conformance-lifecycle-tls
|
||||
# 1h is cert-manager's minimum duration; renewing 59m before expiry puts the
|
||||
# renewal instant ~1 minute after each issuance, so an unattended renewal is
|
||||
# observable inside a test run instead of 30 days from now.
|
||||
duration: 1h
|
||||
renewBefore: 3540s
|
||||
commonName: conformance-lifecycle.invalid
|
||||
dnsNames:
|
||||
- conformance-lifecycle.invalid
|
||||
privateKey:
|
||||
rotationPolicy: Always
|
||||
issuerRef:
|
||||
kind: Issuer
|
||||
name: conformance-lifecycle
|
||||
- assert:
|
||||
resource:
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: conformance-lifecycle
|
||||
status:
|
||||
conditions:
|
||||
- type: Ready
|
||||
status: "True"
|
||||
revision: 1
|
||||
- script:
|
||||
content: |
|
||||
set -u
|
||||
crt="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \
|
||||
-o jsonpath='{.data.tls\.crt}' 2>/dev/null || true)"
|
||||
key="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \
|
||||
-o jsonpath='{.data.tls\.key}' 2>/dev/null || true)"
|
||||
[ -n "$crt" ] && [ -n "$key" ] || { echo "FAIL: no certificate material was issued into conformance-lifecycle-tls"; exit 1; }
|
||||
# Record the serving bytes so the renewal step can prove they CHANGED.
|
||||
echo "$crt" | cksum > /tmp/conformance-cert-r1.cksum
|
||||
echo "revision-1 cert fingerprint: $(cat /tmp/conformance-cert-r1.cksum)"
|
||||
echo "CERT_ISSUED_OK"
|
||||
check:
|
||||
(contains($stdout, 'CERT_ISSUED_OK')): true
|
||||
|
||||
- name: then-it-renews-itself-before-expiry-with-no-human-step
|
||||
description: >-
|
||||
Nothing touches the Certificate here. Poll until the controller has reissued it
|
||||
at least twice on its own and the serving bytes differ from revision 1 — docs/08's
|
||||
"observed across a renewal", and every request auto-approved.
|
||||
try:
|
||||
- script:
|
||||
content: |
|
||||
set -u
|
||||
want=3 # revision 1 = issuance; >=3 means TWO unattended renewals
|
||||
for i in $(seq 1 30); do
|
||||
rev="$(kubectl -n "$NAMESPACE" get certificate conformance-lifecycle \
|
||||
-o jsonpath='{.status.revision}' 2>/dev/null || echo 0)"
|
||||
echo "poll $i: revision=${rev:-0}"
|
||||
[ "${rev:-0}" -ge "$want" ] && break
|
||||
sleep 20
|
||||
done
|
||||
rev="$(kubectl -n "$NAMESPACE" get certificate conformance-lifecycle \
|
||||
-o jsonpath='{.status.revision}' 2>/dev/null || echo 0)"
|
||||
[ "${rev:-0}" -ge "$want" ] || { echo "FAIL: revision reached only ${rev:-0} — cert-manager did not renew unattended"; exit 1; }
|
||||
|
||||
# The serving material must actually have rotated, not just the revision count.
|
||||
now="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \
|
||||
-o jsonpath='{.data.tls\.crt}' | cksum)"
|
||||
was="$(cat /tmp/conformance-cert-r1.cksum)"
|
||||
echo "r1=$was now=$now"
|
||||
[ "$now" != "$was" ] || { echo "FAIL: the Secret still holds the revision-1 certificate — nothing was actually reissued"; exit 1; }
|
||||
|
||||
# Every renewal request was approved by the controller, never by a person.
|
||||
bad=0
|
||||
for cr in $(kubectl -n "$NAMESPACE" get certificaterequest -o name 2>/dev/null); do
|
||||
st="$(kubectl -n "$NAMESPACE" get "$cr" \
|
||||
-o jsonpath='{.status.conditions[?(@.type=="Approved")].status}' 2>/dev/null || true)"
|
||||
who="$(kubectl -n "$NAMESPACE" get "$cr" -o jsonpath='{.spec.username}' 2>/dev/null || true)"
|
||||
echo " $cr approved=$st requester=$who"
|
||||
[ "$st" = "True" ] || bad=1
|
||||
case "$who" in system:serviceaccount:*) : ;; *) bad=1 ;; esac
|
||||
done
|
||||
[ "$bad" = 0 ] || { echo "FAIL: a CertificateRequest was not auto-approved, or was not submitted by a controller"; exit 1; }
|
||||
echo "CERT_RENEWED_UNATTENDED_OK (revision=$rev)"
|
||||
check:
|
||||
(contains($stdout, 'CERT_RENEWED_UNATTENDED_OK')): true
|
||||
|
||||
- name: then-removing-the-certificate-withdraws-the-serving-material
|
||||
description: >-
|
||||
docs/08 reads revocation as "removing/rotating a certificate is a declarative
|
||||
change". The declarative change is the deletion; withdrawal of the credential
|
||||
must follow with no manual cleanup. EXPECTED TO FAIL TODAY — cert-manager needs
|
||||
`enableCertificateOwnerRef: true` for the Secret to be garbage-collected, and the
|
||||
tracked HelmRelease does not set it. Kept as the requirement reads (R-TEST-06).
|
||||
try:
|
||||
- delete:
|
||||
ref:
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
name: conformance-lifecycle
|
||||
- script:
|
||||
content: |
|
||||
set -u
|
||||
for i in $(seq 1 15); do
|
||||
if ! kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls >/dev/null 2>&1; then
|
||||
echo "CERT_WITHDRAWN_OK (serving material gone after $((i * 4))s)"
|
||||
exit 0
|
||||
fi
|
||||
sleep 4
|
||||
done
|
||||
owners="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \
|
||||
-o jsonpath='{.metadata.ownerReferences}' 2>/dev/null || true)"
|
||||
echo "FAIL: the Certificate was deleted but Secret conformance-lifecycle-tls still"
|
||||
echo " holds usable certificate material (ownerReferences=${owners:-none})."
|
||||
echo " Removing a certificate therefore does NOT withdraw the credential, so"
|
||||
echo " R-TLS-02's 'revoked ... without manual steps' does not hold. FIX:"
|
||||
echo " set enableCertificateOwnerRef: true in platform/cert-manager values"
|
||||
echo " (owner decision — WS-TEST does not weaken the assertion instead)."
|
||||
exit 1
|
||||
check:
|
||||
(contains($stdout, 'CERT_WITHDRAWN_OK')): true
|
||||
finally:
|
||||
# Namespace teardown removes both objects anyway; explicit so a leftover Secret
|
||||
# from the expected failure above is never left holding key material.
|
||||
- script:
|
||||
content: |
|
||||
kubectl -n "$NAMESPACE" delete secret conformance-lifecycle-tls --ignore-not-found --wait=false || true
|
||||
kubectl -n "$NAMESPACE" delete certificate conformance-lifecycle --ignore-not-found --wait=false || true
|
||||
kubectl -n "$NAMESPACE" delete issuer conformance-lifecycle --ignore-not-found --wait=false || true
|
||||
rm -f /tmp/conformance-cert-r1.cksum || true
|
||||
Reference in new issue
Block a user