jakman85andClaude Opus 5
6a8953e288
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
184 lines
10 KiB
Bash
Executable File
184 lines
10 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# ci/l0.sh — Stage 2+3: ServiceDefinition schema validation + Kyverno L0 policy gate.
|
|
# Both run in CI WITHOUT a cluster (docs/14 §"Where each layer runs") — live now.
|
|
#
|
|
# Stage 2 schema : catalog/*.yaml validated against the ServiceDefinition
|
|
# schema (owner: T-0.3, schemas/).
|
|
# Stage 3 L0 : `kyverno test` over platform/kyverno/ — default-deny baseline
|
|
# present, no plaintext secrets, required labels/egress-class,
|
|
# ServiceDefinition schema conformance (owner: T-0.5, ADR-0011).
|
|
# Stage 3b L0 : conformance/run.sh --layer l0 — the WHOLE-REPO L0 invariants that
|
|
# a per-resource policy cannot express (secret hygiene across every
|
|
# tracked file, least-privilege RBAC, spec-immutability enforcement).
|
|
#
|
|
# Integration seams are env-overridable (see ci/lib.sh + README). When an owning
|
|
# subtree is still a placeholder, this reports SKIPPED (exit 3) for that sub-stage
|
|
# rather than a false pass — the gate then knows the seam is unwired, not green.
|
|
set -euo pipefail
|
|
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
# shellcheck disable=SC1091
|
|
source "${CI_DIR}/lib.sh"
|
|
|
|
rc=0
|
|
|
|
# ---- Stage 2: ServiceDefinition schema validation ---------------------------
|
|
log "Stage: ServiceDefinition schema validation (catalog/ against schemas/)"
|
|
if ! entrypoint_ready "$SCHEMA_VALIDATE_CMD"; then
|
|
skip "schema validator not present at '${SCHEMA_VALIDATE_CMD}' (T-0.3 / schemas/ not delivered yet) — UNWIRED, not green"
|
|
summary "- schema: **SKIPPED** (validator entrypoint \`${SCHEMA_VALIDATE_CMD}\` missing — T-0.3)"
|
|
else
|
|
if [ -d "$CATALOG_DIR" ] && compgen -G "${CATALOG_DIR}/*.y*ml" >/dev/null 2>&1; then
|
|
if "$SCHEMA_VALIDATE_CMD" "${CATALOG_DIR}"/*.y*ml; then
|
|
ok "schema validation passed"
|
|
summary "- schema: **passed**"
|
|
else
|
|
err "schema validation failed"; summary "- schema: **FAILED**"; rc=1
|
|
fi
|
|
else
|
|
log "no ServiceDefinitions in ${CATALOG_DIR} yet — validating validator is callable only"
|
|
"$SCHEMA_VALIDATE_CMD" --version >/dev/null 2>&1 || true
|
|
ok "schema validator present; no catalog entries to validate"
|
|
summary "- schema: **passed** (no catalog entries yet)"
|
|
fi
|
|
fi
|
|
|
|
# ---- Stage 3: Kyverno L0 policy gate ----------------------------------------
|
|
log "Stage: Kyverno L0 policy gate (kyverno test ${KYVERNO_POLICY_DIR})"
|
|
# `kyverno test <dir>` discovers kyverno-test.yaml files recursively.
|
|
if ! have kyverno; then
|
|
skip "kyverno CLI not installed (pin v${KYVERNO_VERSION}) — cannot run L0 gate"
|
|
summary "- kyverno-l0: **SKIPPED** (kyverno CLI not installed)"
|
|
# Missing tool is an environment gap, not a code pass -> exit 3 unless already failing.
|
|
[ "$rc" -eq 0 ] && rc=3
|
|
elif [ ! -d "$KYVERNO_POLICY_DIR" ] || ! compgen -G "${KYVERNO_POLICY_DIR}/**/kyverno-test.yaml" >/dev/null 2>&1 \
|
|
&& ! find "$KYVERNO_POLICY_DIR" -name 'kyverno-test.yaml' 2>/dev/null | grep -q .; then
|
|
skip "no kyverno-test.yaml under '${KYVERNO_POLICY_DIR}' (T-0.5 / platform/kyverno not delivered yet) — UNWIRED, not green"
|
|
summary "- kyverno-l0: **SKIPPED** (no policy tests yet — T-0.5)"
|
|
else
|
|
# The ClusterPolicies carry a Flux postBuild token in validationFailureAction
|
|
# (${KYVERNO_VALIDATION_ACTION}, substituted in-cluster from
|
|
# clusters/base/cluster-vars.yaml). The offline `kyverno test` CLI does NOT run
|
|
# Flux substitution, so the literal token is an invalid enum value and those
|
|
# policies fail to load (no result rows -> "Not found"). Render a throwaway copy
|
|
# with the base/prod value (Enforce — the strict posture the gate must prove)
|
|
# before testing. This does not change what the policies enforce in-cluster.
|
|
: "${KYVERNO_VALIDATION_ACTION:=Enforce}"
|
|
l0_render="$(mktemp -d)"
|
|
cp -a "${KYVERNO_POLICY_DIR}/." "$l0_render/"
|
|
find "$l0_render" -type f -name '*.yaml' -print0 \
|
|
| xargs -0 sed -i "s/\${KYVERNO_VALIDATION_ACTION}/${KYVERNO_VALIDATION_ACTION}/g"
|
|
if kyverno test "$l0_render"; then
|
|
ok "Kyverno L0 policy tests passed"
|
|
summary "- kyverno-l0: **passed**"
|
|
else
|
|
err "Kyverno L0 policy tests failed"; summary "- kyverno-l0: **FAILED**"; rc=1
|
|
fi
|
|
rm -rf "$l0_render"
|
|
fi
|
|
|
|
# ---- Stage 3b: L0 STATIC conformance checks ---------------------------------
|
|
# docs/14's L0 row is "manifests/ServiceDefinitions obey invariants", which is wider
|
|
# than what Kyverno alone can express over single resources: whole-repo invariants
|
|
# (no plaintext secrets anywhere in git, least-privilege RBAC across every tracked
|
|
# binding, the spec-immutability gate's own behaviour) live in the conformance
|
|
# harness's tests/l0-static subtree and run with no cluster. This stage executes them
|
|
# through the same single-command runner CI and a laptop both use (R-TEST-02).
|
|
log "Stage: L0 static conformance checks (${CONFORMANCE_CMD} --layer l0)"
|
|
if ! entrypoint_ready "$CONFORMANCE_CMD"; then
|
|
skip "conformance runner not present at '${CONFORMANCE_CMD}' — UNWIRED, not green"
|
|
summary "- l0-static: **SKIPPED** (conformance runner missing)"
|
|
if [ "$rc" -eq 0 ]; then rc=3; fi
|
|
else
|
|
load_touched_requirements
|
|
if "$CONFORMANCE_CMD" --layer l0 ${REQ_ARGS[@]+"${REQ_ARGS[@]}"}; then
|
|
ok "L0 static conformance checks passed"
|
|
summary "- l0-static: **passed**"
|
|
else
|
|
err "L0 static conformance checks failed"
|
|
summary "- l0-static: **FAILED**"
|
|
rc=1
|
|
fi
|
|
fi
|
|
|
|
# ---- Stage 3c: Flux post-build substitution safety (UNCONDITIONAL) ----------
|
|
# R-ENV-02's promote-once mechanism is `postBuild.substituteFrom cluster-vars`, and it
|
|
# FAILS OPEN: an undefined ${TOKEN} becomes the empty string and the Kustomization still
|
|
# reports Ready. That emptied Cilium's k8sServiceHost and broke the CNI on 2/3 dev nodes
|
|
# (DECISIONS-LOG 2026-09-01, "substitution race that fails OPEN"). The check asserts every
|
|
# ${...} in every substituted path is DEFINED in cluster-vars or deliberately ESCAPED
|
|
# `$${...}` — i.e. that enabling the StrictPostBuildSubstitutions feature gate cannot fail
|
|
# a legitimate manifest, and that no expansion silently empties today.
|
|
#
|
|
# WHY THIS IS ITS OWN STAGE and not left to Stage 3b: 3b scopes the conformance run to the
|
|
# TOUCHED requirements (R-TEST-03), and ci/touched-requirements.sh's coarse path map keys
|
|
# only clusters/* to R-ENV-02 — an edit to a SUBSTITUTED manifest under apps/ or platform/
|
|
# attributes no requirement, so the requirement-scoped selection would filter this check
|
|
# out of exactly the change that introduces the bug. The invariant is whole-repo (same
|
|
# reasoning as spec-immutability's own gate stage), so it runs on EVERY gate invocation.
|
|
# When 3b's touched set does include R-ENV-02/R-GITOPS-01 the check runs twice; it is a
|
|
# sub-second offline scan with no side effects, so the duplication is not worth avoiding.
|
|
SUBST_CHECK="${SUBST_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/flux-postbuild-substitution/check.sh}"
|
|
log "Stage: Flux postBuild substitution safety (${SUBST_CHECK})"
|
|
if [ ! -f "$SUBST_CHECK" ]; then
|
|
skip "substitution check not present at '${SUBST_CHECK}' — UNWIRED, not green"
|
|
summary "- subst-safety: **SKIPPED** (check missing)"
|
|
if [ "$rc" -eq 0 ]; then rc=3; fi
|
|
else
|
|
subst_rc=0
|
|
bash "$SUBST_CHECK" || subst_rc=$?
|
|
case "$subst_rc" in
|
|
0) ok "Flux postBuild substitution safety passed"
|
|
summary "- subst-safety: **passed**" ;;
|
|
3) skip "substitution check SKIPPED (clusters/ absent) — not a pass"
|
|
summary "- subst-safety: **SKIPPED** (nothing to audit)"
|
|
if [ "$rc" -eq 0 ]; then rc=3; fi ;;
|
|
*) err "Flux postBuild substitution safety FAILED"
|
|
summary "- subst-safety: **FAILED**"; rc=1 ;;
|
|
esac
|
|
fi
|
|
|
|
# ---- Stage 3d: Keycloak client-secret declaration + reconciler (UNCONDITIONAL) ----
|
|
# The Keycloak realm import resolves each OIDC client's secret from an `(env:NAME)`
|
|
# placeholder. When the operator cannot resolve one it stores the LITERAL placeholder as
|
|
# the client secret AND THE IMPORT JOB STILL REPORTS SUCCESS, then rests at Done=True and
|
|
# never re-runs. On 2026-09-06 that had all 12 prod clients unusable ("Invalid client or
|
|
# Invalid client credentials") behind an entirely green cluster; dev had one. The check
|
|
# asserts the declaration is self-consistent (every placeholder defined, every Secret key
|
|
# present, names matching the reconciler's contract), that the Flux ordering is a REAL
|
|
# dependsOn edge rather than a comment, and — by extracting the shipped script from its
|
|
# ConfigMap and running its --selftest — that the reconciler is compare-then-write,
|
|
# idempotent, non-logging and blast-radius-bounded.
|
|
#
|
|
# WHY ITS OWN UNCONDITIONAL STAGE, same reasoning as 3c: Stage 3b scopes the conformance
|
|
# run to the TOUCHED requirements (R-TEST-03). This check's invariant spans platform/,
|
|
# platform-config/ and clusters/, so a change that breaks it need not attribute R-IDP-03/04
|
|
# and the requirement-scoped selection would filter it out of exactly the change that
|
|
# introduces the bug. It is a sub-second offline scan plus a loopback selftest, with no
|
|
# side effects, so running it twice when 3b also selects it is not worth avoiding.
|
|
KEYCLOAK_SECRET_CHECK="${KEYCLOAK_SECRET_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh}"
|
|
log "Stage: Keycloak client-secret reconciler safety (${KEYCLOAK_SECRET_CHECK})"
|
|
if [ ! -f "$KEYCLOAK_SECRET_CHECK" ]; then
|
|
skip "keycloak client-secret check not present at '${KEYCLOAK_SECRET_CHECK}' — UNWIRED, not green"
|
|
summary "- keycloak-client-secrets: **SKIPPED** (check missing)"
|
|
if [ "$rc" -eq 0 ]; then rc=3; fi
|
|
else
|
|
kc_rc=0
|
|
bash "$KEYCLOAK_SECRET_CHECK" || kc_rc=$?
|
|
case "$kc_rc" in
|
|
0) ok "Keycloak client-secret declaration + reconciler passed"
|
|
summary "- keycloak-client-secrets: **passed**" ;;
|
|
3) skip "keycloak client-secret check SKIPPED (platform/keycloak absent) — not a pass"
|
|
summary "- keycloak-client-secrets: **SKIPPED** (nothing to audit)"
|
|
if [ "$rc" -eq 0 ]; then rc=3; fi ;;
|
|
*) err "Keycloak client-secret declaration + reconciler FAILED"
|
|
summary "- keycloak-client-secrets: **FAILED**"; rc=1 ;;
|
|
esac
|
|
fi
|
|
|
|
case "$rc" in
|
|
0) ok "L0 stage complete" ;;
|
|
3) skip "L0 stage incomplete (unwired seam / missing tool) — not a pass" ;;
|
|
*) err "L0 stage failed" ;;
|
|
esac
|
|
exit "$rc"
|