jakman85andClaude Opus 5
6a8953e288
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
70 lines
4.0 KiB
YAML
70 lines
4.0 KiB
YAML
# =============================================================================
|
|
# L0 STATIC — certificate lifecycle needs no manual step (R-TLS-02) [EXECUTABLE]
|
|
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
|
# Spec: docs/00 R-TLS-02 ("Certificates MUST be automatically issued, renewed, and
|
|
# revoked without manual steps"); mechanism in docs/08 + ADR-0006.
|
|
#
|
|
# R-TLS-02 is one adverbial clause plus three runtime verbs. This test asserts the
|
|
# clause — "without manual steps" — which in a GitOps platform is a property of the
|
|
# REPOSITORY: a human step in the certificate path leaves evidence in git (committed
|
|
# cert material, a Gateway serving a Secret nothing declares, an issuer with no
|
|
# automated solver, a credential never delivered, auto-approval switched off, or CRs
|
|
# nothing reconciles). The three verbs are observed by the companion L2 test.
|
|
#
|
|
# ENVIRONMENT-AGNOSTIC (R-ENV-02): the issuer a Certificate names is `${ACME_CLUSTER_ISSUER}`,
|
|
# so the check resolves it against the BASE cluster-vars default AND every overlay's value and
|
|
# requires each resolved issuer to exist. dev's issuer and prod's issuer are both verified from
|
|
# one authored assertion, with no hostname, address or environment name written down.
|
|
#
|
|
# GIVEN the repository as git tracks it,
|
|
# WHEN every serving-TLS reference, Certificate, Issuer and the cert-manager release
|
|
# is inspected offline,
|
|
# THEN every Gateway listener that terminates TLS serves a Secret a tracked Certificate
|
|
# produces; no certificate material is committed anywhere; every Certificate's
|
|
# issuer resolves — in EVERY environment — to a tracked ACME issuer with an
|
|
# automated dns01/http01 solver whose credential is SOPS-delivered from git;
|
|
# cert-manager's automatic request approval is not disabled; and every one of
|
|
# those objects is rendered by a Flux Kustomization, so nothing in the chain
|
|
# depends on a human running kubectl.
|
|
#
|
|
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git + PyYAML, no
|
|
# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and
|
|
# this Chainsaw Test wraps the SAME script.
|
|
#
|
|
# KNOWN, DELIBERATE GAPS (named in check.sh's header too — this test claims NONE of them):
|
|
# * ISSUED / RENEWED as runtime events: L2 (l2-behavioral/cert-lifecycle-automated).
|
|
# * REVOKED: not proven at any layer today. cert-manager performs no ACME revocation,
|
|
# and the spec's own reading (docs/08: "removing/rotating a certificate is a
|
|
# declarative change") needs `enableCertificateOwnerRef`, which the tracked
|
|
# cert-manager HelmRelease does not set — so a removed Certificate leaves its
|
|
# serving Secret behind. The L2 test asserts the withdrawal behaviour and is
|
|
# EXPECTED TO FAIL until that knob is set; escalated, not softened (R-TEST-06).
|
|
# * That the live cluster is serving the tracked Certificate (cluster state).
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: tls-lifecycle-automated
|
|
labels:
|
|
conformance.home.k3s/R-TLS-02: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-TLS-02"
|
|
conformance.home.k3s/layer: "L0"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
Given the tracked tree, when the certificate path is inspected offline, then no
|
|
manual step exists anywhere in it: serving Secrets are cert-manager-produced, no
|
|
cert material is committed, every environment's issuer is a tracked ACME issuer
|
|
with an automated solver and a git-delivered credential, auto-approval is on, and
|
|
the whole chain is Flux-reconciled.
|
|
spec:
|
|
steps:
|
|
- name: then-no-manual-step-exists-in-the-certificate-path
|
|
description: >-
|
|
Run the offline assertion script over the tracked tree (same script ci/l0.sh
|
|
and conformance/run.sh --layer l0 execute).
|
|
try:
|
|
- script:
|
|
content: bash ./check.sh
|