feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
1 parent
d1a6628a97
commit
6a8953e288
46 files changed
+7898
-71
No files matched your search
@@ -0,0 +1,69 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — certificate lifecycle needs no manual step (R-TLS-02) [EXECUTABLE]
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
# Spec: docs/00 R-TLS-02 ("Certificates MUST be automatically issued, renewed, and
|
||||
# revoked without manual steps"); mechanism in docs/08 + ADR-0006.
|
||||
#
|
||||
# R-TLS-02 is one adverbial clause plus three runtime verbs. This test asserts the
|
||||
# clause — "without manual steps" — which in a GitOps platform is a property of the
|
||||
# REPOSITORY: a human step in the certificate path leaves evidence in git (committed
|
||||
# cert material, a Gateway serving a Secret nothing declares, an issuer with no
|
||||
# automated solver, a credential never delivered, auto-approval switched off, or CRs
|
||||
# nothing reconciles). The three verbs are observed by the companion L2 test.
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC (R-ENV-02): the issuer a Certificate names is `${ACME_CLUSTER_ISSUER}`,
|
||||
# so the check resolves it against the BASE cluster-vars default AND every overlay's value and
|
||||
# requires each resolved issuer to exist. dev's issuer and prod's issuer are both verified from
|
||||
# one authored assertion, with no hostname, address or environment name written down.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN every serving-TLS reference, Certificate, Issuer and the cert-manager release
|
||||
# is inspected offline,
|
||||
# THEN every Gateway listener that terminates TLS serves a Secret a tracked Certificate
|
||||
# produces; no certificate material is committed anywhere; every Certificate's
|
||||
# issuer resolves — in EVERY environment — to a tracked ACME issuer with an
|
||||
# automated dns01/http01 solver whose credential is SOPS-delivered from git;
|
||||
# cert-manager's automatic request approval is not disabled; and every one of
|
||||
# those objects is rendered by a Flux Kustomization, so nothing in the chain
|
||||
# depends on a human running kubectl.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git + PyYAML, no
|
||||
# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and
|
||||
# this Chainsaw Test wraps the SAME script.
|
||||
#
|
||||
# KNOWN, DELIBERATE GAPS (named in check.sh's header too — this test claims NONE of them):
|
||||
# * ISSUED / RENEWED as runtime events: L2 (l2-behavioral/cert-lifecycle-automated).
|
||||
# * REVOKED: not proven at any layer today. cert-manager performs no ACME revocation,
|
||||
# and the spec's own reading (docs/08: "removing/rotating a certificate is a
|
||||
# declarative change") needs `enableCertificateOwnerRef`, which the tracked
|
||||
# cert-manager HelmRelease does not set — so a removed Certificate leaves its
|
||||
# serving Secret behind. The L2 test asserts the withdrawal behaviour and is
|
||||
# EXPECTED TO FAIL until that knob is set; escalated, not softened (R-TEST-06).
|
||||
# * That the live cluster is serving the tracked Certificate (cluster state).
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: tls-lifecycle-automated
|
||||
labels:
|
||||
conformance.home.k3s/R-TLS-02: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-TLS-02"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Given the tracked tree, when the certificate path is inspected offline, then no
|
||||
manual step exists anywhere in it: serving Secrets are cert-manager-produced, no
|
||||
cert material is committed, every environment's issuer is a tracked ACME issuer
|
||||
with an automated solver and a git-delivered credential, auto-approval is on, and
|
||||
the whole chain is Flux-reconciled.
|
||||
spec:
|
||||
steps:
|
||||
- name: then-no-manual-step-exists-in-the-certificate-path
|
||||
description: >-
|
||||
Run the offline assertion script over the tracked tree (same script ci/l0.sh
|
||||
and conformance/run.sh --layer l0 execute).
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -0,0 +1,478 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — tls-lifecycle-automated
|
||||
# Serves: R-TLS-02 ("Certificates MUST be automatically **issued, renewed, and revoked**
|
||||
# without manual steps." — spec docs/00-vision-and-scope.md; mechanism in
|
||||
# docs/08-tls-and-certificates.md, ADR-0006.)
|
||||
#
|
||||
# WHICH CLAUSE THIS LAYER CAN PROVE — AND WHICH IT CANNOT
|
||||
# ------------------------------------------------------
|
||||
# R-TLS-02 has one adverbial clause ("without manual steps") and three verbs
|
||||
# ("issued", "renewed", "revoked"). In a GitOps platform the *without manual steps*
|
||||
# clause IS a repository property, and it is the half that can be proven with no
|
||||
# cluster: if a human had to mint, install, or approve a serving certificate, the
|
||||
# evidence of that human step would be IN THE REPO — a committed `tls.crt`, a Gateway
|
||||
# pointing at a Secret nothing declares, an issuer that no automated solver can
|
||||
# satisfy, a per-cert credential handed over out of band, or an admission path that
|
||||
# waits for a manual CertificateRequest approval. This check asserts that none of
|
||||
# those exist and that the automated path is completely wired instead.
|
||||
#
|
||||
# The three VERBS are runtime events. They are asserted by the companion L2 test
|
||||
# (conformance/tests/l2-behavioral/cert-lifecycle-automated), which observes an
|
||||
# issuance, an unattended renewal, and the withdrawal of a removed certificate. This
|
||||
# check does NOT claim them; see "WHAT THIS DOES *NOT* PROVE" below.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN every serving-TLS reference, Certificate, Issuer and the cert-manager release
|
||||
# are inspected offline,
|
||||
# THEN (1) at least one cert-manager `Certificate` is authored (vacuity floor: a repo
|
||||
# with no Certificate cannot pass this check by having nothing to check);
|
||||
# (2) every Gateway HTTPS listener that TERMINATES TLS references ONLY Secrets
|
||||
# that a tracked Certificate declares as its `secretName` — so the serving
|
||||
# credential is produced by the controller, never dropped in by hand;
|
||||
# (3) NO certificate material is tracked anywhere: no Secret manifest of type
|
||||
# kubernetes.io/tls, no tls.crt/tls.key/ca.crt key in any committed Secret
|
||||
# (plain OR SOPS-encrypted), and no PEM certificate body in any tracked
|
||||
# YAML — i.e. issuance is not, and cannot silently become, a human step;
|
||||
# (4) every Certificate's `issuerRef` resolves to a TRACKED issuer IN EVERY
|
||||
# ENVIRONMENT — the `${ACME_CLUSTER_ISSUER}` token is resolved against the
|
||||
# base cluster-vars default AND every overlay's value, so an environment
|
||||
# whose issuer does not exist is caught here rather than at prod bring-up;
|
||||
# (5) every issuer so referenced is an ACME issuer with a NON-EMPTY solver list
|
||||
# whose every solver is an automated challenge type (dns01 / http01) — the
|
||||
# same code path serves issuance AND renewal, so a solver a human must
|
||||
# complete would defeat both;
|
||||
# (6) each dns01 solver's provider credential is referenced BY NAME and that
|
||||
# Secret is delivered SOPS-encrypted from git via a kustomization — the
|
||||
# operator's one-time token hand-off (docs/08 OQ-4), never a per-cert step;
|
||||
# (7) cert-manager's automatic CertificateRequest approval is NOT disabled
|
||||
# (`disableAutoApproval`), and any `approveSignerNames` narrowing still
|
||||
# covers the ClusterIssuers actually used — a request that waits for a
|
||||
# human approver is the definition of a manual step;
|
||||
# (8) the whole chain is reconciled from git — cert-manager's HelmRelease, the
|
||||
# ClusterIssuers and the Certificate are each included in a kustomization
|
||||
# that a Flux Kustomization renders — so a deleted Certificate is
|
||||
# re-created and re-issued with no human action;
|
||||
# (9) no Certificate defeats renewal arithmetically (`renewBefore` >= `duration`
|
||||
# would mean the renewal instant never arrives).
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them):
|
||||
# - ISSUED: that an issuance actually happens. Asserted at L2 (step 2 of
|
||||
# l2-behavioral/cert-lifecycle-automated) and, for the real ACME/DNS-01 path,
|
||||
# evidenced by l2-behavioral/trusted-tls serving a publicly-trusted cert.
|
||||
# - RENEWED: that a renewal actually happens unattended. docs/08's acceptance gate
|
||||
# is explicit that this must be "observed across a renewal" — L2 step 3 does that
|
||||
# by fast-forwarding a certificate's renewal window; nothing static can.
|
||||
# - REVOKED: NOT PROVEN AT ANY LAYER TODAY, and not asserted here. Two distinct
|
||||
# readings, neither established: (a) ACME revocation — cert-manager does not
|
||||
# revoke certificates at all, so no repo state could show it; (b) the spec's own
|
||||
# reading in docs/08 ("removing/rotating a certificate is a declarative change")
|
||||
# — withdrawal of a REMOVED certificate's serving material requires cert-manager's
|
||||
# `enableCertificateOwnerRef`, which the tracked HelmRelease does NOT set (chart
|
||||
# default false). L2 step 4 asserts the withdrawal behaviour and is expected to
|
||||
# FAIL until that knob is set; escalated to the PM rather than softened (R-TEST-06).
|
||||
# - That the tracked Certificate is the one the live Gateway is actually serving
|
||||
# (cluster state), or that the ACME account/DNS credential is still valid.
|
||||
# - Certificate EXPIRY ALERTING, which docs/08 delegates to R-OBS-04 (see
|
||||
# l3-fault-injection/alerting).
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def tracked() -> list[str]:
|
||||
out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
|
||||
capture_output=True, text=True, check=True).stdout
|
||||
return [p for p in out.split("\0") if p]
|
||||
|
||||
|
||||
FILES = tracked()
|
||||
check(len(FILES) > 50, f"git ls-files returned only {len(FILES)} paths — not a real checkout")
|
||||
YAML_FILES = [f for f in FILES if re.search(r"\.(ya?ml)$", f)]
|
||||
|
||||
|
||||
def docs_of(rel: str):
|
||||
"""Parsed YAML documents of a tracked file, or [] when it is not parseable YAML."""
|
||||
try:
|
||||
return [d for d in yaml.safe_load_all((ROOT / rel).read_text(errors="replace"))
|
||||
if isinstance(d, dict)]
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
|
||||
DOCS: list[tuple[str, dict]] = [(rel, d) for rel in YAML_FILES for d in docs_of(rel)]
|
||||
|
||||
|
||||
def of_kind(kind: str, group_prefix: str = "") -> list[tuple[str, dict]]:
|
||||
out = []
|
||||
for rel, d in DOCS:
|
||||
if d.get("kind") != kind:
|
||||
continue
|
||||
if group_prefix and not str(d.get("apiVersion", "")).startswith(group_prefix):
|
||||
continue
|
||||
out.append((rel, d))
|
||||
return out
|
||||
|
||||
|
||||
def ns_of(d: dict, default: str = "default") -> str:
|
||||
return (d.get("metadata") or {}).get("namespace") or default
|
||||
|
||||
|
||||
def name_of(d: dict) -> str:
|
||||
return (d.get("metadata") or {}).get("name") or "<unnamed>"
|
||||
|
||||
|
||||
# ------------------------------------------------------------ cluster-vars resolution
|
||||
# Every environment's value for a ${TOKEN}: the base default plus each overlay's JSON6902
|
||||
# `- op: replace / path: /data/KEY / value: ...` patch on the cluster-vars ConfigMap. This
|
||||
# is the same mechanism flux-postbuild-substitution audits; here we need the VALUES, so an
|
||||
# issuer name can be resolved per environment instead of being asserted for dev only.
|
||||
def env_values(key: str) -> dict[str, str]:
|
||||
vals: dict[str, str] = {}
|
||||
base = ROOT / "clusters" / "base" / "cluster-vars.yaml"
|
||||
if base.is_file():
|
||||
for d in yaml.safe_load_all(base.read_text()):
|
||||
if isinstance(d, dict) and d.get("kind") == "ConfigMap" \
|
||||
and name_of(d) == "cluster-vars":
|
||||
v = (d.get("data") or {}).get(key)
|
||||
if v is not None:
|
||||
vals["base"] = str(v)
|
||||
for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
|
||||
env = overlay.parent.name
|
||||
if env == "base":
|
||||
continue
|
||||
try:
|
||||
k = yaml.safe_load(overlay.read_text()) or {}
|
||||
except Exception:
|
||||
continue
|
||||
for p in (k.get("patches") or []):
|
||||
tgt = p.get("target") or {}
|
||||
if tgt.get("kind") != "ConfigMap" or tgt.get("name") != "cluster-vars":
|
||||
continue
|
||||
try:
|
||||
ops = yaml.safe_load(p.get("patch") or "") or []
|
||||
except Exception:
|
||||
continue
|
||||
for op in ops if isinstance(ops, list) else []:
|
||||
if isinstance(op, dict) and str(op.get("path", "")) == f"/data/{key}":
|
||||
vals[env] = str(op.get("value"))
|
||||
return vals
|
||||
|
||||
|
||||
TOKEN_RE = re.compile(r"^\$\{([A-Z0-9_]+)\}$")
|
||||
|
||||
|
||||
def resolve_per_env(value: str) -> dict[str, str]:
|
||||
"""A literal resolves to itself in every environment; a ${TOKEN} to each env's value."""
|
||||
m = TOKEN_RE.match(value.strip())
|
||||
if not m:
|
||||
return {"(literal)": value.strip()}
|
||||
return env_values(m.group(1))
|
||||
|
||||
|
||||
# --------------------------------------------- what Flux actually renders (reachability)
|
||||
# A manifest is DELIVERED only if it is reachable from a Flux Kustomization's `path` through
|
||||
# the kustomize resource graph: <path>/kustomization.yaml -> resources -> (file | directory
|
||||
# -> that directory's kustomization.yaml -> ...). Membership is deliberately computed by
|
||||
# walking that graph rather than by asking "is this file's DIRECTORY mentioned somewhere":
|
||||
# the loose form silently passes when an entry is deleted from the very kustomization.yaml
|
||||
# that the directory reference renders (verified — it made two assertions here no-ops).
|
||||
KUST_RESOURCES: dict[str, list[str]] = {}
|
||||
for rel in [f for f in YAML_FILES if f.endswith("kustomization.yaml")]:
|
||||
refs: list[str] = []
|
||||
for d in docs_of(rel):
|
||||
for r in (d.get("resources") or []) + (d.get("components") or []):
|
||||
if not isinstance(r, str) or "://" in r: # skip remote bases
|
||||
continue
|
||||
try:
|
||||
p = (ROOT / rel).parent.joinpath(r).resolve()
|
||||
refs.append(str(p.relative_to(ROOT)))
|
||||
except Exception:
|
||||
continue
|
||||
KUST_RESOURCES[rel] = refs
|
||||
|
||||
FLUX_PATHS: set[str] = set()
|
||||
for _rel, _d in DOCS:
|
||||
if _d.get("kind") == "Kustomization" and \
|
||||
str(_d.get("apiVersion", "")).startswith("kustomize.toolkit.fluxcd.io/"):
|
||||
p = str((_d.get("spec") or {}).get("path", "")).lstrip("./").rstrip("/")
|
||||
if p:
|
||||
FLUX_PATHS.add(p)
|
||||
|
||||
|
||||
def _walk(entry: str, seen: set[str], out: set[str]) -> None:
|
||||
if entry in seen:
|
||||
return
|
||||
seen.add(entry)
|
||||
kust = entry if entry.endswith("kustomization.yaml") else f"{entry}/kustomization.yaml"
|
||||
if kust in KUST_RESOURCES: # a directory (or a kustomization)
|
||||
for r in KUST_RESOURCES[kust]:
|
||||
_walk(r, seen, out)
|
||||
return
|
||||
out.add(entry) # a leaf manifest
|
||||
|
||||
|
||||
RENDERED: set[str] = set()
|
||||
for _p in FLUX_PATHS:
|
||||
_walk(_p, set(), RENDERED)
|
||||
|
||||
|
||||
def delivered(rel: str) -> bool:
|
||||
"""True iff Flux renders this file from one of its Kustomization paths."""
|
||||
return rel in RENDERED
|
||||
|
||||
|
||||
# --------------------------------------------------- (1) there is something to check
|
||||
CERTS = of_kind("Certificate", "cert-manager.io/")
|
||||
check(len(CERTS) >= 1,
|
||||
"no cert-manager.io Certificate is authored anywhere in the tracked tree — every "
|
||||
"assertion below would be vacuous, so this is a FAILURE, not a pass (R-TLS-02)")
|
||||
|
||||
ISSUERS = of_kind("ClusterIssuer", "cert-manager.io/") + of_kind("Issuer", "cert-manager.io/")
|
||||
check(len(ISSUERS) >= 1,
|
||||
"no cert-manager ClusterIssuer/Issuer is authored — nothing could issue automatically")
|
||||
|
||||
# ---------------------------- (2) every terminating listener serves a controller cert
|
||||
CERT_SECRETS: dict[tuple[str, str], str] = { # (namespace, secretName) -> file
|
||||
(ns_of(d, "default"), str((d.get("spec") or {}).get("secretName", ""))): rel
|
||||
for rel, d in CERTS
|
||||
}
|
||||
listeners_checked = 0
|
||||
for rel, gw in of_kind("Gateway", "gateway.networking.k8s.io/"):
|
||||
gw_ns = ns_of(gw)
|
||||
for lis in ((gw.get("spec") or {}).get("listeners") or []):
|
||||
tls = lis.get("tls") or {}
|
||||
if str(tls.get("mode", "")) != "Terminate":
|
||||
continue
|
||||
listeners_checked += 1
|
||||
refs = tls.get("certificateRefs") or []
|
||||
check(len(refs) > 0,
|
||||
f"{rel}: Gateway {gw_ns}/{name_of(gw)} listener {lis.get('name')!r} terminates "
|
||||
"TLS with NO certificateRefs — it can only serve a certificate someone puts "
|
||||
"there by hand")
|
||||
for r in refs:
|
||||
kind = str(r.get("kind", "Secret"))
|
||||
key = (str(r.get("namespace") or gw_ns), str(r.get("name")))
|
||||
check(kind == "Secret",
|
||||
f"{rel}: listener {lis.get('name')!r} certificateRef kind is {kind!r}; only "
|
||||
"Secret refs can be traced to a cert-manager Certificate")
|
||||
check(key in CERT_SECRETS,
|
||||
f"{rel}: listener {lis.get('name')!r} serves Secret {key[0]}/{key[1]}, which "
|
||||
"NO tracked cert-manager Certificate declares as its secretName — that "
|
||||
"credential would have to be created by hand (R-TLS-02 'without manual steps')")
|
||||
check(listeners_checked > 0,
|
||||
"no Gateway listener with tls.mode=Terminate found in the tracked tree — there is no "
|
||||
"serving-TLS reference to trace, so (2) proved nothing")
|
||||
|
||||
# --------------------------------- (3) no certificate material is tracked, anywhere
|
||||
TLS_KEYS = {"tls.crt", "tls.key", "ca.crt", "tls-combined.pem"}
|
||||
for rel, d in DOCS:
|
||||
if d.get("kind") != "Secret":
|
||||
continue
|
||||
stype = str(d.get("type", ""))
|
||||
check(stype != "kubernetes.io/tls",
|
||||
f"{rel}: a Secret of type kubernetes.io/tls is COMMITTED (metadata.name="
|
||||
f"{name_of(d)}) — a serving certificate authored by a human, not issued")
|
||||
for field in ("data", "stringData"):
|
||||
for k in (d.get(field) or {}):
|
||||
check(str(k) not in TLS_KEYS,
|
||||
f"{rel}: committed Secret {name_of(d)} carries {field}.{k} — certificate "
|
||||
"material in git means a manual issuance step exists (R-TLS-02)")
|
||||
|
||||
PEM_RE = re.compile(r"-----BEGIN (?:[A-Z ]+ )?CERTIFICATE-----")
|
||||
for rel in YAML_FILES:
|
||||
body = (ROOT / rel).read_text(errors="replace")
|
||||
# Scoped to YAML manifests deliberately: prose/docs may legitimately SHOW a PEM header
|
||||
# in an example, and a .md file is not applied to any cluster. A manifest cannot.
|
||||
check(not PEM_RE.search(body),
|
||||
f"{rel}: contains a PEM CERTIFICATE body — a hand-minted certificate in a manifest")
|
||||
|
||||
# --------------------------- (4)+(5)+(6) the issuer chain is automated in EVERY env
|
||||
# The cert-manager release's namespace IS the controller's `--cluster-resource-namespace`
|
||||
# (chart default), i.e. where a ClusterIssuer's solver credential must live. Derived from the
|
||||
# HelmRelease, never written down here, so this holds if the release namespace ever changes.
|
||||
cm_releases = [(rel, d) for rel, d in of_kind("HelmRelease") if name_of(d) == "cert-manager"]
|
||||
CM_NS = ns_of(cm_releases[0][1], "cert-manager") if cm_releases else "cert-manager"
|
||||
|
||||
ISSUER_INDEX = {(d.get("kind"), name_of(d)): (rel, d) for rel, d in ISSUERS}
|
||||
solver_creds: list[tuple[str, str, str, str]] = [] # (issuer, ns, secretName, ref field)
|
||||
issuers_verified: set[str] = set()
|
||||
for rel, cert in CERTS:
|
||||
spec = cert.get("spec") or {}
|
||||
ref = spec.get("issuerRef") or {}
|
||||
kind = str(ref.get("kind", "Issuer"))
|
||||
per_env = resolve_per_env(str(ref.get("name", "")))
|
||||
check(len(per_env) > 0,
|
||||
f"{rel}: Certificate {name_of(cert)} issuerRef.name is a token with NO value in "
|
||||
"clusters/base/cluster-vars.yaml or any overlay — it would render empty and never issue")
|
||||
for env, iname in sorted(per_env.items()):
|
||||
hit = ISSUER_INDEX.get((kind, iname))
|
||||
check(hit is not None,
|
||||
f"{rel}: Certificate {name_of(cert)} resolves issuerRef to {kind}/{iname} in "
|
||||
f"environment {env!r}, but no such issuer is tracked — that environment cannot "
|
||||
"issue anything without a hand-created issuer (R-TLS-02, R-ENV-02)")
|
||||
if hit is None:
|
||||
continue
|
||||
irel, issuer = hit
|
||||
issuers_verified.add(iname)
|
||||
ispec = issuer.get("spec") or {}
|
||||
acme = ispec.get("acme")
|
||||
check(isinstance(acme, dict),
|
||||
f"{irel}: {kind}/{iname} (used by {name_of(cert)} in env {env!r}) is not an ACME "
|
||||
f"issuer — its issuance/renewal path is {sorted(ispec)}; docs/08/ADR-0006 mandate "
|
||||
"ACME so both are unattended")
|
||||
if not isinstance(acme, dict):
|
||||
continue
|
||||
check(str(acme.get("server", "")).startswith("https://"),
|
||||
f"{irel}: {kind}/{iname} has no https ACME directory endpoint")
|
||||
check(bool((acme.get("privateKeySecretRef") or {}).get("name")),
|
||||
f"{irel}: {kind}/{iname} declares no privateKeySecretRef — the ACME account key "
|
||||
"would have to be provisioned by hand")
|
||||
solvers = acme.get("solvers") or []
|
||||
check(len(solvers) > 0,
|
||||
f"{irel}: {kind}/{iname} declares NO solvers — every order would sit unsolved "
|
||||
"until a human completed the challenge")
|
||||
for s in solvers:
|
||||
types = [t for t in ("dns01", "http01") if t in s]
|
||||
check(len(types) == 1,
|
||||
f"{irel}: {kind}/{iname} has a solver with challenge types {sorted(s)} — only "
|
||||
"the automated dns01/http01 solvers satisfy 'without manual steps'")
|
||||
for provider, cfg in (s.get("dns01") or {}).items():
|
||||
if not isinstance(cfg, dict):
|
||||
continue
|
||||
for field, val in cfg.items():
|
||||
if isinstance(val, dict) and str(field).endswith(("SecretRef", "secretRef")):
|
||||
n = str(val.get("name", ""))
|
||||
check(bool(n),
|
||||
f"{irel}: {kind}/{iname} dns01 {provider}.{field} has no secret name")
|
||||
if n:
|
||||
# A ClusterIssuer resolves secret refs in the controller's
|
||||
# cluster-resource-namespace; a namespaced Issuer in its own.
|
||||
want_ns = CM_NS if kind == "ClusterIssuer" else ns_of(issuer, CM_NS)
|
||||
solver_creds.append((iname, want_ns, n, f"{provider}.{field}"))
|
||||
|
||||
check(len(issuers_verified) > 0, "no issuer was reachable from any Certificate — (5) proved nothing")
|
||||
|
||||
# (6) the solver credential is delivered, SOPS-encrypted, from git — never per-cert by hand.
|
||||
check(len(solver_creds) > 0,
|
||||
"no dns01 solver credential reference found — docs/08 (OQ-4) requires the scoped "
|
||||
"Cloudflare token to drive DNS-01, so this check would otherwise be vacuous")
|
||||
# Keyed by (namespace, name): a same-named Secret in ANOTHER namespace is NOT this credential.
|
||||
# (Verified: keying by name alone let the external-dns copy of `cloudflare-api-token` satisfy
|
||||
# the cert-manager reference, so deleting cert-manager's own copy passed — a silent no-op.)
|
||||
SECRET_DOCS = {(ns_of(d, CM_NS), name_of(d)): (rel, d)
|
||||
for rel, d in DOCS if d.get("kind") == "Secret"}
|
||||
for iname, sns, sname, field in solver_creds:
|
||||
hit = SECRET_DOCS.get((sns, sname))
|
||||
check(hit is not None,
|
||||
f"{iname}: dns01 {field} references Secret {sname!r}, which no tracked manifest "
|
||||
f"delivers into namespace {sns!r} (the issuer's secret-resolution namespace) — the "
|
||||
"credential would be created by hand in-cluster (R-TLS-02)")
|
||||
if hit is None:
|
||||
continue
|
||||
srel, sdoc = hit
|
||||
check("sops" in sdoc,
|
||||
f"{srel}: Secret {sname} is the DNS-01 credential but is not SOPS-encrypted (R-SEC-01)")
|
||||
check(delivered(srel),
|
||||
f"{srel}: Secret {sname} is not rendered by any Flux Kustomization path — Flux would "
|
||||
"never deliver it, so issuance would wait on a manual kubectl apply")
|
||||
|
||||
# ------------------------------- (7) automatic CertificateRequest approval is intact
|
||||
check(len(cm_releases) == 1,
|
||||
f"expected exactly one cert-manager HelmRelease, found {[r for r, _ in cm_releases]} — "
|
||||
"the controller that issues and renews must itself be GitOps-managed")
|
||||
for rel, hr in cm_releases:
|
||||
values = ((hr.get("spec") or {}).get("values") or {})
|
||||
check(values.get("disableAutoApproval") in (None, False),
|
||||
f"{rel}: cert-manager values set disableAutoApproval={values.get('disableAutoApproval')!r} "
|
||||
"— every CertificateRequest would then wait for an approver, which is a manual step")
|
||||
signers = values.get("approveSignerNames")
|
||||
if signers is not None:
|
||||
pats = [str(s) for s in (signers if isinstance(signers, list) else [signers])]
|
||||
for iname in sorted(issuers_verified):
|
||||
ok = any(p in ("*", "*/*") or "clusterissuers.cert-manager.io" in p and
|
||||
(p.endswith("/*") or p.endswith(f"/{iname}")) for p in pats)
|
||||
check(ok,
|
||||
f"{rel}: approveSignerNames={pats} does not cover ClusterIssuer {iname} — "
|
||||
"requests for it would never be auto-approved")
|
||||
# The chart's own CRD install must be on, else the CRs below cannot even be applied.
|
||||
check((values.get("crds") or {}).get("enabled") is True,
|
||||
f"{rel}: cert-manager values do not enable CRD installation — the Certificate CRD "
|
||||
"would have to be installed by hand")
|
||||
|
||||
# ------------------------------- (8) the whole chain is continuously reconciled from git
|
||||
check(len(FLUX_PATHS) > 0 and len(RENDERED) > 20,
|
||||
f"the kustomize resource graph resolved {len(RENDERED)} manifest(s) from {len(FLUX_PATHS)} "
|
||||
"Flux Kustomization path(s) — too few to be a real graph, so (8) would prove nothing")
|
||||
for rel, _ in cm_releases + CERTS + ISSUERS:
|
||||
check(delivered(rel),
|
||||
f"{rel} is not rendered by any Flux Kustomization path — nothing reconciles it, so a "
|
||||
"deleted Certificate/issuer/controller would stay deleted until a human re-applied it")
|
||||
|
||||
# --------------------------------------------- (9) renewal is not defeated arithmetically
|
||||
def to_seconds(v) -> int | None:
|
||||
m = re.fullmatch(r"\s*(\d+)\s*([smh])\s*", str(v))
|
||||
if not m:
|
||||
return None
|
||||
return int(m.group(1)) * {"s": 1, "m": 60, "h": 3600}[m.group(2)]
|
||||
|
||||
|
||||
renew_pairs = 0
|
||||
for rel, cert in CERTS:
|
||||
spec = cert.get("spec") or {}
|
||||
dur, rb = to_seconds(spec.get("duration")), to_seconds(spec.get("renewBefore"))
|
||||
if dur is None or rb is None:
|
||||
continue
|
||||
renew_pairs += 1
|
||||
check(rb < dur,
|
||||
f"{rel}: Certificate {name_of(cert)} sets renewBefore={spec.get('renewBefore')} >= "
|
||||
f"duration={spec.get('duration')} — the renewal instant never arrives")
|
||||
|
||||
# --------------------------------------------------------------------------- verdict
|
||||
# NO-OP GUARD: this repo has been bitten by checks that degrade to a successful no-op.
|
||||
# The floor below is a count of assertions that MUST have run for the verdict to mean
|
||||
# anything; it is not a style rule.
|
||||
check(CHECKS >= 20,
|
||||
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
|
||||
"would be meaningless")
|
||||
|
||||
if FAILURES:
|
||||
print(f"CHECK FAIL (tls-lifecycle-automated): {len(FAILURES)} of {CHECKS} assertions failed",
|
||||
file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
print(f"check OK: tls-lifecycle-automated — {CHECKS} assertions; {len(CERTS)} Certificate(s), "
|
||||
f"{len(ISSUERS)} issuer(s), {len(issuers_verified)} issuer(s) reached from a Certificate "
|
||||
f"across every environment, {listeners_checked} terminating listener(s), "
|
||||
f"{len(solver_creds)} dns01 credential ref(s), {renew_pairs} duration/renewBefore pair(s). "
|
||||
"R-TLS-02 'without manual steps' clause only — issued/renewed/revoked are RUNTIME verbs "
|
||||
"(see l2-behavioral/cert-lifecycle-automated; 'revoked' is NOT proven at any layer).")
|
||||
PY
|
||||
Reference in new issue
Block a user