From fe45d0f5c510dd443e8191eee5b0da42f679157f Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 5 Sep 2026 14:17:03 -0500 Subject: [PATCH] =?UTF-8?q?feat(conformance):=20cover=20R-TLS-02,=20R-ENV-?= =?UTF-8?q?01,=20R-ENV-03=20=E2=80=94=2042/51=20->=2045/51,=20and=20find?= =?UTF-8?q?=20a=20real=20gap?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit DECISIONS-LOG records Phase 7 being opened knowingly while the uncovered set included R-ENV-01/02/03 — the very requirements prod provisioning serves. Two of those three are now covered, plus R-TLS-02. tls-lifecycle-automated R-TLS-02 L0 (in the merge gate) cert-lifecycle-automated R-TLS-02 L2 (authored, NOT executed) dev-nodes-provisioned-as-code R-ENV-01 L0 prod-baremetal-not-dev-surface R-ENV-03 L0 Coverage 42/51 (82.4%) -> 45/51 (88.2%), 6 gaps, 0 safety-critical. Remaining: R-C2-01, R-C2-02, R-NET-04, R-PLAT-01, R-SEC-03, R-STATE-04. Verified through the real runner: --layer l0 7/7 PASS, the three new ones 3/3 PASS, --strict-coverage exit 0 with 0 label/annotation mismatches. *** R-TLS-02 "REVOKED" IS NOT SATISFIED — A REAL GAP, NOT A TEST DEFECT. *** The spec requires certificates be "issued, renewed, AND REVOKED without manual steps", and docs/08 reads revocation as "removing/rotating a certificate is a declarative change". That needs enableCertificateOwnerRef: true; platform/cert-manager/helmrelease.yaml does not set it (chart default false). Verified read-only on BOTH live clusters: gateway/wildcard-home-tls has ZERO ownerReferences, so deleting the Certificate would leave the private key serving indefinitely. The L2 test asserts the requirement AS WRITTEN and will fail until this is decided. It was deliberately NOT softened — weakening a test to match the implementation is how a suite stops meaning anything. NOT FLIPPED UNATTENDED, and the reason is specific rather than caution: the wildcard Certificate lives under ./platform, which runs prune: true. With owner refs ON, any transient prune of that Certificate — exactly what happened to the CDI StorageProfiles hours ago — would delete the SERVING TLS Secret for every host, and reissuance is bounded by Let's Encrypt's duplicate-certificate limit (5/week per identical name set), which could leave a cluster with no trusted cert for a week. Security gap vs a plausible week-long TLS outage is an operator call. NON-VACUITY DEMONSTRATED, which is the part that makes these worth having: 7/7 perturbations fail for R-TLS-02, 8/8 for R-ENV-01, 13/13 for R-ENV-03, and each restored copy passes. Perturbations were applied to a throwaway copy; neither this repo nor the read-only spec repo was modified. TWO CHECKS INITIALLY PASSED WHEN PERTURBED and were rewritten — the exact degrade-to-no-op failure this repo keeps hitting. The first draft asked "is this file's DIRECTORY referenced by a kustomization", which a directory reference always satisfies, and keyed Secrets by name alone, so external-dns's copy of `cloudflare-api-token` satisfied cert-manager's reference. Now a recursive kustomize resource-graph walk from each Flux path, and Secrets keyed by (namespace, name). Both bite. SECURITY NOTE, recorded because this project has already leaked a password once: `ansible-inventory --list` PRINTS DECRYPTED VAULT VALUES — it emitted the prod become-password to stdout during exploration, since the vault passphrase file is mounted into the dev container. Both new checks use only --syntax-check and --list-hosts, and say so in a comment. Confirmed no plaintext value reached disk: vault.yml is still AES256-encrypted and the tree has no assigned value. WHAT IS NOT PROVEN, per requirement: R-TLS-02 "revoked" (above), and issuance + observed renewal await the unexecuted L2. R-ENV-01 cannot prove the running VMs came from this module (needs OpenTofu state) — it proves the four lifecycle stages exist as code and cover every dev node, and that the chain RESOLVES under real ansible-playbook. R-ENV-03 cannot prove the HARDWARE (the K8s API exposes no "am I virtual" fact; mini-PC/bare-metal is operator-attested) — it proves prod is AUTHORED as 3 directly-reached nodes with no IaC creating them, sharing dev's playbooks and roles, with no prod-only manifest, branch or role anywhere. VERIFIED BY ME: coverage delta and the 3/3 L0 pass through the real runner; the zero-ownerReferences finding on both clusters; that the new checks avoid ansible-inventory; that no credential material reached disk. Full CI gate PASS. Refs R-TLS-02, R-ENV-01, R-ENV-03, R-TEST-01..05. Co-Authored-By: Claude Opus 5 --- .../chainsaw-test.yaml | 75 +++ .../dev-nodes-provisioned-as-code/check.sh | 442 +++++++++++++++ .../chainsaw-test.yaml | 77 +++ .../prod-baremetal-not-dev-surface/check.sh | 528 ++++++++++++++++++ .../chainsaw-test.yaml | 69 +++ .../tls-lifecycle-automated/check.sh | 478 ++++++++++++++++ .../chainsaw-test.yaml | 391 +++++++++++++ 7 files changed, 2060 insertions(+) create mode 100644 conformance/tests/l0-static/dev-nodes-provisioned-as-code/chainsaw-test.yaml create mode 100755 conformance/tests/l0-static/dev-nodes-provisioned-as-code/check.sh create mode 100644 conformance/tests/l0-static/prod-baremetal-not-dev-surface/chainsaw-test.yaml create mode 100755 conformance/tests/l0-static/prod-baremetal-not-dev-surface/check.sh create mode 100644 conformance/tests/l0-static/tls-lifecycle-automated/chainsaw-test.yaml create mode 100755 conformance/tests/l0-static/tls-lifecycle-automated/check.sh create mode 100644 conformance/tests/l2-behavioral/cert-lifecycle-automated/chainsaw-test.yaml diff --git a/conformance/tests/l0-static/dev-nodes-provisioned-as-code/chainsaw-test.yaml b/conformance/tests/l0-static/dev-nodes-provisioned-as-code/chainsaw-test.yaml new file mode 100644 index 0000000..68f3af5 --- /dev/null +++ b/conformance/tests/l0-static/dev-nodes-provisioned-as-code/chainsaw-test.yaml @@ -0,0 +1,75 @@ +# ============================================================================= +# L0 STATIC — dev/test nodes are provisioned as code, end to end (R-ENV-01) [EXECUTABLE] +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# Spec: docs/00 R-ENV-01 ("dev/test MUST be provisioned as local KVMs via IaC: +# created, baselined, cluster-joined, and managed as code"); the four stages +# are enumerated in docs/02 §"Node lifecycle stages". +# +# WHY L0: R-ENV-01 is a requirement about HOW the dev nodes came to exist, and a live +# Node object cannot answer it — a node built by OpenTofu and one built by hand in a +# hypervisor UI are indistinguishable through the Kubernetes API. What is provable, with +# no cluster, is that a COMPLETE and EXECUTABLE provisioning path exists for all four +# stages and covers EVERY dev node. For the Ansible half that is proven by DRIVING the +# real toolchain offline (`--syntax-check`, `--list-hosts` — neither opens a connection +# to any node), not by reading files: a chain that does not resolve fails the check. +# +# ENVIRONMENT-AGNOSTIC: no hostname, address, prefix or node count appears in the test. +# The dev group names are read from the playbooks' own default host patterns, the node +# names are generated from the IaC module's prefix/count variables, and the two sets are +# compared against each other — so the assertion survives a rename or a resize. +# +# GIVEN the repository as git tracks it, +# WHEN the provisioning chain is parsed and (where offline execution is possible) driven, +# THEN the IaC module creates the dev nodes on a LOCAL KVM hypervisor (allowlisted +# provider — a cloud provider fails), from a for_each map rather than a count index, +# each with CPU/RAM/OS disk/dedicated Longhorn data disk/NIC/cloud-init seed; the +# node set it creates is EXACTLY the set Ansible manages; the baseline stage applies +# the package floor, kernel modules, sysctls, data-disk mount, swap-off and iscsid +# declaratively; one k3s role founds the cluster with --cluster-init and joins the +# rest with --server while disabling flannel/kube-proxy/traefik/servicelb at +# install; and site.yml's chain (baseline -> k3s -> Cilium -> verify -> Flux) +# resolves and targets every dev node at every stage. +# +# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git, PyYAML, and — +# where present — ansible-playbook in no-connection modes; no cluster, no network); +# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test +# wraps the SAME script. +# +# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them): +# * That the RUNNING dev VMs were created by this module — needs the OpenTofu state or +# hypervisor access, neither available offline. +# * `tofu validate` — needs `tofu init` (provider registry), i.e. network. The module is +# parsed, not evaluated; a network-allowed CI stage would close this. +# * "Never hand-edited on the box" (docs/02 stage 4's second half) — a live drift check. +# * REBUILD/re-converge (docs/02's own proof line) — exercised at L3 by +# tests/l3-fault-injection/node-loss (R-STATE-02). +# * docs/02 prose puts the baseline in cloud-init; the implementation moved it into the +# Ansible node_baseline role. R-ENV-01 requires the node to be "baselined ... as code", +# not that cloud-init be the tool, so the check asserts the STAGE and RECORDS the +# divergence rather than hiding it. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: dev-nodes-provisioned-as-code + labels: + conformance.home.k3s/R-ENV-01: "true" + annotations: + conformance.home.k3s/requirements: "R-ENV-01" + conformance.home.k3s/layer: "L0" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Given the tracked tree, when the dev provisioning chain is parsed and driven + offline, then every dev node is created by local-KVM IaC, baselined, cluster-joined + and handed to Flux by one code path that covers all of them — with no node outside + it and no stage missing. +spec: + steps: + - name: then-every-dev-node-is-created-baselined-joined-and-managed-as-code + description: >- + Run the offline assertion script over the tracked tree (same script ci/l0.sh and + conformance/run.sh --layer l0 execute). + try: + - script: + content: bash ./check.sh diff --git a/conformance/tests/l0-static/dev-nodes-provisioned-as-code/check.sh b/conformance/tests/l0-static/dev-nodes-provisioned-as-code/check.sh new file mode 100755 index 0000000..0f033ef --- /dev/null +++ b/conformance/tests/l0-static/dev-nodes-provisioned-as-code/check.sh @@ -0,0 +1,442 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 static check (offline, no cluster) — dev-nodes-provisioned-as-code +# Serves: R-ENV-01 ("dev/test MUST be provisioned as local KVMs via IaC: created, +# baselined, cluster-joined, and managed as code." — spec +# docs/00-vision-and-scope.md; the four stages are enumerated in +# docs/02-environments-and-provisioning.md §"Node lifecycle stages +# (R-ENV-01 — 'created, baselined, cluster-joined, managed as code')": +# 1. Create — OpenTofu instantiates the VM +# 2. Baseline — OS baseline and prerequisites +# 3. Cluster-join — Ansible installs/joins k3s with Cilium as CNI; disables +# k3s bundled flannel/traefik/servicelb/kube-proxy +# 4. Manage — GitOps-managed by Flux; node config changes are made via IaC +# and re-applied, never hand-edited on the box.) +# +# WHY L0 IS THE RIGHT LAYER, AND WHAT IT MEANS TO TEST THIS BEHAVIOURALLY +# ---------------------------------------------------------------------- +# R-ENV-01 is a requirement about HOW the dev nodes come to exist. The live cluster +# cannot answer it: a Kubernetes node object looks identical whether it was built by +# OpenTofu or by hand in a hypervisor UI (there is no API field for "was I created by +# IaC"). What CAN be proven, with no cluster, is that a COMPLETE, EXECUTABLE +# provisioning path exists for all four stages and covers every dev node — and, for the +# Ansible half, that is proven by RUNNING the real toolchain in a no-connection mode +# (`--syntax-check`, `--list-hosts`) rather than by reading the files, so a chain that +# does not actually resolve fails here. The IaC half is asserted by parsing the module: +# `tofu init/validate` needs the provider registry, and L0 is offline by contract. +# +# GIVEN the repository as git tracks it, +# WHEN the provisioning chain is parsed and (where offline execution is possible) +# actually driven, +# THEN CREATE: the IaC module targets a LOCAL KVM hypervisor (every declared provider +# is a KVM-family hypervisor provider — no cloud provider may appear), +# instantiates its nodes from a for_each map (never a count index, so one +# node can be replaced without disturbing its siblings), and gives each +# node CPU, RAM, an OS disk, a DEDICATED Longhorn data disk, a NIC and a +# cloud-init seed carrying the login user + SSH key; +# AND the node set it creates is EXACTLY the dev node set Ansible manages — +# names derived from the module's own prefix/count, so no node can be +# hand-added to the inventory or hand-created in the hypervisor; +# BASELINE: a baseline stage exists as declarative Ansible (package floor, kernel +# modules persisted AND loaded, sysctls, the Longhorn data disk formatted +# and mounted, swap off, iscsid enabled), driven by variables with +# defaults rather than per-host special cases; +# JOIN: one role installs k3s for both roles of every node — `--cluster-init` +# on the primary, `--server` join on the secondaries — with flannel, +# kube-proxy, traefik and servicelb disabled AT INSTALL, and Cilium +# installed as the CNI afterwards; +# MANAGE: site.yml chains baseline -> k3s -> Cilium -> verify -> Flux bootstrap; +# `ansible-playbook --syntax-check` on that chain SUCCEEDS; and +# `--list-hosts` shows every one of those stages targeting EVERY dev +# node, so no node is outside the code path. +# +# ENVIRONMENT-AGNOSTIC: no hostname, address, prefix or node count is written down here. +# The dev group names come from the playbooks' own default host patterns, the node names +# from the IaC module's prefix/count variables, and the two sets are compared to each +# other. This check therefore keeps working when the estate is renamed or resized. +# +# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them): +# - That the RUNNING dev VMs were in fact created by this module. Nothing offline can: +# it would need the OpenTofu state (a remote R2 backend + credentials) or hypervisor +# access. `tofu plan` against the live estate is the operator's own check. +# - That the module APPLIES cleanly. `tofu validate` needs `tofu init`, which downloads +# the provider from the registry; L0 is offline by contract, so the module is parsed, +# not evaluated. ci/lint.sh does not cover .tf either — a gap worth closing in CI +# with a network-allowed stage, not here. +# - "Never hand-edited on the box" (stage 4's second half) — a drift check against the +# live nodes, which is an operator/Ansible-check-mode exercise, not a repo property. +# - REBUILD/re-converge ("destroy a dev node VM and rebuild it", docs/02's own proof +# line) — that is exercised at L3 by tests/l3-fault-injection/node-loss (R-STATE-02). +# - That the baseline actually ran on a node (cluster/host state), and that the +# hypervisor is on the operator's own LAN rather than remote — "local" is asserted as +# "a KVM hypervisor provider driving a self-hosted endpoint variable", not as a +# network topology fact. +# - DEVIATION FROM docs/02 PROSE, ASSERTED DELIBERATELY: docs/02 describes stage 2 as +# cloud-init applying the OS baseline. The implementation moved the rich baseline into +# the Ansible `node_baseline` role (cloud-init keeps hostname/user/SSH/network), +# because Proxmox provisioning is API-only and a custom cloud-init snippet needs +# SSH/SFTP to the hypervisor. R-ENV-01's own wording requires the node to be +# "baselined ... as code", not that cloud-init be the tool, so this check asserts the +# STAGE, wherever it lives, and records the divergence rather than hiding it. +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +PYTHON="${PYTHON:-python3}" + +cd "${ROOT}" +"${PYTHON}" - "${ROOT}" <<'PY' +import re +import shutil +import subprocess +import sys +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover + print("check FAIL: PyYAML is required (same dependency as catalog/render.py)", + file=sys.stderr) + raise SystemExit(1) + +ROOT = Path(sys.argv[1]) +FAILURES: list[str] = [] +NOTES: list[str] = [] +CHECKS = 0 + + +def check(cond, msg): + global CHECKS + CHECKS += 1 + if not cond: + FAILURES.append(msg) + + +def note(msg): + NOTES.append(msg) + + +# A LOCAL DEPENDENCY the repo deliberately does not carry — the vault passphrase (kept +# outside the repo, R-SEC-01) or the Galaxy collections (gitignored; installed from +# collections/requirements.yml). Their absence is an ENVIRONMENT gap, not a repo defect, +# and is the ONLY tolerated reason the driven sub-assertions may not run. Deliberately +# narrow: every other non-zero exit is a real failure. The parse assertions always run. +ENV_ONLY_FAILURE = re.compile( + r"vault password|vault-id|Attempting to decrypt" + r"|No module named 'ansible_collections|couldn't resolve module/action", re.I) + +TOFU = ROOT / "infra" / "tofu" +ANS = ROOT / "infra" / "ansible" +check(TOFU.is_dir(), "infra/tofu/ is missing — the CREATE stage has no IaC at all (R-ENV-01)") +check(ANS.is_dir(), "infra/ansible/ is missing — the BASELINE/JOIN/MANAGE stages have no code") +if not TOFU.is_dir() or not ANS.is_dir(): + print(f"CHECK FAIL (dev-nodes-provisioned-as-code): {len(FAILURES)} of {CHECKS} failed", + file=sys.stderr) + for f in FAILURES: + print(f" - {f}", file=sys.stderr) + raise SystemExit(1) + +TF = "\n".join((p.read_text(errors="replace") for p in sorted(TOFU.glob("*.tf")))) +check(len(TF) > 500, "infra/tofu/*.tf is empty or a stub — nothing creates the dev nodes") + +# ============================================================ STAGE 1 — CREATE (IaC) +# A LOCAL KVM hypervisor, by allowlist: any provider outside this set fails, so moving +# dev onto a cloud (or anything that is not a KVM hypervisor the operator runs) is caught +# here rather than being noticed later. Each entry is justified, not just listed: +# bpg/proxmox Proxmox VE — a KVM/QEMU hypervisor the operator hosts. +# dmacvicar/libvirt libvirt/KVM directly (the pre-Proxmox substrate). +KVM_PROVIDERS = {"bpg/proxmox", "dmacvicar/libvirt"} +providers = dict(re.findall(r'source\s*=\s*"([^"]+)"', TF) and + [(m.group(1), m.group(2)) for m in + re.finditer(r'(\w[\w-]*)\s*=\s*\{[^}]*?source\s*=\s*"([^"]+)"', TF, re.S)]) +check(len(providers) > 0, + "infra/tofu declares no required_providers — the substrate is unpinned, so 'via IaC' " + "cannot be checked at all") +for alias, source in sorted(providers.items()): + check(source.lower() in KVM_PROVIDERS, + f"infra/tofu declares provider {alias}={source!r}, which is not a local KVM hypervisor " + f"provider (allowed: {sorted(KVM_PROVIDERS)}) — R-ENV-01 requires dev/test to be LOCAL KVMs") + +# The VM resource: for_each-keyed, and carrying the hardware + seed docs/02 enumerates. +vm_blocks = re.findall(r'resource\s+"([\w]*(?:_vm|_domain))"\s+"(\w+)"\s*\{(.*?)\n\}', + TF, re.S) +check(len(vm_blocks) >= 1, + "no VM/domain resource found in infra/tofu — nothing in IaC creates a dev node") +for rtype, rname, body in vm_blocks: + check("for_each" in body, + f'resource "{rtype}" "{rname}" is not for_each-keyed. A count index makes single-node ' + "replace unsafe (a -target'ed destroy cascaded to sibling nodes — DECISIONS-LOG F1), so " + "'created ... as code' would not survive a rebuild") + check(not re.search(r"^\s*count\s*=", body, re.M), + f'resource "{rtype}" "{rname}" uses a positional count index') + for want, why in ( + (r"\bcpu\b", "CPU"), + (r"\bmemory\b", "RAM"), + (r"network_device|network_interface", "a NIC"), + (r"initialization|cloud_init|cloudinit", "a cloud-init seed"), + ): + check(re.search(want, body) is not None, + f'resource "{rtype}" "{rname}" declares no {why} — docs/02 stage 1 requires CPU, RAM, ' + "disks, NICs and a cloud-init seed to come from IaC") + disks = len(re.findall(r"^\s*disk\s*\{", body, re.M)) + check(disks >= 2, + f'resource "{rtype}" "{rname}" declares {disks} disk block(s); docs/02 requires the OS ' + "disk PLUS a dedicated data disk for Longhorn") + check(re.search(r"user_account|ssh_authorized_key|ssh_keys", body) is not None, + f'resource "{rtype}" "{rname}" seeds no login user / SSH key — the node could not be ' + "reached by the BASELINE stage without a hand step") + +# Node identity comes from variables, and those variables define the whole node set. +def tf_default(var: str): + m = re.search(r'variable\s+"' + re.escape(var) + r'"\s*\{(.*?)\n\}', TF, re.S) + if not m: + return None + d = re.search(r'default\s*=\s*(.+)', m.group(1)) + return d.group(1).strip() if d else None + + +prefix_raw = tf_default("node_name_prefix") +count_raw = tf_default("node_count") +ips_raw = tf_default("node_ips") +check(prefix_raw is not None and count_raw is not None, + "infra/tofu has no node_name_prefix/node_count variables — the node set is not derived " + "from code, so the created nodes cannot be compared with the managed ones") +prefix = (prefix_raw or "").strip('"') +try: + node_count = int((count_raw or "0").strip()) +except ValueError: + node_count = 0 +check(node_count >= 1, f"node_count default is {count_raw!r} — no node would be created") +if ips_raw is not None: + n_ips = len(re.findall(r'"[^"]+"', ips_raw)) + check(n_ips >= node_count, + f"node_ips default has {n_ips} address(es) for node_count={node_count} — the module " + "cannot address every node it creates") +check((TOFU / "outputs.tf").is_file() and + re.search(r'output\s+"(ansible_inventory|nodes)"', (TOFU / "outputs.tf").read_text()) is not None, + "infra/tofu/outputs.tf surfaces no node inventory output — the JOIN stage would have to be " + "told the node facts by hand instead of consuming them from IaC") + +# ------------------------------------------------- the dev node set, derived twice over +# Group names are taken from the PLAYBOOKS' own default host patterns, so nothing here +# restates the estate's naming. +PLAYBOOKS = ANS / "playbooks" + + +def default_patterns(playbook: str) -> list[str]: + p = PLAYBOOKS / playbook + if not p.is_file(): + return [] + return re.findall(r"default\('([^']+)'\)", p.read_text()) + + +dev_groups = set(default_patterns("node-baseline.yml") + default_patterns("k3s.yml")) +check(len(dev_groups) > 0, + "no playbook declares a default host pattern — the dev target is not expressed in code") + +INV = ANS / "inventory" / "hosts.ini" +check(INV.is_file(), "infra/ansible/inventory/hosts.ini is missing — no node is managed as code") +ini_groups: dict[str, list[str]] = {} +children: dict[str, list[str]] = {} +cur = None +for line in (INV.read_text().splitlines() if INV.is_file() else []): + s = line.strip() + if not s or s.startswith("#"): + continue + m = re.match(r"^\[([^\]]+)\]$", s) + if m: + cur = m.group(1) + (children if cur.endswith(":children") else ini_groups).setdefault( + cur.split(":")[0] if cur.endswith(":children") else cur, []) + continue + if cur is None or cur.endswith(":vars"): + continue + host = s.split()[0] + if cur.endswith(":children"): + children.setdefault(cur.split(":")[0], []).append(host) + else: + ini_groups.setdefault(cur, []).append(host) + + +def hosts_in(group: str, seen=None) -> set[str]: + seen = seen or set() + if group in seen: + return set() + seen.add(group) + out = set(ini_groups.get(group, [])) + for child in children.get(group, []): + out |= hosts_in(child, seen) + return out + + +dev_hosts: set[str] = set() +for g in sorted(dev_groups): + dev_hosts |= hosts_in(g) +check(len(dev_hosts) > 0, + f"the dev groups {sorted(dev_groups)} resolve to NO hosts in hosts.ini — nothing is managed") +expected = {f"{prefix}{i + 1}" for i in range(node_count)} +check(dev_hosts == expected, + f"the dev nodes Ansible manages ({sorted(dev_hosts)}) are not exactly the nodes the IaC " + f"module creates ({sorted(expected)}) — a node outside IaC is a hand-built node (R-ENV-01)") + +# ============================================================ STAGE 2 — BASELINE (code) +BASE_ROLE = ANS / "roles" / "node_baseline" +check(BASE_ROLE.is_dir(), + "no node_baseline role — the BASELINE stage has no code (docs/02 stage 2)") +base_tasks = "\n".join(p.read_text(errors="replace") + for p in sorted((BASE_ROLE / "tasks").glob("*.yml"))) if BASE_ROLE.is_dir() else "" +base_defaults = "\n".join(p.read_text(errors="replace") + for p in sorted((BASE_ROLE / "defaults").glob("*.yml"))) if BASE_ROLE.is_dir() else "" +# Each item docs/02 names for stage 2, matched on the ANSIBLE MODULE that would do it — +# a declarative, idempotent task, not a shell one-liner. +for pattern, what in ( + (r"ansible\.builtin\.apt:", "a package baseline (apt module)"), + (r"modprobe:", "kernel modules loaded (modprobe module)"), + (r"modules-load|/etc/modules", "kernel modules PERSISTED across reboot"), + (r"sysctl", "sysctl tuning"), + (r"filesystem:|mkfs", "a filesystem on the Longhorn data disk"), + (r"mount:|ansible\.posix\.mount", "the Longhorn data disk MOUNTED"), + (r"swap", "swap disabled"), + (r"iscsid", "iscsid enabled (Longhorn prerequisite)"), +): + check(re.search(pattern, base_tasks) is not None, + f"node_baseline does not declare {what} — docs/02 stage 2 requires the OS baseline and " + "k3s/Longhorn/KubeVirt prerequisites to be applied as code") +check(re.search(r"^node_baseline_packages:", base_defaults, re.M) is not None and + re.search(r"^node_baseline_kernel_modules:", base_defaults, re.M) is not None, + "node_baseline's package/module sets are not variables with defaults — a per-host special " + "case is not 'managed as code' (and could not be scoped per environment, R-ENV-02)") + +# ========================================================== STAGE 3 — CLUSTER-JOIN (k3s) +K3S_ROLE = ANS / "roles" / "k3s_server" +k3s_text = "\n".join(p.read_text(errors="replace") + for p in sorted((K3S_ROLE).rglob("*.yml"))) if K3S_ROLE.is_dir() else "" +check(K3S_ROLE.is_dir() and len(k3s_text) > 200, + "no k3s_server role — nothing joins the nodes into a cluster (docs/02 stage 3)") +check("--cluster-init" in k3s_text, + "the k3s role never passes --cluster-init — the HA embedded-etcd cluster is never founded") +check(re.search(r"--server\s+https", k3s_text) is not None, + "the k3s role never joins secondaries with --server — nodes would not form one cluster") +check(re.search(r"K3S_TOKEN|--token", k3s_text) is not None, + "the k3s role never supplies a join token — the join could only be completed by hand") +# The bundled components docs/02 requires to be disabled AT INSTALL, each checked by the +# flag that actually disables it. +for pattern, what in ( + (r"--flannel-backend=none", "flannel (Cilium is the CNI)"), + (r"--disable-kube-proxy", "kube-proxy (Cilium kube-proxy replacement)"), + (r"traefik", "traefik (Gateway API via Cilium)"), + (r"servicelb", "servicelb"), +): + check(re.search(pattern, k3s_text) is not None, + f"the k3s install does not disable {what} — docs/02 stage 3 requires it disabled at " + "install, or the bundled component fights the platform's own") +check((ANS / "roles" / "cilium").is_dir(), + "no cilium role — stage 3 requires Cilium to be installed as the CNI at bootstrap") +check((ANS / "roles" / "flux_bootstrap").is_dir(), + "no flux_bootstrap role — stage 4's hand-off to GitOps has no code") + +# ============================================== STAGE 4 — MANAGE + the chain is REAL +SITE = PLAYBOOKS / "site.yml" +check(SITE.is_file(), "no playbooks/site.yml — the provisioning chain is not expressed as one path") +site_txt = SITE.read_text() if SITE.is_file() else "" +order = [m for m in re.findall(r"import_playbook:\s*(\S+)", site_txt)] +check(order[:1] == ["node-baseline.yml"], + f"site.yml imports {order} — the BASELINE stage must come first (docs/02 sequencing)") +for want in ("k3s.yml", "cilium.yml", "flux-bootstrap.yml"): + check(want in order, f"site.yml does not import {want} — a lifecycle stage is missing from the chain") +if "k3s.yml" in order and "cilium.yml" in order: + check(order.index("k3s.yml") < order.index("cilium.yml"), + "site.yml installs Cilium before k3s — the CNI cannot precede the cluster") + +# --- DRIVE THE REAL TOOLCHAIN, OFFLINE ------------------------------------------------ +# --syntax-check and --list-hosts open NO connection to any node: they resolve the +# playbook, roles, collections and inventory only. Running them is the difference between +# "the chain is written down" and "the chain resolves". +if shutil.which("ansible-playbook") is None: + note("ansible-playbook is not installed here (e.g. the in-cluster chainsaw image), so the " + "chain was PARSED but not DRIVEN; the merge gate's container has it and does drive it") +else: + # cwd=infra/ansible so the tracked ansible.cfg (roles_path, collections_path, inventory) + # is the one in force — the same resolution a real run gets. + def ansible(*args) -> subprocess.CompletedProcess: + return subprocess.run(["ansible-playbook", *args], cwd=ANS, + capture_output=True, text=True) + + syn = ansible("--syntax-check", "playbooks/site.yml") + syn_out = (syn.stderr or "") + (syn.stdout or "") + lh = None + if syn.returncode != 0 and ENV_ONLY_FAILURE.search(syn_out): + # NARROW, SELF-POLICING SKIP — see ENV_ONLY_FAILURE above. + note("ansible-playbook lacks a LOCAL dependency kept outside the repo by design (the vault " + "passphrase and/or the gitignored Galaxy collections), so --syntax-check/--list-hosts " + "were not driven; every parse assertion above still ran") + else: + check(syn.returncode == 0, + "`ansible-playbook --syntax-check playbooks/site.yml` FAILED — the provisioning chain " + f"does not even load: {syn_out.strip()[:400]}") + lh = ansible("--list-hosts", "playbooks/site.yml") + check(lh.returncode == 0, + "`ansible-playbook --list-hosts playbooks/site.yml` FAILED — the chain does not " + f"resolve against the tracked inventory: " + f"{((lh.stderr or '') + (lh.stdout or '')).strip()[:400]}") + if lh is not None and lh.returncode == 0: + # Parse plays -> hosts. NEVER `ansible-inventory --list`: it prints DECRYPTED vault values. + plays: list[tuple[str, set[str]]] = [] + name, hosts, in_hosts = None, set(), False + for line in lh.stdout.splitlines(): + m = re.match(r"\s*play #\d+ \([^)]*\):\s*(.*?)\s*TAGS:", line) + if m: + if name is not None: + plays.append((name, hosts)) + name, hosts, in_hosts = m.group(1), set(), False + continue + if re.match(r"\s*hosts \(\d+\):", line): + in_hosts = True + continue + if in_hosts and line.strip() and not line.strip().startswith("pattern:"): + hosts.add(line.strip()) + if name is not None: + plays.append((name, hosts)) + check(len(plays) >= 4, + f"--list-hosts resolved {len(plays)} play(s) — the four-stage chain did not expand") + # Every lifecycle stage must cover EVERY dev node. A stage that reaches two of three + # nodes means the third was finished by hand. + STAGES = { + "baseline": r"baseline", + "cluster-join": r"^k3s (primary|secondar)", + "cni": r"cilium", + "manage (flux hand-off)": r"flux", + } + for stage, rx in STAGES.items(): + covered: set[str] = set() + matched = 0 + for pname, phosts in plays: + if re.search(rx, pname, re.I): + matched += 1 + covered |= phosts + check(matched >= 1, f"no play in site.yml implements the {stage} stage") + check(covered == dev_hosts, + f"the {stage} stage targets {sorted(covered)}, not every dev node " + f"{sorted(dev_hosts)} — a node outside the code path is a hand-provisioned " + "node (R-ENV-01)") + +# --------------------------------------------------------------------------- verdict +# NO-OP GUARD: a check that quietly asserts nothing is worse than no check. +check(CHECKS >= 30, + f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass " + "would be meaningless") + +if FAILURES: + print(f"CHECK FAIL (dev-nodes-provisioned-as-code): {len(FAILURES)} of {CHECKS} assertions failed", + file=sys.stderr) + for f in FAILURES: + print(f" - {f}", file=sys.stderr) + raise SystemExit(1) +for n in NOTES: + print(f" note: {n}") +print(f"check OK: dev-nodes-provisioned-as-code — {CHECKS} assertions; " + f"{len(dev_hosts)} dev node(s) {sorted(dev_hosts)} created by IaC ({sorted(providers.values())}), " + f"baselined, cluster-joined and Flux-managed by the same chain. R-ENV-01's authored-path " + "clause only — the live VMs' provenance, `tofu validate`, and rebuild/re-converge are NOT " + "proven here (see the header's gap list; rebuild is L3 node-loss).") +PY diff --git a/conformance/tests/l0-static/prod-baremetal-not-dev-surface/chainsaw-test.yaml b/conformance/tests/l0-static/prod-baremetal-not-dev-surface/chainsaw-test.yaml new file mode 100644 index 0000000..532aa7e --- /dev/null +++ b/conformance/tests/l0-static/prod-baremetal-not-dev-surface/chainsaw-test.yaml @@ -0,0 +1,77 @@ +# ============================================================================= +# L0 STATIC — prod is bare metal, and is not a development surface (R-ENV-03) [EXECUTABLE] +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# Spec: docs/00 R-ENV-03 ("prod is a 3× mini-PC bare-metal k3s cluster with local +# storage; it MUST NOT be an active development surface until the roadmap +# declares it ready"); docs/02 (prod = 3 physical nodes, local SSD/NVMe to +# Longhorn, "the *same* Ansible bootstrap runs; only the machine-creation step +# differs", "Never develop against prod"); roadmap/phases.md Phase 7. +# +# HOW A "MUST NOT" IS MADE TESTABLE — the same discipline as R-TEST-06's test: a +# governance rule is verified through the property that ENFORCES it, never by observing +# that nobody misbehaved today. Here that property is structural: prod receives the base +# configuration plus environment-scoped overrides and nothing else, from the same branch +# and the same Ansible code as dev, so there is no place to put prod-only work at all. +# +# ENVIRONMENT-AGNOSTIC: the prod inventory group is DERIVED (it is the group whose +# group_vars retargets the k3s roles away from the global default), the prod overlay is +# derived from CLUSTER_ENV, and the storage/replica expectations are compared against the +# node count that inventory resolves to. No hostname, address or environment name is +# written into the test. +# +# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted), +# WHEN the prod inventory, overlay, IaC module and readiness record are inspected — and +# the shared provisioning chain is actually retargeted at prod with --list-hosts, +# THEN prod resolves to exactly three nodes reached directly on their own sshd; no IaC +# resource creates any of them (bare metal is imaged, not instantiated); prod's +# Longhorn path is node-local with one replica per node and the default +# StorageClass is the local driver; prod runs the same playbooks/roles as dev, +# retargeted only by variables the shared code already defines; the prod overlay +# carries only the base plus environment-scoped patches on the same Git ref dev +# tracks, with no prod-only manifest anywhere in the shared configuration; and prod +# artifacts exist only alongside the operator's recorded opening of the prod phase. +# +# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git, PyYAML, and — +# where present — ansible-playbook in --list-hosts mode, which opens no connection); +# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test +# wraps the SAME script. +# +# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them): +# * THE HARDWARE. That the three machines are mini-PCs, are bare metal, and hold local +# SSD/NVMe is operator-attested (infra/PROD-STANDUP.md, PROD-PROVISIONING.md); the +# node-health/BIOS surveys that establish it write to gitignored artifacts, and the +# Kubernetes API exposes no "am I virtual" fact, so nothing here asserts it. +# * The LIVE prod cluster's shape (three Ready nodes, all control-plane+etcd, Longhorn +# disks at the local path) — cluster state, belonging to R-PLAT-01's still-uncovered +# HA test; asserting it here would also make this test environment-specific, since +# the same suite must pass against dev. +# * Git HISTORY: this is the working tree at HEAD. +# * Whether the roadmap SHOULD have opened prod — only that prod work is accompanied by +# the recorded declaration. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: prod-baremetal-not-dev-surface + labels: + conformance.home.k3s/R-ENV-03: "true" + annotations: + conformance.home.k3s/requirements: "R-ENV-03" + conformance.home.k3s/layer: "L0" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Given the tracked tree, when prod's inventory, overlay, IaC and readiness record are + inspected, then prod is authored as three bare-metal nodes with node-local storage, + bootstrapped by dev's own code retargeted by variables, receiving only + environment-scoped overrides on the same branch — so there is nowhere to develop + against prod — and it exists only because the roadmap phase was opened. +spec: + steps: + - name: then-prod-is-bare-metal-and-carries-no-development-of-its-own + description: >- + Run the offline assertion script over the tracked tree (same script ci/l0.sh and + conformance/run.sh --layer l0 execute). + try: + - script: + content: bash ./check.sh diff --git a/conformance/tests/l0-static/prod-baremetal-not-dev-surface/check.sh b/conformance/tests/l0-static/prod-baremetal-not-dev-surface/check.sh new file mode 100755 index 0000000..1fd041a --- /dev/null +++ b/conformance/tests/l0-static/prod-baremetal-not-dev-surface/check.sh @@ -0,0 +1,528 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 static check (offline, no cluster) — prod-baremetal-not-dev-surface +# Serves: R-ENV-03 ("prod is a 3× mini-PC bare-metal k3s cluster with local storage; it +# MUST NOT be an active development surface until the roadmap declares it ready." +# — spec docs/00-vision-and-scope.md. Detail in docs/02: prod substrate "3× mini-PC +# bare metal", "3 physical nodes", storage "local SSD/NVMe presented to Longhorn", +# "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation +# step differs", and "**Never develop against prod.** Prod hardware is a target, not +# a sandbox (R-ENV-03)". The roadmap gate is roadmap/phases.md Phase 7: entry +# "operator declares prod hardware ready"; note "Do not begin building on prod +# hardware before this phase is explicitly opened".) +# +# HOW A "MUST NOT" IS MADE TESTABLE +# -------------------------------- +# The second clause is a governance rule, and the same discipline applies as to R-TEST-06 +# (tests/l0-static/spec-immutability-enforced): "nobody developed against prod today" +# cannot be observed, but the PROPERTY THAT MAKES DEVELOPING AGAINST PROD IMPOSSIBLE can +# be. In this repo that property is structural — prod receives the base configuration plus +# environment-scoped overrides and NOTHING ELSE, so there is no place to put prod-only work +# even if someone wanted to. This check asserts that, together with the first clause's +# authored facts (three nodes, no VM IaC behind them, node-local storage, the same +# bootstrap code as dev). +# +# The first clause's HARDWARE facts (that the three machines really are mini-PCs, really +# are bare metal, really hold local NVMe) are operator-attested, not machine-checkable — +# see the gap list. What IS checkable is that the repository describes prod that way and +# that no IaC creates prod nodes as virtual machines. +# +# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted), +# WHEN the prod inventory, the prod overlay, the IaC module and the roadmap record are +# inspected — and, where offline execution is possible, the provisioning chain is +# actually retargeted at prod with `--list-hosts`, +# THEN (1) prod resolves to EXACTLY THREE nodes, reached directly on their own sshd +# (no NAT indirection, i.e. real machines on the LAN); +# (2) NO IaC resource creates any prod node: the VM module's generated node names are +# disjoint from the prod host set and no prod hostname appears anywhere under +# infra/tofu — prod is imaged by the operator (docs/02 stage 1), not instantiated; +# (3) prod's storage is NODE-LOCAL: the Longhorn data path for prod is an absolute +# local path, and the replica count equals the node count (one replica per node's +# own disk). A distinct bulk/media tier is explicitly permitted (R-STATE-04) and +# is not counted against this; +# (4) prod runs THE SAME bootstrap code as dev: the same playbooks/roles, retargeted +# by variables only — every prod group_vars key is an existing role/global +# variable (an invented prod-only knob would be a fork), and no prod-only role +# or playbook exists; +# (5) prod is NOT a development surface: clusters/prod carries only the base plus +# sync.yaml, every patch it applies targets an ENVIRONMENT-SCOPED object +# (cluster-vars, a Flux Kustomization's sops decryption, or the Git source), +# prod's Git ref is the SAME ref dev tracks (no prod-only branch to develop on), +# and no prod-only manifest exists anywhere in the rendered configuration; +# (6) prod artifacts exist only WITH a declaration of readiness: the spec roadmap +# carries the phase that opens prod build (checked when the spec repo is mounted), +# and the repository records that the operator opened it. +# +# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them): +# - The HARDWARE. That the three nodes are mini-PCs, are bare metal, and present local +# SSD/NVMe is operator-attested in infra/PROD-STANDUP.md / PROD-PROVISIONING.md; the +# node-health and BIOS surveys that establish it write to gitignored artifacts, so +# nothing tracked can be asserted against. Cluster-side, the Kubernetes API exposes no +# "am I virtual" fact either — a bare-metal check would need a privileged host-path +# probe of DMI data, which is a heavier and more invasive test than the claim warrants. +# - The LIVE prod cluster's shape (three Ready nodes, all control-plane+etcd, Longhorn +# disks at the local path). That is cluster state and belongs to R-PLAT-01's +# (currently uncovered) HA test; asserting it here would also make this test +# environment-specific, since the same suite must pass against dev. +# - That prod's HISTORY contains no development. This is the working tree at HEAD. +# - Whether the roadmap SHOULD have opened prod (the entry-gate judgement). Only that +# prod work is accompanied by the operator's recorded declaration. +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +PYTHON="${PYTHON:-python3}" +: "${SPEC_REPO:=${ROOT}/../home-network-k3s}" +export SPEC_REPO + +cd "${ROOT}" +"${PYTHON}" - "${ROOT}" <<'PY' +import os +import re +import shutil +import subprocess +import sys +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover + print("check FAIL: PyYAML is required (same dependency as catalog/render.py)", + file=sys.stderr) + raise SystemExit(1) + +ROOT = Path(sys.argv[1]) +FAILURES: list[str] = [] +NOTES: list[str] = [] +CHECKS = 0 + + +def check(cond, msg): + global CHECKS + CHECKS += 1 + if not cond: + FAILURES.append(msg) + + +def note(msg): + NOTES.append(msg) + + +# A LOCAL DEPENDENCY the repo deliberately does not carry — the vault passphrase (kept +# outside the repo, R-SEC-01) or the Galaxy collections (gitignored; installed from +# collections/requirements.yml). Their absence is an ENVIRONMENT gap, not a repo defect, +# and is the ONLY tolerated reason the driven sub-assertions may not run. Deliberately +# narrow: every other non-zero exit is a real failure. The parse assertions always run. +ENV_ONLY_FAILURE = re.compile( + r"vault password|vault-id|Attempting to decrypt" + r"|No module named 'ansible_collections|couldn't resolve module/action", re.I) + +ANS = ROOT / "infra" / "ansible" +INV = ANS / "inventory" / "hosts.ini" +GV = ANS / "inventory" / "group_vars" +TOFU = ROOT / "infra" / "tofu" +check(INV.is_file(), "infra/ansible/inventory/hosts.ini is missing — prod is not described anywhere") +if not INV.is_file(): + print("CHECK FAIL (prod-baremetal-not-dev-surface): no inventory", file=sys.stderr) + raise SystemExit(1) + +# --------------------------------------------------------------- inventory parsing +ini_groups: dict[str, list[dict]] = {} +children: dict[str, list[str]] = {} +cur = None +for line in INV.read_text().splitlines(): + s = line.strip() + if not s or s.startswith("#"): + continue + m = re.match(r"^\[([^\]]+)\]$", s) + if m: + cur = m.group(1) + continue + if cur is None or cur.endswith(":vars"): + continue + if cur.endswith(":children"): + children.setdefault(cur.split(":")[0], []).append(s.split()[0]) + continue + parts = s.split() + host = {"name": parts[0]} + for kv in parts[1:]: + if "=" in kv: + k, v = kv.split("=", 1) + host[k] = v + ini_groups.setdefault(cur, []).append(host) + + +def hosts_in(group: str, seen=None) -> list[dict]: + seen = seen or set() + if group in seen: + return [] + seen.add(group) + out = list(ini_groups.get(group, [])) + for child in children.get(group, []): + out += hosts_in(child, seen) + return out + + +def load_gv(group: str) -> dict: + d: dict = {} + for p in [GV / f"{group}.yml", GV / group / "vars.yml"]: + if p.is_file(): + try: + d.update(yaml.safe_load(p.read_text()) or {}) + except Exception: + pass + return d + + +# WHICH GROUP IS PROD? Derived, never named here: it is the group whose group_vars +# RETARGETS the k3s roles (k3s_primary_group / k3s_servers_group) away from the global +# dev default. That is exactly the seam docs/02 + infra/ansible/README describe for prod. +ALL_VARS = load_gv("all") +dev_primary_group = str(ALL_VARS.get("k3s_primary_group", "")) +candidates = [] +for entry in sorted(GV.iterdir()) if GV.is_dir() else []: + g = entry.name[:-4] if entry.name.endswith(".yml") else entry.name + if g == "all": + continue + v = load_gv(g) + if v.get("k3s_primary_group") and str(v["k3s_primary_group"]) != dev_primary_group: + candidates.append((g, v)) +check(len(candidates) == 1, + f"expected exactly ONE environment whose group_vars retargets the k3s roles (prod); found " + f"{[c[0] for c in candidates]} — R-ENV-03 describes a single prod estate reached through " + "the same roles as dev") +if not candidates: + print("CHECK FAIL (prod-baremetal-not-dev-surface): no prod group_vars found", file=sys.stderr) + for f in FAILURES: + print(f" - {f}", file=sys.stderr) + raise SystemExit(1) +PROD_GROUP, PROD_VARS = candidates[0] +PROD_PRIMARY = str(PROD_VARS["k3s_primary_group"]) +prod_hosts = hosts_in(PROD_GROUP) +prod_names = sorted(h["name"] for h in prod_hosts) + +# ------------------------------------------------- (1) exactly three, directly reachable +check(len(prod_names) == 3, + f"prod group {PROD_GROUP!r} resolves to {len(prod_names)} host(s) {prod_names} — R-ENV-03 " + "says prod is a 3× mini-PC cluster") +dev_hosts = hosts_in(str(ALL_VARS.get("k3s_servers_group", "k3s_servers"))) +for h in prod_hosts: + # Real machines on the LAN answer their own sshd. Dev's hosts are reached through a + # NAT port-forward (ansible_port 225x -> a VM's :22), which is precisely the + # indirection a hypervisor-hosted estate needs and bare metal does not. + check(str(h.get("ansible_port", "22")) == "22", + f"prod node {h['name']} is reached on port {h.get('ansible_port')} — a forwarded port " + "implies an indirection (NAT to a virtual estate), not a machine on the LAN") + check(h.get("k3s_node_ip") == h.get("ansible_host"), + f"prod node {h['name']} has ansible_host={h.get('ansible_host')} but " + f"k3s_node_ip={h.get('k3s_node_ip')} — a split address is the dev NAT pattern; a " + "bare-metal node's SSH address IS its cluster address") + +# ------------------------------------------------------ (2) nothing creates prod as a VM +TF = "\n".join(p.read_text(errors="replace") for p in sorted(TOFU.glob("*.tf"))) \ + if TOFU.is_dir() else "" +check(len(TF) > 500, "infra/tofu/*.tf is empty — cannot show that it does NOT create prod") + + +def tf_default(var: str): + m = re.search(r'variable\s+"' + re.escape(var) + r'"\s*\{(.*?)\n\}', TF, re.S) + if not m: + return None + d = re.search(r"default\s*=\s*(.+)", m.group(1)) + return d.group(1).strip() if d else None + + +prefix = (tf_default("node_name_prefix") or "").strip('"') +try: + node_count = int((tf_default("node_count") or "0").strip()) +except ValueError: + node_count = 0 +iac_names = {f"{prefix}{i + 1}" for i in range(node_count)} +check(len(iac_names) > 0, "the IaC module generates no node names — cannot compare with prod") +check(iac_names.isdisjoint(set(prod_names)), + f"the IaC VM module would create {sorted(iac_names & set(prod_names))}, which are PROD nodes " + "— prod is bare metal that the operator images (docs/02 stage 1), never instantiated by the " + "hypervisor module") +for name in prod_names: + check(name not in TF, + f"prod node {name} is named inside infra/tofu — prod hardware must not be described by " + "the VM module at all") +# The prod host addresses must not appear as IaC node addresses either (a renamed VM is +# still a VM). +tf_ips = set(re.findall(r'"(\d+\.\d+\.\d+\.\d+)"', tf_default("node_ips") or "")) +prod_ips = {str(h.get("ansible_host")) for h in prod_hosts} +check(tf_ips.isdisjoint(prod_ips), + f"the IaC module addresses {sorted(tf_ips & prod_ips)}, which are prod node addresses") + +# --------------------------------------------------------------- (3) node-local storage +# Every environment's value for a cluster-vars knob (base default + each overlay patch). +def env_values(key: str) -> dict[str, str]: + vals: dict[str, str] = {} + base = ROOT / "clusters" / "base" / "cluster-vars.yaml" + if base.is_file(): + for d in yaml.safe_load_all(base.read_text()): + if isinstance(d, dict) and d.get("kind") == "ConfigMap" \ + and (d.get("metadata") or {}).get("name") == "cluster-vars": + v = (d.get("data") or {}).get(key) + if v is not None: + vals["base"] = str(v) + for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")): + env = overlay.parent.name + if env == "base": + continue + try: + k = yaml.safe_load(overlay.read_text()) or {} + except Exception: + continue + for p in (k.get("patches") or []): + tgt = p.get("target") or {} + if tgt.get("kind") != "ConfigMap" or tgt.get("name") != "cluster-vars": + continue + try: + ops = yaml.safe_load(p.get("patch") or "") or [] + except Exception: + continue + for op in ops if isinstance(ops, list) else []: + if isinstance(op, dict) and str(op.get("path", "")) == f"/data/{key}": + vals[env] = str(op.get("value")) + return vals + + +# The prod OVERLAY is the one whose CLUSTER_ENV is not the base default. +cluster_envs = env_values("CLUSTER_ENV") +prod_overlays = [e for e, v in cluster_envs.items() + if e != "base" and v != cluster_envs.get("base")] +check(len(prod_overlays) >= 1, + "no clusters/ overlay sets its own CLUSTER_ENV — prod has no overlay of its own") +prod_overlay = None +for e in prod_overlays: + if PROD_GROUP.replace("k3s_", "") in cluster_envs[e] or cluster_envs[e] in PROD_GROUP: + prod_overlay = e +check(prod_overlay is not None, + f"no overlay's CLUSTER_ENV corresponds to the prod inventory group {PROD_GROUP!r} — the " + "cluster config and the machine inventory disagree about which estate is prod") + +data_paths = env_values("LONGHORN_DATA_PATH") +replicas = env_values("LONGHORN_DEFAULT_REPLICA_COUNT") +if prod_overlay: + p = data_paths.get(prod_overlay, data_paths.get("base", "")) + check(p.startswith("/"), + f"prod's LONGHORN_DATA_PATH is {p!r} — R-ENV-03's 'local storage' requires an absolute " + "node-local filesystem path") + check(not re.match(r"^/(net|nfs|mnt/nfs)|:", p), + f"prod's LONGHORN_DATA_PATH {p!r} looks like a network mount, not local SSD/NVMe") + r = replicas.get(prod_overlay, replicas.get("base", "")) + check(r.isdigit() and int(r) == len(prod_names), + f"prod's LONGHORN_DEFAULT_REPLICA_COUNT is {r!r} for {len(prod_names)} node(s) — " + "'local storage' on a 3-node estate means one replica per node's own disk") +# The DEFAULT storage class must be the local (Longhorn) driver. A separate bulk/media tier +# on network storage is explicitly a different requirement (R-STATE-04, docs/04) and is not +# counted here. +default_scs = [] +for f in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z", "*.yaml", "*.yml"], + capture_output=True, text=True, check=True).stdout.split("\0"): + if not f: + continue + try: + for d in yaml.safe_load_all((ROOT / f).read_text(errors="replace")): + if isinstance(d, dict) and d.get("kind") == "StorageClass" and \ + str(((d.get("metadata") or {}).get("annotations") or {}) + .get("storageclass.kubernetes.io/is-default-class", "")) == "true": + default_scs.append((f, str(d.get("provisioner", "")))) + except Exception: + continue +check(len(default_scs) == 1, + f"expected exactly one default StorageClass, found {default_scs} — the tier prod's workloads " + "land on by default must be unambiguous") +for f, prov in default_scs: + check(prov == "driver.longhorn.io", + f"{f}: the default StorageClass provisioner is {prov!r}, not Longhorn — prod's default " + "tier would not be its own local disks") + +# ------------------------------------- (4) prod runs the SAME bootstrap code, retargeted +PLAYBOOKS = ANS / "playbooks" +ROLES = ANS / "roles" +prod_token = PROD_GROUP.replace("k3s_", "") +for d, what in ((ROLES, "role"), (PLAYBOOKS, "playbook")): + for entry in sorted(d.iterdir()) if d.is_dir() else []: + check(prod_token not in entry.name.lower(), + f"infra/ansible/{d.name}/{entry.name} is a prod-specific {what} — docs/02 requires " + "the SAME Ansible bootstrap for prod, differing only in the machine-creation step") +# Every prod group_vars key must be a variable the shared code already knows: a role default, +# a global, or an ansible built-in. An invented key is a fork of the role by another name. +known: set[str] = set(ALL_VARS) +for rd in sorted(ROLES.iterdir()) if ROLES.is_dir() else []: + for p in (rd / "defaults").glob("*.yml") if (rd / "defaults").is_dir() else []: + try: + known |= set((yaml.safe_load(p.read_text()) or {}).keys()) + except Exception: + pass +check(len(known) > 20, f"only {len(known)} shared variables found — (4) would prove nothing") +for k in sorted(PROD_VARS): + check(k.startswith("ansible_") or k in known, + f"prod group_vars sets {k!r}, which is not a role default or a global — prod-only " + "configuration that the dev path never exercises is developing against prod") +# ...and it must retarget, not re-implement: the group it names must exist in the inventory. +check(PROD_PRIMARY in ini_groups or PROD_PRIMARY in children, + f"prod group_vars names primary group {PROD_PRIMARY!r}, which the inventory does not define") + +# ---------------------------------------- (5) prod is not a development surface (overlay) +if prod_overlay: + odir = ROOT / "clusters" / prod_overlay + files = sorted(p.name for p in odir.iterdir() if p.is_file()) + check(files == ["kustomization.yaml", "sync.yaml"], + f"clusters/{prod_overlay}/ contains {files} — a prod overlay may hold only its " + "kustomization and its own root sync; anything else is configuration authored FOR PROD") + k = yaml.safe_load((odir / "kustomization.yaml").read_text()) or {} + check(sorted(str(r) for r in (k.get("resources") or [])) == ["../base", "sync.yaml"], + f"clusters/{prod_overlay} resources are {k.get('resources')} — prod must render the base " + "plus its own sync only (docs/02: 'apply the prod overlay (no base edits)')") + # Every patch must be an ENVIRONMENT-SCOPED override, never a change to what runs. + ALLOWED = {("ConfigMap", "cluster-vars"), ("GitRepository", None)} + for p in (k.get("patches") or []): + tgt = p.get("target") or {} + body = p.get("patch") or "" + kind = tgt.get("kind") + if kind is None: # inline patch: kind comes from the body + try: + doc = yaml.safe_load(body) or {} + except Exception: + doc = {} + kind = doc.get("kind") + name = (doc.get("metadata") or {}).get("name") + if kind == "Kustomization": + # Flux Kustomization patches are allowed ONLY to wire sops decryption — + # anything else (path, interval, images, values) changes what prod RUNS. + keys = set((doc.get("spec") or {}).keys()) + check(keys <= {"decryption"}, + f"clusters/{prod_overlay}: patch of Kustomization/{name} touches {sorted(keys)} " + "— only `decryption` (the per-cluster sops key) is an environment-scoped " + "override; changing anything else is developing against prod") + continue + check((kind, None) in ALLOWED or (kind, name) in ALLOWED, + f"clusters/{prod_overlay}: patch targets {kind}/{name}, which is not an " + "environment-scoped object — prod must not carry configuration of its own") + else: + check((kind, tgt.get("name")) in ALLOWED or (kind, None) in ALLOWED, + f"clusters/{prod_overlay}: patch targets {kind}/{tgt.get('name')}, which is not " + "an environment-scoped object — prod must not carry configuration of its own") + + # Same Git ref as the other environment(s): a prod-only branch is a development line + # that only prod runs, which is exactly what R-ENV-03 forbids. + def git_ref(overlay: str): + kk = yaml.safe_load((ROOT / "clusters" / overlay / "kustomization.yaml").read_text()) or {} + for p in (kk.get("patches") or []): + body = p.get("patch") or "" + try: + doc = yaml.safe_load(body) or {} + except Exception: + continue + if doc.get("kind") == "GitRepository": + return (doc.get("spec") or {}).get("ref") + return None + + others = [o.name for o in (ROOT / "clusters").iterdir() + if o.is_dir() and o.name not in ("base", prod_overlay)] + prod_ref = git_ref(prod_overlay) + check(prod_ref is not None, + f"clusters/{prod_overlay} does not pin a Git ref — prod would follow whatever the base " + "points at, with no record of what it runs") + for o in others: + check(git_ref(o) == prod_ref, + f"clusters/{o} tracks {git_ref(o)} but prod tracks {prod_ref} — a prod-only ref is a " + "development line only prod runs (R-ENV-03: prod is a target, not a sandbox)") + +# No prod-only manifest anywhere in the rendered configuration. +tracked = [f for f in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"], + capture_output=True, text=True, check=True + ).stdout.split("\0") if f] +RENDERED_DIRS = ("platform/", "platform-config", "apps/", "catalog/", "clusters/base/") +for f in tracked: + if f.startswith(RENDERED_DIRS) and re.search(rf"(^|[/_.-]){re.escape(prod_token)}([/_.-]|$)", + f, re.I): + check(False, + f"{f} is a prod-specific file inside the shared configuration — every environment " + "renders these paths, so prod-only content here is development against prod") +check(len(tracked) > 50, f"git ls-files returned only {len(tracked)} paths — not a real checkout") + +# --------------------------- (6) prod work exists only with a declaration of readiness +# In-repo: the operator's recorded decision to open the prod phase. +log = ROOT / "DECISIONS-LOG.md" +check(log.is_file() and re.search(r"PHASE\s*7\b.*OPEN", log.read_text(errors="replace"), re.I), + "DECISIONS-LOG.md carries no record of the operator OPENING the prod phase, yet prod " + "inventory/overlay exist — R-ENV-03 allows prod build only once the roadmap declares it ready") +# In the spec (read-only, may not be mounted): the roadmap phase that opens prod. +spec = Path(os.environ.get("SPEC_REPO", "")) +phases = spec / "roadmap" / "phases.md" +if phases.is_file(): + txt = phases.read_text(errors="replace") + check(re.search(r"##\s*Phase\s*7\b", txt) is not None and + re.search(r"prod", txt, re.I) is not None, + "the spec roadmap has no Phase 7 opening prod build — the declaration R-ENV-03 defers " + "to does not exist") + check(re.search(r"explicitly opened|declares prod hardware ready|operator declares", txt, re.I) + is not None, + "the spec roadmap's prod phase states no explicit entry declaration — the gate R-ENV-03 " + "names would be unenforceable") +else: + note(f"spec repo not mounted at {spec} — the roadmap half of (6) was NOT checked " + "(the in-repo declaration still was)") + +# --------------------------- (4b) DRIVE the retarget offline: same chain, prod hosts +if shutil.which("ansible-playbook") is None: + note("ansible-playbook is not installed here (e.g. the in-cluster chainsaw image), so the " + "prod retarget was PARSED but not DRIVEN; the merge gate's container has it") +else: + sec_groups = [g for g in children.get(PROD_GROUP, []) if g != PROD_PRIMARY] + args = ["--list-hosts", "playbooks/site.yml", + "-e", f"k3s_hosts={PROD_GROUP}", + "-e", f"k3s_primary_hosts={PROD_PRIMARY}"] + if sec_groups: + args += ["-e", f"k3s_secondary_hosts={sec_groups[0]}"] + lh = subprocess.run(["ansible-playbook", *args], cwd=ANS, capture_output=True, text=True) + out = (lh.stderr or "") + (lh.stdout or "") + if lh.returncode != 0 and ENV_ONLY_FAILURE.search(out): + # See the same narrow, self-policing skip in dev-nodes-provisioned-as-code. + note("ansible-playbook lacks a LOCAL dependency kept outside the repo by design (the vault " + "passphrase and/or the gitignored Galaxy collections), so the prod retarget was not " + "driven; every parse assertion above still ran") + else: + check(lh.returncode == 0, + f"retargeting the SHARED chain at prod failed: {out.strip()[:400]} — docs/02 requires " + "the same Ansible bootstrap to run for prod") + seen: set[str] = set() + in_hosts = False + for line in lh.stdout.splitlines(): + if re.match(r"\s*hosts \(\d+\):", line): + in_hosts = True + continue + if re.search(r"play #\d+", line): + in_hosts = False + continue + if in_hosts and line.strip(): + seen.add(line.strip()) + check(seen == set(prod_names), + f"the shared chain retargeted at prod resolves to {sorted(seen)}, not the three prod " + f"nodes {prod_names} — prod would be bootstrapped by something other than dev's path") + +# --------------------------------------------------------------------------- verdict +check(CHECKS >= 25, + f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass " + "would be meaningless") + +if FAILURES: + print(f"CHECK FAIL (prod-baremetal-not-dev-surface): {len(FAILURES)} of {CHECKS} assertions failed", + file=sys.stderr) + for f in FAILURES: + print(f" - {f}", file=sys.stderr) + raise SystemExit(1) +for n in NOTES: + print(f" note: {n}") +print(f"check OK: prod-baremetal-not-dev-surface — {CHECKS} assertions; prod group {PROD_GROUP!r} = " + f"{prod_names} (3 nodes, none created by IaC), overlay clusters/{prod_overlay} renders the base " + f"plus environment-scoped patches only, local Longhorn path " + f"{data_paths.get(prod_overlay, data_paths.get('base'))!r} with " + f"{replicas.get(prod_overlay, replicas.get('base'))} replica(s). R-ENV-03's AUTHORED clauses " + "only — the hardware facts (mini-PC, bare metal, local NVMe) are operator-attested and the " + "live cluster's shape is NOT asserted here (see the header's gap list).") +PY diff --git a/conformance/tests/l0-static/tls-lifecycle-automated/chainsaw-test.yaml b/conformance/tests/l0-static/tls-lifecycle-automated/chainsaw-test.yaml new file mode 100644 index 0000000..f137268 --- /dev/null +++ b/conformance/tests/l0-static/tls-lifecycle-automated/chainsaw-test.yaml @@ -0,0 +1,69 @@ +# ============================================================================= +# L0 STATIC — certificate lifecycle needs no manual step (R-TLS-02) [EXECUTABLE] +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# Spec: docs/00 R-TLS-02 ("Certificates MUST be automatically issued, renewed, and +# revoked without manual steps"); mechanism in docs/08 + ADR-0006. +# +# R-TLS-02 is one adverbial clause plus three runtime verbs. This test asserts the +# clause — "without manual steps" — which in a GitOps platform is a property of the +# REPOSITORY: a human step in the certificate path leaves evidence in git (committed +# cert material, a Gateway serving a Secret nothing declares, an issuer with no +# automated solver, a credential never delivered, auto-approval switched off, or CRs +# nothing reconciles). The three verbs are observed by the companion L2 test. +# +# ENVIRONMENT-AGNOSTIC (R-ENV-02): the issuer a Certificate names is `${ACME_CLUSTER_ISSUER}`, +# so the check resolves it against the BASE cluster-vars default AND every overlay's value and +# requires each resolved issuer to exist. dev's issuer and prod's issuer are both verified from +# one authored assertion, with no hostname, address or environment name written down. +# +# GIVEN the repository as git tracks it, +# WHEN every serving-TLS reference, Certificate, Issuer and the cert-manager release +# is inspected offline, +# THEN every Gateway listener that terminates TLS serves a Secret a tracked Certificate +# produces; no certificate material is committed anywhere; every Certificate's +# issuer resolves — in EVERY environment — to a tracked ACME issuer with an +# automated dns01/http01 solver whose credential is SOPS-delivered from git; +# cert-manager's automatic request approval is not disabled; and every one of +# those objects is rendered by a Flux Kustomization, so nothing in the chain +# depends on a human running kubectl. +# +# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git + PyYAML, no +# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and +# this Chainsaw Test wraps the SAME script. +# +# KNOWN, DELIBERATE GAPS (named in check.sh's header too — this test claims NONE of them): +# * ISSUED / RENEWED as runtime events: L2 (l2-behavioral/cert-lifecycle-automated). +# * REVOKED: not proven at any layer today. cert-manager performs no ACME revocation, +# and the spec's own reading (docs/08: "removing/rotating a certificate is a +# declarative change") needs `enableCertificateOwnerRef`, which the tracked +# cert-manager HelmRelease does not set — so a removed Certificate leaves its +# serving Secret behind. The L2 test asserts the withdrawal behaviour and is +# EXPECTED TO FAIL until that knob is set; escalated, not softened (R-TEST-06). +# * That the live cluster is serving the tracked Certificate (cluster state). +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: tls-lifecycle-automated + labels: + conformance.home.k3s/R-TLS-02: "true" + annotations: + conformance.home.k3s/requirements: "R-TLS-02" + conformance.home.k3s/layer: "L0" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Given the tracked tree, when the certificate path is inspected offline, then no + manual step exists anywhere in it: serving Secrets are cert-manager-produced, no + cert material is committed, every environment's issuer is a tracked ACME issuer + with an automated solver and a git-delivered credential, auto-approval is on, and + the whole chain is Flux-reconciled. +spec: + steps: + - name: then-no-manual-step-exists-in-the-certificate-path + description: >- + Run the offline assertion script over the tracked tree (same script ci/l0.sh + and conformance/run.sh --layer l0 execute). + try: + - script: + content: bash ./check.sh diff --git a/conformance/tests/l0-static/tls-lifecycle-automated/check.sh b/conformance/tests/l0-static/tls-lifecycle-automated/check.sh new file mode 100755 index 0000000..294b4b1 --- /dev/null +++ b/conformance/tests/l0-static/tls-lifecycle-automated/check.sh @@ -0,0 +1,478 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 static check (offline, no cluster) — tls-lifecycle-automated +# Serves: R-TLS-02 ("Certificates MUST be automatically **issued, renewed, and revoked** +# without manual steps." — spec docs/00-vision-and-scope.md; mechanism in +# docs/08-tls-and-certificates.md, ADR-0006.) +# +# WHICH CLAUSE THIS LAYER CAN PROVE — AND WHICH IT CANNOT +# ------------------------------------------------------ +# R-TLS-02 has one adverbial clause ("without manual steps") and three verbs +# ("issued", "renewed", "revoked"). In a GitOps platform the *without manual steps* +# clause IS a repository property, and it is the half that can be proven with no +# cluster: if a human had to mint, install, or approve a serving certificate, the +# evidence of that human step would be IN THE REPO — a committed `tls.crt`, a Gateway +# pointing at a Secret nothing declares, an issuer that no automated solver can +# satisfy, a per-cert credential handed over out of band, or an admission path that +# waits for a manual CertificateRequest approval. This check asserts that none of +# those exist and that the automated path is completely wired instead. +# +# The three VERBS are runtime events. They are asserted by the companion L2 test +# (conformance/tests/l2-behavioral/cert-lifecycle-automated), which observes an +# issuance, an unattended renewal, and the withdrawal of a removed certificate. This +# check does NOT claim them; see "WHAT THIS DOES *NOT* PROVE" below. +# +# GIVEN the repository as git tracks it, +# WHEN every serving-TLS reference, Certificate, Issuer and the cert-manager release +# are inspected offline, +# THEN (1) at least one cert-manager `Certificate` is authored (vacuity floor: a repo +# with no Certificate cannot pass this check by having nothing to check); +# (2) every Gateway HTTPS listener that TERMINATES TLS references ONLY Secrets +# that a tracked Certificate declares as its `secretName` — so the serving +# credential is produced by the controller, never dropped in by hand; +# (3) NO certificate material is tracked anywhere: no Secret manifest of type +# kubernetes.io/tls, no tls.crt/tls.key/ca.crt key in any committed Secret +# (plain OR SOPS-encrypted), and no PEM certificate body in any tracked +# YAML — i.e. issuance is not, and cannot silently become, a human step; +# (4) every Certificate's `issuerRef` resolves to a TRACKED issuer IN EVERY +# ENVIRONMENT — the `${ACME_CLUSTER_ISSUER}` token is resolved against the +# base cluster-vars default AND every overlay's value, so an environment +# whose issuer does not exist is caught here rather than at prod bring-up; +# (5) every issuer so referenced is an ACME issuer with a NON-EMPTY solver list +# whose every solver is an automated challenge type (dns01 / http01) — the +# same code path serves issuance AND renewal, so a solver a human must +# complete would defeat both; +# (6) each dns01 solver's provider credential is referenced BY NAME and that +# Secret is delivered SOPS-encrypted from git via a kustomization — the +# operator's one-time token hand-off (docs/08 OQ-4), never a per-cert step; +# (7) cert-manager's automatic CertificateRequest approval is NOT disabled +# (`disableAutoApproval`), and any `approveSignerNames` narrowing still +# covers the ClusterIssuers actually used — a request that waits for a +# human approver is the definition of a manual step; +# (8) the whole chain is reconciled from git — cert-manager's HelmRelease, the +# ClusterIssuers and the Certificate are each included in a kustomization +# that a Flux Kustomization renders — so a deleted Certificate is +# re-created and re-issued with no human action; +# (9) no Certificate defeats renewal arithmetically (`renewBefore` >= `duration` +# would mean the renewal instant never arrives). +# +# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them): +# - ISSUED: that an issuance actually happens. Asserted at L2 (step 2 of +# l2-behavioral/cert-lifecycle-automated) and, for the real ACME/DNS-01 path, +# evidenced by l2-behavioral/trusted-tls serving a publicly-trusted cert. +# - RENEWED: that a renewal actually happens unattended. docs/08's acceptance gate +# is explicit that this must be "observed across a renewal" — L2 step 3 does that +# by fast-forwarding a certificate's renewal window; nothing static can. +# - REVOKED: NOT PROVEN AT ANY LAYER TODAY, and not asserted here. Two distinct +# readings, neither established: (a) ACME revocation — cert-manager does not +# revoke certificates at all, so no repo state could show it; (b) the spec's own +# reading in docs/08 ("removing/rotating a certificate is a declarative change") +# — withdrawal of a REMOVED certificate's serving material requires cert-manager's +# `enableCertificateOwnerRef`, which the tracked HelmRelease does NOT set (chart +# default false). L2 step 4 asserts the withdrawal behaviour and is expected to +# FAIL until that knob is set; escalated to the PM rather than softened (R-TEST-06). +# - That the tracked Certificate is the one the live Gateway is actually serving +# (cluster state), or that the ACME account/DNS credential is still valid. +# - Certificate EXPIRY ALERTING, which docs/08 delegates to R-OBS-04 (see +# l3-fault-injection/alerting). +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +PYTHON="${PYTHON:-python3}" + +cd "${ROOT}" +"${PYTHON}" - "${ROOT}" <<'PY' +import re +import subprocess +import sys +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover + print("check FAIL: PyYAML is required (same dependency as catalog/render.py)", + file=sys.stderr) + raise SystemExit(1) + +ROOT = Path(sys.argv[1]) +FAILURES: list[str] = [] +CHECKS = 0 + + +def check(cond, msg): + global CHECKS + CHECKS += 1 + if not cond: + FAILURES.append(msg) + + +def tracked() -> list[str]: + out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"], + capture_output=True, text=True, check=True).stdout + return [p for p in out.split("\0") if p] + + +FILES = tracked() +check(len(FILES) > 50, f"git ls-files returned only {len(FILES)} paths — not a real checkout") +YAML_FILES = [f for f in FILES if re.search(r"\.(ya?ml)$", f)] + + +def docs_of(rel: str): + """Parsed YAML documents of a tracked file, or [] when it is not parseable YAML.""" + try: + return [d for d in yaml.safe_load_all((ROOT / rel).read_text(errors="replace")) + if isinstance(d, dict)] + except Exception: + return [] + + +DOCS: list[tuple[str, dict]] = [(rel, d) for rel in YAML_FILES for d in docs_of(rel)] + + +def of_kind(kind: str, group_prefix: str = "") -> list[tuple[str, dict]]: + out = [] + for rel, d in DOCS: + if d.get("kind") != kind: + continue + if group_prefix and not str(d.get("apiVersion", "")).startswith(group_prefix): + continue + out.append((rel, d)) + return out + + +def ns_of(d: dict, default: str = "default") -> str: + return (d.get("metadata") or {}).get("namespace") or default + + +def name_of(d: dict) -> str: + return (d.get("metadata") or {}).get("name") or "" + + +# ------------------------------------------------------------ cluster-vars resolution +# Every environment's value for a ${TOKEN}: the base default plus each overlay's JSON6902 +# `- op: replace / path: /data/KEY / value: ...` patch on the cluster-vars ConfigMap. This +# is the same mechanism flux-postbuild-substitution audits; here we need the VALUES, so an +# issuer name can be resolved per environment instead of being asserted for dev only. +def env_values(key: str) -> dict[str, str]: + vals: dict[str, str] = {} + base = ROOT / "clusters" / "base" / "cluster-vars.yaml" + if base.is_file(): + for d in yaml.safe_load_all(base.read_text()): + if isinstance(d, dict) and d.get("kind") == "ConfigMap" \ + and name_of(d) == "cluster-vars": + v = (d.get("data") or {}).get(key) + if v is not None: + vals["base"] = str(v) + for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")): + env = overlay.parent.name + if env == "base": + continue + try: + k = yaml.safe_load(overlay.read_text()) or {} + except Exception: + continue + for p in (k.get("patches") or []): + tgt = p.get("target") or {} + if tgt.get("kind") != "ConfigMap" or tgt.get("name") != "cluster-vars": + continue + try: + ops = yaml.safe_load(p.get("patch") or "") or [] + except Exception: + continue + for op in ops if isinstance(ops, list) else []: + if isinstance(op, dict) and str(op.get("path", "")) == f"/data/{key}": + vals[env] = str(op.get("value")) + return vals + + +TOKEN_RE = re.compile(r"^\$\{([A-Z0-9_]+)\}$") + + +def resolve_per_env(value: str) -> dict[str, str]: + """A literal resolves to itself in every environment; a ${TOKEN} to each env's value.""" + m = TOKEN_RE.match(value.strip()) + if not m: + return {"(literal)": value.strip()} + return env_values(m.group(1)) + + +# --------------------------------------------- what Flux actually renders (reachability) +# A manifest is DELIVERED only if it is reachable from a Flux Kustomization's `path` through +# the kustomize resource graph: /kustomization.yaml -> resources -> (file | directory +# -> that directory's kustomization.yaml -> ...). Membership is deliberately computed by +# walking that graph rather than by asking "is this file's DIRECTORY mentioned somewhere": +# the loose form silently passes when an entry is deleted from the very kustomization.yaml +# that the directory reference renders (verified — it made two assertions here no-ops). +KUST_RESOURCES: dict[str, list[str]] = {} +for rel in [f for f in YAML_FILES if f.endswith("kustomization.yaml")]: + refs: list[str] = [] + for d in docs_of(rel): + for r in (d.get("resources") or []) + (d.get("components") or []): + if not isinstance(r, str) or "://" in r: # skip remote bases + continue + try: + p = (ROOT / rel).parent.joinpath(r).resolve() + refs.append(str(p.relative_to(ROOT))) + except Exception: + continue + KUST_RESOURCES[rel] = refs + +FLUX_PATHS: set[str] = set() +for _rel, _d in DOCS: + if _d.get("kind") == "Kustomization" and \ + str(_d.get("apiVersion", "")).startswith("kustomize.toolkit.fluxcd.io/"): + p = str((_d.get("spec") or {}).get("path", "")).lstrip("./").rstrip("/") + if p: + FLUX_PATHS.add(p) + + +def _walk(entry: str, seen: set[str], out: set[str]) -> None: + if entry in seen: + return + seen.add(entry) + kust = entry if entry.endswith("kustomization.yaml") else f"{entry}/kustomization.yaml" + if kust in KUST_RESOURCES: # a directory (or a kustomization) + for r in KUST_RESOURCES[kust]: + _walk(r, seen, out) + return + out.add(entry) # a leaf manifest + + +RENDERED: set[str] = set() +for _p in FLUX_PATHS: + _walk(_p, set(), RENDERED) + + +def delivered(rel: str) -> bool: + """True iff Flux renders this file from one of its Kustomization paths.""" + return rel in RENDERED + + +# --------------------------------------------------- (1) there is something to check +CERTS = of_kind("Certificate", "cert-manager.io/") +check(len(CERTS) >= 1, + "no cert-manager.io Certificate is authored anywhere in the tracked tree — every " + "assertion below would be vacuous, so this is a FAILURE, not a pass (R-TLS-02)") + +ISSUERS = of_kind("ClusterIssuer", "cert-manager.io/") + of_kind("Issuer", "cert-manager.io/") +check(len(ISSUERS) >= 1, + "no cert-manager ClusterIssuer/Issuer is authored — nothing could issue automatically") + +# ---------------------------- (2) every terminating listener serves a controller cert +CERT_SECRETS: dict[tuple[str, str], str] = { # (namespace, secretName) -> file + (ns_of(d, "default"), str((d.get("spec") or {}).get("secretName", ""))): rel + for rel, d in CERTS +} +listeners_checked = 0 +for rel, gw in of_kind("Gateway", "gateway.networking.k8s.io/"): + gw_ns = ns_of(gw) + for lis in ((gw.get("spec") or {}).get("listeners") or []): + tls = lis.get("tls") or {} + if str(tls.get("mode", "")) != "Terminate": + continue + listeners_checked += 1 + refs = tls.get("certificateRefs") or [] + check(len(refs) > 0, + f"{rel}: Gateway {gw_ns}/{name_of(gw)} listener {lis.get('name')!r} terminates " + "TLS with NO certificateRefs — it can only serve a certificate someone puts " + "there by hand") + for r in refs: + kind = str(r.get("kind", "Secret")) + key = (str(r.get("namespace") or gw_ns), str(r.get("name"))) + check(kind == "Secret", + f"{rel}: listener {lis.get('name')!r} certificateRef kind is {kind!r}; only " + "Secret refs can be traced to a cert-manager Certificate") + check(key in CERT_SECRETS, + f"{rel}: listener {lis.get('name')!r} serves Secret {key[0]}/{key[1]}, which " + "NO tracked cert-manager Certificate declares as its secretName — that " + "credential would have to be created by hand (R-TLS-02 'without manual steps')") +check(listeners_checked > 0, + "no Gateway listener with tls.mode=Terminate found in the tracked tree — there is no " + "serving-TLS reference to trace, so (2) proved nothing") + +# --------------------------------- (3) no certificate material is tracked, anywhere +TLS_KEYS = {"tls.crt", "tls.key", "ca.crt", "tls-combined.pem"} +for rel, d in DOCS: + if d.get("kind") != "Secret": + continue + stype = str(d.get("type", "")) + check(stype != "kubernetes.io/tls", + f"{rel}: a Secret of type kubernetes.io/tls is COMMITTED (metadata.name=" + f"{name_of(d)}) — a serving certificate authored by a human, not issued") + for field in ("data", "stringData"): + for k in (d.get(field) or {}): + check(str(k) not in TLS_KEYS, + f"{rel}: committed Secret {name_of(d)} carries {field}.{k} — certificate " + "material in git means a manual issuance step exists (R-TLS-02)") + +PEM_RE = re.compile(r"-----BEGIN (?:[A-Z ]+ )?CERTIFICATE-----") +for rel in YAML_FILES: + body = (ROOT / rel).read_text(errors="replace") + # Scoped to YAML manifests deliberately: prose/docs may legitimately SHOW a PEM header + # in an example, and a .md file is not applied to any cluster. A manifest cannot. + check(not PEM_RE.search(body), + f"{rel}: contains a PEM CERTIFICATE body — a hand-minted certificate in a manifest") + +# --------------------------- (4)+(5)+(6) the issuer chain is automated in EVERY env +# The cert-manager release's namespace IS the controller's `--cluster-resource-namespace` +# (chart default), i.e. where a ClusterIssuer's solver credential must live. Derived from the +# HelmRelease, never written down here, so this holds if the release namespace ever changes. +cm_releases = [(rel, d) for rel, d in of_kind("HelmRelease") if name_of(d) == "cert-manager"] +CM_NS = ns_of(cm_releases[0][1], "cert-manager") if cm_releases else "cert-manager" + +ISSUER_INDEX = {(d.get("kind"), name_of(d)): (rel, d) for rel, d in ISSUERS} +solver_creds: list[tuple[str, str, str, str]] = [] # (issuer, ns, secretName, ref field) +issuers_verified: set[str] = set() +for rel, cert in CERTS: + spec = cert.get("spec") or {} + ref = spec.get("issuerRef") or {} + kind = str(ref.get("kind", "Issuer")) + per_env = resolve_per_env(str(ref.get("name", ""))) + check(len(per_env) > 0, + f"{rel}: Certificate {name_of(cert)} issuerRef.name is a token with NO value in " + "clusters/base/cluster-vars.yaml or any overlay — it would render empty and never issue") + for env, iname in sorted(per_env.items()): + hit = ISSUER_INDEX.get((kind, iname)) + check(hit is not None, + f"{rel}: Certificate {name_of(cert)} resolves issuerRef to {kind}/{iname} in " + f"environment {env!r}, but no such issuer is tracked — that environment cannot " + "issue anything without a hand-created issuer (R-TLS-02, R-ENV-02)") + if hit is None: + continue + irel, issuer = hit + issuers_verified.add(iname) + ispec = issuer.get("spec") or {} + acme = ispec.get("acme") + check(isinstance(acme, dict), + f"{irel}: {kind}/{iname} (used by {name_of(cert)} in env {env!r}) is not an ACME " + f"issuer — its issuance/renewal path is {sorted(ispec)}; docs/08/ADR-0006 mandate " + "ACME so both are unattended") + if not isinstance(acme, dict): + continue + check(str(acme.get("server", "")).startswith("https://"), + f"{irel}: {kind}/{iname} has no https ACME directory endpoint") + check(bool((acme.get("privateKeySecretRef") or {}).get("name")), + f"{irel}: {kind}/{iname} declares no privateKeySecretRef — the ACME account key " + "would have to be provisioned by hand") + solvers = acme.get("solvers") or [] + check(len(solvers) > 0, + f"{irel}: {kind}/{iname} declares NO solvers — every order would sit unsolved " + "until a human completed the challenge") + for s in solvers: + types = [t for t in ("dns01", "http01") if t in s] + check(len(types) == 1, + f"{irel}: {kind}/{iname} has a solver with challenge types {sorted(s)} — only " + "the automated dns01/http01 solvers satisfy 'without manual steps'") + for provider, cfg in (s.get("dns01") or {}).items(): + if not isinstance(cfg, dict): + continue + for field, val in cfg.items(): + if isinstance(val, dict) and str(field).endswith(("SecretRef", "secretRef")): + n = str(val.get("name", "")) + check(bool(n), + f"{irel}: {kind}/{iname} dns01 {provider}.{field} has no secret name") + if n: + # A ClusterIssuer resolves secret refs in the controller's + # cluster-resource-namespace; a namespaced Issuer in its own. + want_ns = CM_NS if kind == "ClusterIssuer" else ns_of(issuer, CM_NS) + solver_creds.append((iname, want_ns, n, f"{provider}.{field}")) + +check(len(issuers_verified) > 0, "no issuer was reachable from any Certificate — (5) proved nothing") + +# (6) the solver credential is delivered, SOPS-encrypted, from git — never per-cert by hand. +check(len(solver_creds) > 0, + "no dns01 solver credential reference found — docs/08 (OQ-4) requires the scoped " + "Cloudflare token to drive DNS-01, so this check would otherwise be vacuous") +# Keyed by (namespace, name): a same-named Secret in ANOTHER namespace is NOT this credential. +# (Verified: keying by name alone let the external-dns copy of `cloudflare-api-token` satisfy +# the cert-manager reference, so deleting cert-manager's own copy passed — a silent no-op.) +SECRET_DOCS = {(ns_of(d, CM_NS), name_of(d)): (rel, d) + for rel, d in DOCS if d.get("kind") == "Secret"} +for iname, sns, sname, field in solver_creds: + hit = SECRET_DOCS.get((sns, sname)) + check(hit is not None, + f"{iname}: dns01 {field} references Secret {sname!r}, which no tracked manifest " + f"delivers into namespace {sns!r} (the issuer's secret-resolution namespace) — the " + "credential would be created by hand in-cluster (R-TLS-02)") + if hit is None: + continue + srel, sdoc = hit + check("sops" in sdoc, + f"{srel}: Secret {sname} is the DNS-01 credential but is not SOPS-encrypted (R-SEC-01)") + check(delivered(srel), + f"{srel}: Secret {sname} is not rendered by any Flux Kustomization path — Flux would " + "never deliver it, so issuance would wait on a manual kubectl apply") + +# ------------------------------- (7) automatic CertificateRequest approval is intact +check(len(cm_releases) == 1, + f"expected exactly one cert-manager HelmRelease, found {[r for r, _ in cm_releases]} — " + "the controller that issues and renews must itself be GitOps-managed") +for rel, hr in cm_releases: + values = ((hr.get("spec") or {}).get("values") or {}) + check(values.get("disableAutoApproval") in (None, False), + f"{rel}: cert-manager values set disableAutoApproval={values.get('disableAutoApproval')!r} " + "— every CertificateRequest would then wait for an approver, which is a manual step") + signers = values.get("approveSignerNames") + if signers is not None: + pats = [str(s) for s in (signers if isinstance(signers, list) else [signers])] + for iname in sorted(issuers_verified): + ok = any(p in ("*", "*/*") or "clusterissuers.cert-manager.io" in p and + (p.endswith("/*") or p.endswith(f"/{iname}")) for p in pats) + check(ok, + f"{rel}: approveSignerNames={pats} does not cover ClusterIssuer {iname} — " + "requests for it would never be auto-approved") + # The chart's own CRD install must be on, else the CRs below cannot even be applied. + check((values.get("crds") or {}).get("enabled") is True, + f"{rel}: cert-manager values do not enable CRD installation — the Certificate CRD " + "would have to be installed by hand") + +# ------------------------------- (8) the whole chain is continuously reconciled from git +check(len(FLUX_PATHS) > 0 and len(RENDERED) > 20, + f"the kustomize resource graph resolved {len(RENDERED)} manifest(s) from {len(FLUX_PATHS)} " + "Flux Kustomization path(s) — too few to be a real graph, so (8) would prove nothing") +for rel, _ in cm_releases + CERTS + ISSUERS: + check(delivered(rel), + f"{rel} is not rendered by any Flux Kustomization path — nothing reconciles it, so a " + "deleted Certificate/issuer/controller would stay deleted until a human re-applied it") + +# --------------------------------------------- (9) renewal is not defeated arithmetically +def to_seconds(v) -> int | None: + m = re.fullmatch(r"\s*(\d+)\s*([smh])\s*", str(v)) + if not m: + return None + return int(m.group(1)) * {"s": 1, "m": 60, "h": 3600}[m.group(2)] + + +renew_pairs = 0 +for rel, cert in CERTS: + spec = cert.get("spec") or {} + dur, rb = to_seconds(spec.get("duration")), to_seconds(spec.get("renewBefore")) + if dur is None or rb is None: + continue + renew_pairs += 1 + check(rb < dur, + f"{rel}: Certificate {name_of(cert)} sets renewBefore={spec.get('renewBefore')} >= " + f"duration={spec.get('duration')} — the renewal instant never arrives") + +# --------------------------------------------------------------------------- verdict +# NO-OP GUARD: this repo has been bitten by checks that degrade to a successful no-op. +# The floor below is a count of assertions that MUST have run for the verdict to mean +# anything; it is not a style rule. +check(CHECKS >= 20, + f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass " + "would be meaningless") + +if FAILURES: + print(f"CHECK FAIL (tls-lifecycle-automated): {len(FAILURES)} of {CHECKS} assertions failed", + file=sys.stderr) + for f in FAILURES: + print(f" - {f}", file=sys.stderr) + raise SystemExit(1) +print(f"check OK: tls-lifecycle-automated — {CHECKS} assertions; {len(CERTS)} Certificate(s), " + f"{len(ISSUERS)} issuer(s), {len(issuers_verified)} issuer(s) reached from a Certificate " + f"across every environment, {listeners_checked} terminating listener(s), " + f"{len(solver_creds)} dns01 credential ref(s), {renew_pairs} duration/renewBefore pair(s). " + "R-TLS-02 'without manual steps' clause only — issued/renewed/revoked are RUNTIME verbs " + "(see l2-behavioral/cert-lifecycle-automated; 'revoked' is NOT proven at any layer).") +PY diff --git a/conformance/tests/l2-behavioral/cert-lifecycle-automated/chainsaw-test.yaml b/conformance/tests/l2-behavioral/cert-lifecycle-automated/chainsaw-test.yaml new file mode 100644 index 0000000..14fca5e --- /dev/null +++ b/conformance/tests/l2-behavioral/cert-lifecycle-automated/chainsaw-test.yaml @@ -0,0 +1,391 @@ +# ============================================================================= +# L2 BEHAVIORAL — certificate lifecycle is automatic (R-TLS-02) +# [RUNNABLE / NOT YET EXECUTED — see STATUS] +# Catalog: docs/14 "Trusted TLS (R-TLS-01/02)" +# Spec: docs/00 R-TLS-02 — "Certificates MUST be automatically **issued, renewed, +# and revoked** without manual steps." +# docs/08 §"Renewal & revocation (R-TLS-02)" — "cert-manager renews well +# before expiry automatically; no human action." / "Revocation: removing/ +# rotating a certificate is a declarative change; cert-manager handles +# issuance of replacements." +# docs/08 acceptance gate — "Certificates renew automatically ahead of expiry +# with no manual step (**observed across a renewal**)." +# Impl: platform/cert-manager/**, platform/gateway/wildcard-certificate.yaml. +# Sibling: tests/l0-static/tls-lifecycle-automated proves the "without manual steps" +# clause statically (in the merge gate). This test observes the three VERBS. +# +# ENVIRONMENT-AGNOSTIC (R-ENV-02): NOTHING about either environment is written down. The +# serving certificate is DISCOVERED — find the Gateway whose listener terminates TLS, follow +# its certificateRefs Secret, and read that Secret's `cert-manager.io/certificate-name` +# annotation to reach the Certificate. No hostname, no namespace list, no issuer name, no +# address: dev proves dev's cert and prod proves prod's from the same authored assertions. +# +# GIVEN cert-manager reconciled by Flux and a Gateway terminating TLS, +# WHEN the serving Certificate's provenance and renewal schedule are read, and a NEW +# Certificate is then requested, left to renew itself, and removed, +# THEN 1. the serving credential was produced by the CONTROLLER, not a human: its +# Secret carries cert-manager's certificate-name annotation, its +# CertificateRequest was submitted by cert-manager's own ServiceAccount and +# was APPROVED automatically, and its issuer is a Ready ACME ClusterIssuer; +# 2. an unattended RENEWAL IS SCHEDULED: status.renewalTime is in the future and +# strictly before status.notAfter, so no human is in the renewal path; +# 3. ISSUED — a newly declared Certificate becomes Ready and its Secret appears +# with certificate material, with no step between declaring and having it; +# 4. RENEWED — that certificate, given a renewal window that opens one minute +# after issuance, REISSUES ITSELF at least twice (status.revision >= 3) and +# the serving bytes actually change, with no human action. This is docs/08's +# "observed across a renewal", fast-forwarded; +# 5. REMOVED/REVOKED — deleting the Certificate withdraws its serving material. +# +# WHY A HERMETIC SELF-SIGNED ISSUER FOR STEPS 3-5, AND WHAT THAT COSTS +# ------------------------------------------------------------------- +# Steps 3-5 need a certificate lifecycle that completes in minutes and repeats on every +# run. Driving that through the real ACME issuer would fire a public Let's Encrypt order +# (and a Cloudflare _acme-challenge write) per conformance run, burning issuance rate +# limits for evidence that is not about ACME: cert-manager's renewal scheduler and +# ownership logic are issuer-independent. So steps 3-5 use a `selfSigned` Issuer created +# in this test's OWN namespace, and the ACME/DNS-01 path is proven elsewhere — step 1-2 +# read the REAL wildcard certificate the Gateway serves, and l2-behavioral/trusted-tls +# proves a publicly-trusted LE certificate is actually served (R-TLS-01/03). +# STATED PLAINLY: steps 3-5 do NOT prove the ACME order path renews; they prove the +# CONTROLLER's issue/renew/withdraw automation runs unattended in this cluster, while +# steps 1-2 prove the real serving certificate is on exactly that automated path. +# +# STATUS: RUNNABLE, NOT YET EXECUTED. Steps 1-2 are read-only. Steps 3-5 CREATE an +# Issuer/Certificate (and DELETE the Certificate they created) inside chainsaw's +# per-test namespace — the same class of mutation every other L2 test here performs, +# self-cleaning via namespace teardown, touching no platform object. WS-TEST authored +# them against live dev/prod read-only inspection under an explicit instruction not to +# mutate either cluster, so THE MUTATING STEPS HAVE NOT BEEN RUN. Nothing about the +# suite gates on them: ci/conformance.sh only runs L2/L3 when CLUSTER_AVAILABLE=true. +# +# EXPECTED FAILURE, DELIBERATELY NOT SOFTENED (R-TEST-06) +# ------------------------------------------------------ +# Step 5 WILL FAIL as the platform is authored today. cert-manager only garbage-collects +# a Certificate's Secret when the controller runs with `enableCertificateOwnerRef: true`; +# platform/cert-manager/helmrelease.yaml does not set it (chart default false), and the +# live wildcard Secret on both clusters has NO ownerReferences (verified read-only, +# 2026-09-05). So a removed Certificate leaves its credential in place, and "revoked +# without manual steps" does not hold. That is a finding about the platform, not a +# defect in this test: the assertion stays as the requirement reads and is escalated to +# the PM. (ACME revocation proper is not implemented by cert-manager at all.) +# +# KNOWN, DELIBERATE GAPS (this test claims none of these): +# * Not proven: that the CA would refuse a revoked certificate — there is no +# revocation request to make, so nothing observable exists to assert. +# * Not proven: renewal of the REAL wildcard cert (its window opens 30 days before a +# 90-day expiry; a conformance run cannot wait, and forcing it would rotate the +# platform's serving credential). Step 2 asserts the schedule; step 4 asserts the +# mechanism that will act on it. +# * Not proven: certificate expiry ALERTING — docs/08 delegates that to R-OBS-04 +# (l3-fault-injection/alerting). +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: cert-lifecycle-automated + labels: + conformance.home.k3s/R-TLS-02: "true" + annotations: + conformance.home.k3s/requirements: "R-TLS-02" + conformance.home.k3s/layer: "L2" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Given cert-manager and a TLS-terminating Gateway, when the serving certificate's + provenance and renewal schedule are read and a new certificate is requested, left + to renew, and removed, then issuance, renewal and withdrawal all happen with no + human step — the serving cert was controller-issued and auto-approved, its + renewal is scheduled ahead of expiry, and a declared certificate issues and + reissues itself unattended. +spec: + timeouts: + apply: 60s + assert: 3m + exec: 10m + steps: + # ---------------------------------------------------------------- READ-ONLY HALF + - name: given-cert-manager-is-reconciled-and-healthy + description: >- + The controller that issues and renews must be running, and must have been + installed by Flux rather than by hand — otherwise "automatic" has no engine. + try: + - script: + content: | + set -u + # Discover cert-manager's Deployments by chart label, in whatever namespace + # the release lives (nothing about the environment is assumed). + sel='app.kubernetes.io/instance=cert-manager' + rows="$(kubectl get deploy -A -l "$sel" \ + -o custom-columns='NS:.metadata.namespace,N:.metadata.name,RDY:.status.readyReplicas' \ + --no-headers 2>/dev/null || true)" + echo "$rows" + [ -n "$rows" ] || { echo "FAIL: no cert-manager Deployment found ($sel)"; exit 1; } + n=0 + echo "$rows" | while read -r ns name rdy; do + [ "${rdy:-0}" -ge 1 ] || { echo "FAIL: $ns/$name has no ready replica"; exit 1; } + done || exit 1 + n="$(echo "$rows" | wc -l)" + # controller + webhook + cainjector: without the webhook a Certificate is + # rejected, without cainjector the webhook has no CA — all three or nothing. + [ "$n" -ge 3 ] || { echo "FAIL: expected >=3 cert-manager Deployments, saw $n"; exit 1; } + # GitOps-managed, not hand-installed: Flux labels what it applies. + ns="$(echo "$rows" | head -1 | awk '{print $1}')" + lbl="$(kubectl get deploy -n "$ns" -l "$sel" \ + -o jsonpath='{.items[0].metadata.labels.helm\.toolkit\.fluxcd\.io/name}' 2>/dev/null || true)" + [ -n "$lbl" ] || { echo "FAIL: cert-manager is not owned by a Flux HelmRelease"; exit 1; } + echo "CERT_MANAGER_HEALTHY_OK ($n deployments, flux HelmRelease $lbl)" + check: + (contains($stdout, 'CERT_MANAGER_HEALTHY_OK')): true + + - name: then-the-serving-certificate-was-issued-by-the-controller + description: >- + Follow the Gateway's terminating listener to the Secret it serves, and prove + that Secret's provenance is cert-manager — annotation, CertificateRequest + requester, automatic approval, and a Ready ACME issuer. + try: + - script: + content: | + set -u + # --- discover the serving Secret from the Gateway itself ------------------- + gw="$(kubectl get gateway -A \ + -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}' \ + 2>/dev/null | head -1)" + [ -n "$gw" ] || { echo "FAIL: no Gateway found — nothing terminates TLS"; exit 1; } + gns="${gw% *}"; gname="${gw#* }" + sec="$(kubectl -n "$gns" get gateway "$gname" \ + -o jsonpath='{.spec.listeners[?(@.tls.mode=="Terminate")].tls.certificateRefs[0].name}' \ + 2>/dev/null | awk '{print $1}')" + [ -n "$sec" ] || { echo "FAIL: Gateway $gns/$gname terminates no TLS listener"; exit 1; } + echo "gateway=$gns/$gname serving-secret=$sec" + + # --- provenance (1): only the controller writes this annotation ------------ + cert="$(kubectl -n "$gns" get secret "$sec" \ + -o jsonpath='{.metadata.annotations.cert-manager\.io/certificate-name}' 2>/dev/null || true)" + [ -n "$cert" ] || { echo "FAIL: Secret $gns/$sec carries no cert-manager.io/certificate-name annotation — it was not produced by cert-manager"; exit 1; } + crt="$(kubectl -n "$gns" get secret "$sec" -o jsonpath='{.data.tls\.crt}' 2>/dev/null || true)" + [ -n "$crt" ] || { echo "FAIL: Secret $gns/$sec holds no tls.crt"; exit 1; } + + # --- provenance (2): the Certificate is Ready and has a revision ----------- + ready="$(kubectl -n "$gns" get certificate "$cert" \ + -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)" + rev="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.status.revision}' 2>/dev/null || true)" + echo "certificate=$gns/$cert ready=$ready revision=$rev" + [ "$ready" = "True" ] || { echo "FAIL: Certificate $cert is not Ready"; exit 1; } + [ -n "$rev" ] && [ "$rev" -ge 1 ] || { echo "FAIL: Certificate $cert has no issued revision"; exit 1; } + + # --- provenance (3): the request was made BY THE CONTROLLER and auto-approved + # spec.username is the identity that created the CertificateRequest. A human + # operator would appear as a user/OIDC identity, not a ServiceAccount. + cr="$(kubectl -n "$gns" get certificaterequest \ + -o jsonpath="{range .items[?(@.metadata.annotations.cert-manager\.io/certificate-name=='${cert}')]}{.metadata.name}{' '}{.spec.username}{'\n'}{end}" \ + 2>/dev/null | tail -1)" + [ -n "$cr" ] || { echo "FAIL: no CertificateRequest for $cert — cannot show who requested it"; exit 1; } + crname="${cr% *}"; requester="${cr#* }" + echo "certificaterequest=$crname requester=$requester" + case "$requester" in + system:serviceaccount:*) : ;; + *) echo "FAIL: CertificateRequest $crname was submitted by '$requester', not a controller ServiceAccount"; exit 1 ;; + esac + appr="$(kubectl -n "$gns" get certificaterequest "$crname" \ + -o jsonpath='{.status.conditions[?(@.type=="Approved")].status}' 2>/dev/null || true)" + [ "$appr" = "True" ] || { echo "FAIL: CertificateRequest $crname is not Approved=True (approval would be a manual step)"; exit 1; } + + # --- provenance (4): an automated ACME issuer, and it is Ready ------------- + ikind="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.spec.issuerRef.kind}')" + iname="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.spec.issuerRef.name}')" + echo "issuer=$ikind/$iname" + if [ "$ikind" = "ClusterIssuer" ]; then + iready="$(kubectl get clusterissuer "$iname" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)" + iacme="$(kubectl get clusterissuer "$iname" -o jsonpath='{.spec.acme.server}' 2>/dev/null || true)" + else + iready="$(kubectl -n "$gns" get issuer "$iname" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)" + iacme="$(kubectl -n "$gns" get issuer "$iname" -o jsonpath='{.spec.acme.server}' 2>/dev/null || true)" + fi + [ "$iready" = "True" ] || { echo "FAIL: issuer $iname is not Ready — renewal would need a human to fix it"; exit 1; } + [ -n "$iacme" ] || { echo "FAIL: issuer $iname is not an ACME issuer"; exit 1; } + echo "SERVING_CERT_CONTROLLER_ISSUED_OK" + check: + (contains($stdout, 'SERVING_CERT_CONTROLLER_ISSUED_OK')): true + + - name: then-an-unattended-renewal-is-already-scheduled + description: >- + cert-manager writes status.renewalTime only after computing when it will renew. + Asserting it is in the future and before expiry proves the renewal is scheduled + and that nothing is waiting on a human (docs/08 "renews well before expiry"). + try: + - script: + content: | + set -u + gw="$(kubectl get gateway -A \ + -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}' | head -1)" + gns="${gw% *}"; gname="${gw#* }" + sec="$(kubectl -n "$gns" get gateway "$gname" \ + -o jsonpath='{.spec.listeners[?(@.tls.mode=="Terminate")].tls.certificateRefs[0].name}' | awk '{print $1}')" + cert="$(kubectl -n "$gns" get secret "$sec" \ + -o jsonpath='{.metadata.annotations.cert-manager\.io/certificate-name}')" + renew="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.status.renewalTime}')" + after="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.status.notAfter}')" + [ -n "$renew" ] || { echo "FAIL: Certificate $cert has no status.renewalTime — no renewal is scheduled"; exit 1; } + [ -n "$after" ] || { echo "FAIL: Certificate $cert has no status.notAfter"; exit 1; } + now_s="$(date -u +%s)" + r_s="$(date -u -d "$renew" +%s)" + a_s="$(date -u -d "$after" +%s)" + echo "now=$(date -u -d "@$now_s" +%FT%TZ) renewalTime=$renew notAfter=$after" + echo "renewal is in $(( (r_s - now_s) / 86400 ))d; expiry in $(( (a_s - now_s) / 86400 ))d" + [ "$r_s" -gt "$now_s" ] || { echo "FAIL: renewalTime is in the PAST — the scheduled renewal did not happen"; exit 1; } + [ "$r_s" -lt "$a_s" ] || { echo "FAIL: renewalTime is not before notAfter — renewal would come too late"; exit 1; } + echo "RENEWAL_SCHEDULED_OK" + check: + (contains($stdout, 'RENEWAL_SCHEDULED_OK')): true + + # ------------------------------------------------- MUTATING HALF (hermetic namespace) + - name: when-a-certificate-is-declared-then-it-is-issued-with-no-human-step + description: >- + Declare an Issuer and a Certificate in this test's own namespace, with a renewal + window that opens one minute after issuance. Nothing else is done to them for the + rest of the test — every later transition is the controller acting on its own. + try: + - apply: + resource: + # selfSigned: no external CA, no rate limit, no DNS write. See the header for + # what this proves (controller automation) and what it does not (the ACME path). + apiVersion: cert-manager.io/v1 + kind: Issuer + metadata: + name: conformance-lifecycle + spec: + selfSigned: {} + - apply: + resource: + apiVersion: cert-manager.io/v1 + kind: Certificate + metadata: + name: conformance-lifecycle + spec: + secretName: conformance-lifecycle-tls + # 1h is cert-manager's minimum duration; renewing 59m before expiry puts the + # renewal instant ~1 minute after each issuance, so an unattended renewal is + # observable inside a test run instead of 30 days from now. + duration: 1h + renewBefore: 3540s + commonName: conformance-lifecycle.invalid + dnsNames: + - conformance-lifecycle.invalid + privateKey: + rotationPolicy: Always + issuerRef: + kind: Issuer + name: conformance-lifecycle + - assert: + resource: + apiVersion: cert-manager.io/v1 + kind: Certificate + metadata: + name: conformance-lifecycle + status: + conditions: + - type: Ready + status: "True" + revision: 1 + - script: + content: | + set -u + crt="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \ + -o jsonpath='{.data.tls\.crt}' 2>/dev/null || true)" + key="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \ + -o jsonpath='{.data.tls\.key}' 2>/dev/null || true)" + [ -n "$crt" ] && [ -n "$key" ] || { echo "FAIL: no certificate material was issued into conformance-lifecycle-tls"; exit 1; } + # Record the serving bytes so the renewal step can prove they CHANGED. + echo "$crt" | cksum > /tmp/conformance-cert-r1.cksum + echo "revision-1 cert fingerprint: $(cat /tmp/conformance-cert-r1.cksum)" + echo "CERT_ISSUED_OK" + check: + (contains($stdout, 'CERT_ISSUED_OK')): true + + - name: then-it-renews-itself-before-expiry-with-no-human-step + description: >- + Nothing touches the Certificate here. Poll until the controller has reissued it + at least twice on its own and the serving bytes differ from revision 1 — docs/08's + "observed across a renewal", and every request auto-approved. + try: + - script: + content: | + set -u + want=3 # revision 1 = issuance; >=3 means TWO unattended renewals + for i in $(seq 1 30); do + rev="$(kubectl -n "$NAMESPACE" get certificate conformance-lifecycle \ + -o jsonpath='{.status.revision}' 2>/dev/null || echo 0)" + echo "poll $i: revision=${rev:-0}" + [ "${rev:-0}" -ge "$want" ] && break + sleep 20 + done + rev="$(kubectl -n "$NAMESPACE" get certificate conformance-lifecycle \ + -o jsonpath='{.status.revision}' 2>/dev/null || echo 0)" + [ "${rev:-0}" -ge "$want" ] || { echo "FAIL: revision reached only ${rev:-0} — cert-manager did not renew unattended"; exit 1; } + + # The serving material must actually have rotated, not just the revision count. + now="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \ + -o jsonpath='{.data.tls\.crt}' | cksum)" + was="$(cat /tmp/conformance-cert-r1.cksum)" + echo "r1=$was now=$now" + [ "$now" != "$was" ] || { echo "FAIL: the Secret still holds the revision-1 certificate — nothing was actually reissued"; exit 1; } + + # Every renewal request was approved by the controller, never by a person. + bad=0 + for cr in $(kubectl -n "$NAMESPACE" get certificaterequest -o name 2>/dev/null); do + st="$(kubectl -n "$NAMESPACE" get "$cr" \ + -o jsonpath='{.status.conditions[?(@.type=="Approved")].status}' 2>/dev/null || true)" + who="$(kubectl -n "$NAMESPACE" get "$cr" -o jsonpath='{.spec.username}' 2>/dev/null || true)" + echo " $cr approved=$st requester=$who" + [ "$st" = "True" ] || bad=1 + case "$who" in system:serviceaccount:*) : ;; *) bad=1 ;; esac + done + [ "$bad" = 0 ] || { echo "FAIL: a CertificateRequest was not auto-approved, or was not submitted by a controller"; exit 1; } + echo "CERT_RENEWED_UNATTENDED_OK (revision=$rev)" + check: + (contains($stdout, 'CERT_RENEWED_UNATTENDED_OK')): true + + - name: then-removing-the-certificate-withdraws-the-serving-material + description: >- + docs/08 reads revocation as "removing/rotating a certificate is a declarative + change". The declarative change is the deletion; withdrawal of the credential + must follow with no manual cleanup. EXPECTED TO FAIL TODAY — cert-manager needs + `enableCertificateOwnerRef: true` for the Secret to be garbage-collected, and the + tracked HelmRelease does not set it. Kept as the requirement reads (R-TEST-06). + try: + - delete: + ref: + apiVersion: cert-manager.io/v1 + kind: Certificate + name: conformance-lifecycle + - script: + content: | + set -u + for i in $(seq 1 15); do + if ! kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls >/dev/null 2>&1; then + echo "CERT_WITHDRAWN_OK (serving material gone after $((i * 4))s)" + exit 0 + fi + sleep 4 + done + owners="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \ + -o jsonpath='{.metadata.ownerReferences}' 2>/dev/null || true)" + echo "FAIL: the Certificate was deleted but Secret conformance-lifecycle-tls still" + echo " holds usable certificate material (ownerReferences=${owners:-none})." + echo " Removing a certificate therefore does NOT withdraw the credential, so" + echo " R-TLS-02's 'revoked ... without manual steps' does not hold. FIX:" + echo " set enableCertificateOwnerRef: true in platform/cert-manager values" + echo " (owner decision — WS-TEST does not weaken the assertion instead)." + exit 1 + check: + (contains($stdout, 'CERT_WITHDRAWN_OK')): true + finally: + # Namespace teardown removes both objects anyway; explicit so a leftover Secret + # from the expected failure above is never left holding key material. + - script: + content: | + kubectl -n "$NAMESPACE" delete secret conformance-lifecycle-tls --ignore-not-found --wait=false || true + kubectl -n "$NAMESPACE" delete certificate conformance-lifecycle --ignore-not-found --wait=false || true + kubectl -n "$NAMESPACE" delete issuer conformance-lifecycle --ignore-not-found --wait=false || true + rm -f /tmp/conformance-cert-r1.cksum || true