diff --git a/conformance/tests/l0-static/dev-nodes-provisioned-as-code/chainsaw-test.yaml b/conformance/tests/l0-static/dev-nodes-provisioned-as-code/chainsaw-test.yaml new file mode 100644 index 0000000..68f3af5 --- /dev/null +++ b/conformance/tests/l0-static/dev-nodes-provisioned-as-code/chainsaw-test.yaml @@ -0,0 +1,75 @@ +# ============================================================================= +# L0 STATIC — dev/test nodes are provisioned as code, end to end (R-ENV-01) [EXECUTABLE] +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# Spec: docs/00 R-ENV-01 ("dev/test MUST be provisioned as local KVMs via IaC: +# created, baselined, cluster-joined, and managed as code"); the four stages +# are enumerated in docs/02 §"Node lifecycle stages". +# +# WHY L0: R-ENV-01 is a requirement about HOW the dev nodes came to exist, and a live +# Node object cannot answer it — a node built by OpenTofu and one built by hand in a +# hypervisor UI are indistinguishable through the Kubernetes API. What is provable, with +# no cluster, is that a COMPLETE and EXECUTABLE provisioning path exists for all four +# stages and covers EVERY dev node. For the Ansible half that is proven by DRIVING the +# real toolchain offline (`--syntax-check`, `--list-hosts` — neither opens a connection +# to any node), not by reading files: a chain that does not resolve fails the check. +# +# ENVIRONMENT-AGNOSTIC: no hostname, address, prefix or node count appears in the test. +# The dev group names are read from the playbooks' own default host patterns, the node +# names are generated from the IaC module's prefix/count variables, and the two sets are +# compared against each other — so the assertion survives a rename or a resize. +# +# GIVEN the repository as git tracks it, +# WHEN the provisioning chain is parsed and (where offline execution is possible) driven, +# THEN the IaC module creates the dev nodes on a LOCAL KVM hypervisor (allowlisted +# provider — a cloud provider fails), from a for_each map rather than a count index, +# each with CPU/RAM/OS disk/dedicated Longhorn data disk/NIC/cloud-init seed; the +# node set it creates is EXACTLY the set Ansible manages; the baseline stage applies +# the package floor, kernel modules, sysctls, data-disk mount, swap-off and iscsid +# declaratively; one k3s role founds the cluster with --cluster-init and joins the +# rest with --server while disabling flannel/kube-proxy/traefik/servicelb at +# install; and site.yml's chain (baseline -> k3s -> Cilium -> verify -> Flux) +# resolves and targets every dev node at every stage. +# +# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git, PyYAML, and — +# where present — ansible-playbook in no-connection modes; no cluster, no network); +# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test +# wraps the SAME script. +# +# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them): +# * That the RUNNING dev VMs were created by this module — needs the OpenTofu state or +# hypervisor access, neither available offline. +# * `tofu validate` — needs `tofu init` (provider registry), i.e. network. The module is +# parsed, not evaluated; a network-allowed CI stage would close this. +# * "Never hand-edited on the box" (docs/02 stage 4's second half) — a live drift check. +# * REBUILD/re-converge (docs/02's own proof line) — exercised at L3 by +# tests/l3-fault-injection/node-loss (R-STATE-02). +# * docs/02 prose puts the baseline in cloud-init; the implementation moved it into the +# Ansible node_baseline role. R-ENV-01 requires the node to be "baselined ... as code", +# not that cloud-init be the tool, so the check asserts the STAGE and RECORDS the +# divergence rather than hiding it. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: dev-nodes-provisioned-as-code + labels: + conformance.home.k3s/R-ENV-01: "true" + annotations: + conformance.home.k3s/requirements: "R-ENV-01" + conformance.home.k3s/layer: "L0" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Given the tracked tree, when the dev provisioning chain is parsed and driven + offline, then every dev node is created by local-KVM IaC, baselined, cluster-joined + and handed to Flux by one code path that covers all of them — with no node outside + it and no stage missing. +spec: + steps: + - name: then-every-dev-node-is-created-baselined-joined-and-managed-as-code + description: >- + Run the offline assertion script over the tracked tree (same script ci/l0.sh and + conformance/run.sh --layer l0 execute). + try: + - script: + content: bash ./check.sh diff --git a/conformance/tests/l0-static/dev-nodes-provisioned-as-code/check.sh b/conformance/tests/l0-static/dev-nodes-provisioned-as-code/check.sh new file mode 100755 index 0000000..0f033ef --- /dev/null +++ b/conformance/tests/l0-static/dev-nodes-provisioned-as-code/check.sh @@ -0,0 +1,442 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 static check (offline, no cluster) — dev-nodes-provisioned-as-code +# Serves: R-ENV-01 ("dev/test MUST be provisioned as local KVMs via IaC: created, +# baselined, cluster-joined, and managed as code." — spec +# docs/00-vision-and-scope.md; the four stages are enumerated in +# docs/02-environments-and-provisioning.md §"Node lifecycle stages +# (R-ENV-01 — 'created, baselined, cluster-joined, managed as code')": +# 1. Create — OpenTofu instantiates the VM +# 2. Baseline — OS baseline and prerequisites +# 3. Cluster-join — Ansible installs/joins k3s with Cilium as CNI; disables +# k3s bundled flannel/traefik/servicelb/kube-proxy +# 4. Manage — GitOps-managed by Flux; node config changes are made via IaC +# and re-applied, never hand-edited on the box.) +# +# WHY L0 IS THE RIGHT LAYER, AND WHAT IT MEANS TO TEST THIS BEHAVIOURALLY +# ---------------------------------------------------------------------- +# R-ENV-01 is a requirement about HOW the dev nodes come to exist. The live cluster +# cannot answer it: a Kubernetes node object looks identical whether it was built by +# OpenTofu or by hand in a hypervisor UI (there is no API field for "was I created by +# IaC"). What CAN be proven, with no cluster, is that a COMPLETE, EXECUTABLE +# provisioning path exists for all four stages and covers every dev node — and, for the +# Ansible half, that is proven by RUNNING the real toolchain in a no-connection mode +# (`--syntax-check`, `--list-hosts`) rather than by reading the files, so a chain that +# does not actually resolve fails here. The IaC half is asserted by parsing the module: +# `tofu init/validate` needs the provider registry, and L0 is offline by contract. +# +# GIVEN the repository as git tracks it, +# WHEN the provisioning chain is parsed and (where offline execution is possible) +# actually driven, +# THEN CREATE: the IaC module targets a LOCAL KVM hypervisor (every declared provider +# is a KVM-family hypervisor provider — no cloud provider may appear), +# instantiates its nodes from a for_each map (never a count index, so one +# node can be replaced without disturbing its siblings), and gives each +# node CPU, RAM, an OS disk, a DEDICATED Longhorn data disk, a NIC and a +# cloud-init seed carrying the login user + SSH key; +# AND the node set it creates is EXACTLY the dev node set Ansible manages — +# names derived from the module's own prefix/count, so no node can be +# hand-added to the inventory or hand-created in the hypervisor; +# BASELINE: a baseline stage exists as declarative Ansible (package floor, kernel +# modules persisted AND loaded, sysctls, the Longhorn data disk formatted +# and mounted, swap off, iscsid enabled), driven by variables with +# defaults rather than per-host special cases; +# JOIN: one role installs k3s for both roles of every node — `--cluster-init` +# on the primary, `--server` join on the secondaries — with flannel, +# kube-proxy, traefik and servicelb disabled AT INSTALL, and Cilium +# installed as the CNI afterwards; +# MANAGE: site.yml chains baseline -> k3s -> Cilium -> verify -> Flux bootstrap; +# `ansible-playbook --syntax-check` on that chain SUCCEEDS; and +# `--list-hosts` shows every one of those stages targeting EVERY dev +# node, so no node is outside the code path. +# +# ENVIRONMENT-AGNOSTIC: no hostname, address, prefix or node count is written down here. +# The dev group names come from the playbooks' own default host patterns, the node names +# from the IaC module's prefix/count variables, and the two sets are compared to each +# other. This check therefore keeps working when the estate is renamed or resized. +# +# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them): +# - That the RUNNING dev VMs were in fact created by this module. Nothing offline can: +# it would need the OpenTofu state (a remote R2 backend + credentials) or hypervisor +# access. `tofu plan` against the live estate is the operator's own check. +# - That the module APPLIES cleanly. `tofu validate` needs `tofu init`, which downloads +# the provider from the registry; L0 is offline by contract, so the module is parsed, +# not evaluated. ci/lint.sh does not cover .tf either — a gap worth closing in CI +# with a network-allowed stage, not here. +# - "Never hand-edited on the box" (stage 4's second half) — a drift check against the +# live nodes, which is an operator/Ansible-check-mode exercise, not a repo property. +# - REBUILD/re-converge ("destroy a dev node VM and rebuild it", docs/02's own proof +# line) — that is exercised at L3 by tests/l3-fault-injection/node-loss (R-STATE-02). +# - That the baseline actually ran on a node (cluster/host state), and that the +# hypervisor is on the operator's own LAN rather than remote — "local" is asserted as +# "a KVM hypervisor provider driving a self-hosted endpoint variable", not as a +# network topology fact. +# - DEVIATION FROM docs/02 PROSE, ASSERTED DELIBERATELY: docs/02 describes stage 2 as +# cloud-init applying the OS baseline. The implementation moved the rich baseline into +# the Ansible `node_baseline` role (cloud-init keeps hostname/user/SSH/network), +# because Proxmox provisioning is API-only and a custom cloud-init snippet needs +# SSH/SFTP to the hypervisor. R-ENV-01's own wording requires the node to be +# "baselined ... as code", not that cloud-init be the tool, so this check asserts the +# STAGE, wherever it lives, and records the divergence rather than hiding it. +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +PYTHON="${PYTHON:-python3}" + +cd "${ROOT}" +"${PYTHON}" - "${ROOT}" <<'PY' +import re +import shutil +import subprocess +import sys +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover + print("check FAIL: PyYAML is required (same dependency as catalog/render.py)", + file=sys.stderr) + raise SystemExit(1) + +ROOT = Path(sys.argv[1]) +FAILURES: list[str] = [] +NOTES: list[str] = [] +CHECKS = 0 + + +def check(cond, msg): + global CHECKS + CHECKS += 1 + if not cond: + FAILURES.append(msg) + + +def note(msg): + NOTES.append(msg) + + +# A LOCAL DEPENDENCY the repo deliberately does not carry — the vault passphrase (kept +# outside the repo, R-SEC-01) or the Galaxy collections (gitignored; installed from +# collections/requirements.yml). Their absence is an ENVIRONMENT gap, not a repo defect, +# and is the ONLY tolerated reason the driven sub-assertions may not run. Deliberately +# narrow: every other non-zero exit is a real failure. The parse assertions always run. +ENV_ONLY_FAILURE = re.compile( + r"vault password|vault-id|Attempting to decrypt" + r"|No module named 'ansible_collections|couldn't resolve module/action", re.I) + +TOFU = ROOT / "infra" / "tofu" +ANS = ROOT / "infra" / "ansible" +check(TOFU.is_dir(), "infra/tofu/ is missing — the CREATE stage has no IaC at all (R-ENV-01)") +check(ANS.is_dir(), "infra/ansible/ is missing — the BASELINE/JOIN/MANAGE stages have no code") +if not TOFU.is_dir() or not ANS.is_dir(): + print(f"CHECK FAIL (dev-nodes-provisioned-as-code): {len(FAILURES)} of {CHECKS} failed", + file=sys.stderr) + for f in FAILURES: + print(f" - {f}", file=sys.stderr) + raise SystemExit(1) + +TF = "\n".join((p.read_text(errors="replace") for p in sorted(TOFU.glob("*.tf")))) +check(len(TF) > 500, "infra/tofu/*.tf is empty or a stub — nothing creates the dev nodes") + +# ============================================================ STAGE 1 — CREATE (IaC) +# A LOCAL KVM hypervisor, by allowlist: any provider outside this set fails, so moving +# dev onto a cloud (or anything that is not a KVM hypervisor the operator runs) is caught +# here rather than being noticed later. Each entry is justified, not just listed: +# bpg/proxmox Proxmox VE — a KVM/QEMU hypervisor the operator hosts. +# dmacvicar/libvirt libvirt/KVM directly (the pre-Proxmox substrate). +KVM_PROVIDERS = {"bpg/proxmox", "dmacvicar/libvirt"} +providers = dict(re.findall(r'source\s*=\s*"([^"]+)"', TF) and + [(m.group(1), m.group(2)) for m in + re.finditer(r'(\w[\w-]*)\s*=\s*\{[^}]*?source\s*=\s*"([^"]+)"', TF, re.S)]) +check(len(providers) > 0, + "infra/tofu declares no required_providers — the substrate is unpinned, so 'via IaC' " + "cannot be checked at all") +for alias, source in sorted(providers.items()): + check(source.lower() in KVM_PROVIDERS, + f"infra/tofu declares provider {alias}={source!r}, which is not a local KVM hypervisor " + f"provider (allowed: {sorted(KVM_PROVIDERS)}) — R-ENV-01 requires dev/test to be LOCAL KVMs") + +# The VM resource: for_each-keyed, and carrying the hardware + seed docs/02 enumerates. +vm_blocks = re.findall(r'resource\s+"([\w]*(?:_vm|_domain))"\s+"(\w+)"\s*\{(.*?)\n\}', + TF, re.S) +check(len(vm_blocks) >= 1, + "no VM/domain resource found in infra/tofu — nothing in IaC creates a dev node") +for rtype, rname, body in vm_blocks: + check("for_each" in body, + f'resource "{rtype}" "{rname}" is not for_each-keyed. A count index makes single-node ' + "replace unsafe (a -target'ed destroy cascaded to sibling nodes — DECISIONS-LOG F1), so " + "'created ... as code' would not survive a rebuild") + check(not re.search(r"^\s*count\s*=", body, re.M), + f'resource "{rtype}" "{rname}" uses a positional count index') + for want, why in ( + (r"\bcpu\b", "CPU"), + (r"\bmemory\b", "RAM"), + (r"network_device|network_interface", "a NIC"), + (r"initialization|cloud_init|cloudinit", "a cloud-init seed"), + ): + check(re.search(want, body) is not None, + f'resource "{rtype}" "{rname}" declares no {why} — docs/02 stage 1 requires CPU, RAM, ' + "disks, NICs and a cloud-init seed to come from IaC") + disks = len(re.findall(r"^\s*disk\s*\{", body, re.M)) + check(disks >= 2, + f'resource "{rtype}" "{rname}" declares {disks} disk block(s); docs/02 requires the OS ' + "disk PLUS a dedicated data disk for Longhorn") + check(re.search(r"user_account|ssh_authorized_key|ssh_keys", body) is not None, + f'resource "{rtype}" "{rname}" seeds no login user / SSH key — the node could not be ' + "reached by the BASELINE stage without a hand step") + +# Node identity comes from variables, and those variables define the whole node set. +def tf_default(var: str): + m = re.search(r'variable\s+"' + re.escape(var) + r'"\s*\{(.*?)\n\}', TF, re.S) + if not m: + return None + d = re.search(r'default\s*=\s*(.+)', m.group(1)) + return d.group(1).strip() if d else None + + +prefix_raw = tf_default("node_name_prefix") +count_raw = tf_default("node_count") +ips_raw = tf_default("node_ips") +check(prefix_raw is not None and count_raw is not None, + "infra/tofu has no node_name_prefix/node_count variables — the node set is not derived " + "from code, so the created nodes cannot be compared with the managed ones") +prefix = (prefix_raw or "").strip('"') +try: + node_count = int((count_raw or "0").strip()) +except ValueError: + node_count = 0 +check(node_count >= 1, f"node_count default is {count_raw!r} — no node would be created") +if ips_raw is not None: + n_ips = len(re.findall(r'"[^"]+"', ips_raw)) + check(n_ips >= node_count, + f"node_ips default has {n_ips} address(es) for node_count={node_count} — the module " + "cannot address every node it creates") +check((TOFU / "outputs.tf").is_file() and + re.search(r'output\s+"(ansible_inventory|nodes)"', (TOFU / "outputs.tf").read_text()) is not None, + "infra/tofu/outputs.tf surfaces no node inventory output — the JOIN stage would have to be " + "told the node facts by hand instead of consuming them from IaC") + +# ------------------------------------------------- the dev node set, derived twice over +# Group names are taken from the PLAYBOOKS' own default host patterns, so nothing here +# restates the estate's naming. +PLAYBOOKS = ANS / "playbooks" + + +def default_patterns(playbook: str) -> list[str]: + p = PLAYBOOKS / playbook + if not p.is_file(): + return [] + return re.findall(r"default\('([^']+)'\)", p.read_text()) + + +dev_groups = set(default_patterns("node-baseline.yml") + default_patterns("k3s.yml")) +check(len(dev_groups) > 0, + "no playbook declares a default host pattern — the dev target is not expressed in code") + +INV = ANS / "inventory" / "hosts.ini" +check(INV.is_file(), "infra/ansible/inventory/hosts.ini is missing — no node is managed as code") +ini_groups: dict[str, list[str]] = {} +children: dict[str, list[str]] = {} +cur = None +for line in (INV.read_text().splitlines() if INV.is_file() else []): + s = line.strip() + if not s or s.startswith("#"): + continue + m = re.match(r"^\[([^\]]+)\]$", s) + if m: + cur = m.group(1) + (children if cur.endswith(":children") else ini_groups).setdefault( + cur.split(":")[0] if cur.endswith(":children") else cur, []) + continue + if cur is None or cur.endswith(":vars"): + continue + host = s.split()[0] + if cur.endswith(":children"): + children.setdefault(cur.split(":")[0], []).append(host) + else: + ini_groups.setdefault(cur, []).append(host) + + +def hosts_in(group: str, seen=None) -> set[str]: + seen = seen or set() + if group in seen: + return set() + seen.add(group) + out = set(ini_groups.get(group, [])) + for child in children.get(group, []): + out |= hosts_in(child, seen) + return out + + +dev_hosts: set[str] = set() +for g in sorted(dev_groups): + dev_hosts |= hosts_in(g) +check(len(dev_hosts) > 0, + f"the dev groups {sorted(dev_groups)} resolve to NO hosts in hosts.ini — nothing is managed") +expected = {f"{prefix}{i + 1}" for i in range(node_count)} +check(dev_hosts == expected, + f"the dev nodes Ansible manages ({sorted(dev_hosts)}) are not exactly the nodes the IaC " + f"module creates ({sorted(expected)}) — a node outside IaC is a hand-built node (R-ENV-01)") + +# ============================================================ STAGE 2 — BASELINE (code) +BASE_ROLE = ANS / "roles" / "node_baseline" +check(BASE_ROLE.is_dir(), + "no node_baseline role — the BASELINE stage has no code (docs/02 stage 2)") +base_tasks = "\n".join(p.read_text(errors="replace") + for p in sorted((BASE_ROLE / "tasks").glob("*.yml"))) if BASE_ROLE.is_dir() else "" +base_defaults = "\n".join(p.read_text(errors="replace") + for p in sorted((BASE_ROLE / "defaults").glob("*.yml"))) if BASE_ROLE.is_dir() else "" +# Each item docs/02 names for stage 2, matched on the ANSIBLE MODULE that would do it — +# a declarative, idempotent task, not a shell one-liner. +for pattern, what in ( + (r"ansible\.builtin\.apt:", "a package baseline (apt module)"), + (r"modprobe:", "kernel modules loaded (modprobe module)"), + (r"modules-load|/etc/modules", "kernel modules PERSISTED across reboot"), + (r"sysctl", "sysctl tuning"), + (r"filesystem:|mkfs", "a filesystem on the Longhorn data disk"), + (r"mount:|ansible\.posix\.mount", "the Longhorn data disk MOUNTED"), + (r"swap", "swap disabled"), + (r"iscsid", "iscsid enabled (Longhorn prerequisite)"), +): + check(re.search(pattern, base_tasks) is not None, + f"node_baseline does not declare {what} — docs/02 stage 2 requires the OS baseline and " + "k3s/Longhorn/KubeVirt prerequisites to be applied as code") +check(re.search(r"^node_baseline_packages:", base_defaults, re.M) is not None and + re.search(r"^node_baseline_kernel_modules:", base_defaults, re.M) is not None, + "node_baseline's package/module sets are not variables with defaults — a per-host special " + "case is not 'managed as code' (and could not be scoped per environment, R-ENV-02)") + +# ========================================================== STAGE 3 — CLUSTER-JOIN (k3s) +K3S_ROLE = ANS / "roles" / "k3s_server" +k3s_text = "\n".join(p.read_text(errors="replace") + for p in sorted((K3S_ROLE).rglob("*.yml"))) if K3S_ROLE.is_dir() else "" +check(K3S_ROLE.is_dir() and len(k3s_text) > 200, + "no k3s_server role — nothing joins the nodes into a cluster (docs/02 stage 3)") +check("--cluster-init" in k3s_text, + "the k3s role never passes --cluster-init — the HA embedded-etcd cluster is never founded") +check(re.search(r"--server\s+https", k3s_text) is not None, + "the k3s role never joins secondaries with --server — nodes would not form one cluster") +check(re.search(r"K3S_TOKEN|--token", k3s_text) is not None, + "the k3s role never supplies a join token — the join could only be completed by hand") +# The bundled components docs/02 requires to be disabled AT INSTALL, each checked by the +# flag that actually disables it. +for pattern, what in ( + (r"--flannel-backend=none", "flannel (Cilium is the CNI)"), + (r"--disable-kube-proxy", "kube-proxy (Cilium kube-proxy replacement)"), + (r"traefik", "traefik (Gateway API via Cilium)"), + (r"servicelb", "servicelb"), +): + check(re.search(pattern, k3s_text) is not None, + f"the k3s install does not disable {what} — docs/02 stage 3 requires it disabled at " + "install, or the bundled component fights the platform's own") +check((ANS / "roles" / "cilium").is_dir(), + "no cilium role — stage 3 requires Cilium to be installed as the CNI at bootstrap") +check((ANS / "roles" / "flux_bootstrap").is_dir(), + "no flux_bootstrap role — stage 4's hand-off to GitOps has no code") + +# ============================================== STAGE 4 — MANAGE + the chain is REAL +SITE = PLAYBOOKS / "site.yml" +check(SITE.is_file(), "no playbooks/site.yml — the provisioning chain is not expressed as one path") +site_txt = SITE.read_text() if SITE.is_file() else "" +order = [m for m in re.findall(r"import_playbook:\s*(\S+)", site_txt)] +check(order[:1] == ["node-baseline.yml"], + f"site.yml imports {order} — the BASELINE stage must come first (docs/02 sequencing)") +for want in ("k3s.yml", "cilium.yml", "flux-bootstrap.yml"): + check(want in order, f"site.yml does not import {want} — a lifecycle stage is missing from the chain") +if "k3s.yml" in order and "cilium.yml" in order: + check(order.index("k3s.yml") < order.index("cilium.yml"), + "site.yml installs Cilium before k3s — the CNI cannot precede the cluster") + +# --- DRIVE THE REAL TOOLCHAIN, OFFLINE ------------------------------------------------ +# --syntax-check and --list-hosts open NO connection to any node: they resolve the +# playbook, roles, collections and inventory only. Running them is the difference between +# "the chain is written down" and "the chain resolves". +if shutil.which("ansible-playbook") is None: + note("ansible-playbook is not installed here (e.g. the in-cluster chainsaw image), so the " + "chain was PARSED but not DRIVEN; the merge gate's container has it and does drive it") +else: + # cwd=infra/ansible so the tracked ansible.cfg (roles_path, collections_path, inventory) + # is the one in force — the same resolution a real run gets. + def ansible(*args) -> subprocess.CompletedProcess: + return subprocess.run(["ansible-playbook", *args], cwd=ANS, + capture_output=True, text=True) + + syn = ansible("--syntax-check", "playbooks/site.yml") + syn_out = (syn.stderr or "") + (syn.stdout or "") + lh = None + if syn.returncode != 0 and ENV_ONLY_FAILURE.search(syn_out): + # NARROW, SELF-POLICING SKIP — see ENV_ONLY_FAILURE above. + note("ansible-playbook lacks a LOCAL dependency kept outside the repo by design (the vault " + "passphrase and/or the gitignored Galaxy collections), so --syntax-check/--list-hosts " + "were not driven; every parse assertion above still ran") + else: + check(syn.returncode == 0, + "`ansible-playbook --syntax-check playbooks/site.yml` FAILED — the provisioning chain " + f"does not even load: {syn_out.strip()[:400]}") + lh = ansible("--list-hosts", "playbooks/site.yml") + check(lh.returncode == 0, + "`ansible-playbook --list-hosts playbooks/site.yml` FAILED — the chain does not " + f"resolve against the tracked inventory: " + f"{((lh.stderr or '') + (lh.stdout or '')).strip()[:400]}") + if lh is not None and lh.returncode == 0: + # Parse plays -> hosts. NEVER `ansible-inventory --list`: it prints DECRYPTED vault values. + plays: list[tuple[str, set[str]]] = [] + name, hosts, in_hosts = None, set(), False + for line in lh.stdout.splitlines(): + m = re.match(r"\s*play #\d+ \([^)]*\):\s*(.*?)\s*TAGS:", line) + if m: + if name is not None: + plays.append((name, hosts)) + name, hosts, in_hosts = m.group(1), set(), False + continue + if re.match(r"\s*hosts \(\d+\):", line): + in_hosts = True + continue + if in_hosts and line.strip() and not line.strip().startswith("pattern:"): + hosts.add(line.strip()) + if name is not None: + plays.append((name, hosts)) + check(len(plays) >= 4, + f"--list-hosts resolved {len(plays)} play(s) — the four-stage chain did not expand") + # Every lifecycle stage must cover EVERY dev node. A stage that reaches two of three + # nodes means the third was finished by hand. + STAGES = { + "baseline": r"baseline", + "cluster-join": r"^k3s (primary|secondar)", + "cni": r"cilium", + "manage (flux hand-off)": r"flux", + } + for stage, rx in STAGES.items(): + covered: set[str] = set() + matched = 0 + for pname, phosts in plays: + if re.search(rx, pname, re.I): + matched += 1 + covered |= phosts + check(matched >= 1, f"no play in site.yml implements the {stage} stage") + check(covered == dev_hosts, + f"the {stage} stage targets {sorted(covered)}, not every dev node " + f"{sorted(dev_hosts)} — a node outside the code path is a hand-provisioned " + "node (R-ENV-01)") + +# --------------------------------------------------------------------------- verdict +# NO-OP GUARD: a check that quietly asserts nothing is worse than no check. +check(CHECKS >= 30, + f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass " + "would be meaningless") + +if FAILURES: + print(f"CHECK FAIL (dev-nodes-provisioned-as-code): {len(FAILURES)} of {CHECKS} assertions failed", + file=sys.stderr) + for f in FAILURES: + print(f" - {f}", file=sys.stderr) + raise SystemExit(1) +for n in NOTES: + print(f" note: {n}") +print(f"check OK: dev-nodes-provisioned-as-code — {CHECKS} assertions; " + f"{len(dev_hosts)} dev node(s) {sorted(dev_hosts)} created by IaC ({sorted(providers.values())}), " + f"baselined, cluster-joined and Flux-managed by the same chain. R-ENV-01's authored-path " + "clause only — the live VMs' provenance, `tofu validate`, and rebuild/re-converge are NOT " + "proven here (see the header's gap list; rebuild is L3 node-loss).") +PY diff --git a/conformance/tests/l0-static/prod-baremetal-not-dev-surface/chainsaw-test.yaml b/conformance/tests/l0-static/prod-baremetal-not-dev-surface/chainsaw-test.yaml new file mode 100644 index 0000000..532aa7e --- /dev/null +++ b/conformance/tests/l0-static/prod-baremetal-not-dev-surface/chainsaw-test.yaml @@ -0,0 +1,77 @@ +# ============================================================================= +# L0 STATIC — prod is bare metal, and is not a development surface (R-ENV-03) [EXECUTABLE] +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# Spec: docs/00 R-ENV-03 ("prod is a 3× mini-PC bare-metal k3s cluster with local +# storage; it MUST NOT be an active development surface until the roadmap +# declares it ready"); docs/02 (prod = 3 physical nodes, local SSD/NVMe to +# Longhorn, "the *same* Ansible bootstrap runs; only the machine-creation step +# differs", "Never develop against prod"); roadmap/phases.md Phase 7. +# +# HOW A "MUST NOT" IS MADE TESTABLE — the same discipline as R-TEST-06's test: a +# governance rule is verified through the property that ENFORCES it, never by observing +# that nobody misbehaved today. Here that property is structural: prod receives the base +# configuration plus environment-scoped overrides and nothing else, from the same branch +# and the same Ansible code as dev, so there is no place to put prod-only work at all. +# +# ENVIRONMENT-AGNOSTIC: the prod inventory group is DERIVED (it is the group whose +# group_vars retargets the k3s roles away from the global default), the prod overlay is +# derived from CLUSTER_ENV, and the storage/replica expectations are compared against the +# node count that inventory resolves to. No hostname, address or environment name is +# written into the test. +# +# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted), +# WHEN the prod inventory, overlay, IaC module and readiness record are inspected — and +# the shared provisioning chain is actually retargeted at prod with --list-hosts, +# THEN prod resolves to exactly three nodes reached directly on their own sshd; no IaC +# resource creates any of them (bare metal is imaged, not instantiated); prod's +# Longhorn path is node-local with one replica per node and the default +# StorageClass is the local driver; prod runs the same playbooks/roles as dev, +# retargeted only by variables the shared code already defines; the prod overlay +# carries only the base plus environment-scoped patches on the same Git ref dev +# tracks, with no prod-only manifest anywhere in the shared configuration; and prod +# artifacts exist only alongside the operator's recorded opening of the prod phase. +# +# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git, PyYAML, and — +# where present — ansible-playbook in --list-hosts mode, which opens no connection); +# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test +# wraps the SAME script. +# +# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them): +# * THE HARDWARE. That the three machines are mini-PCs, are bare metal, and hold local +# SSD/NVMe is operator-attested (infra/PROD-STANDUP.md, PROD-PROVISIONING.md); the +# node-health/BIOS surveys that establish it write to gitignored artifacts, and the +# Kubernetes API exposes no "am I virtual" fact, so nothing here asserts it. +# * The LIVE prod cluster's shape (three Ready nodes, all control-plane+etcd, Longhorn +# disks at the local path) — cluster state, belonging to R-PLAT-01's still-uncovered +# HA test; asserting it here would also make this test environment-specific, since +# the same suite must pass against dev. +# * Git HISTORY: this is the working tree at HEAD. +# * Whether the roadmap SHOULD have opened prod — only that prod work is accompanied by +# the recorded declaration. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: prod-baremetal-not-dev-surface + labels: + conformance.home.k3s/R-ENV-03: "true" + annotations: + conformance.home.k3s/requirements: "R-ENV-03" + conformance.home.k3s/layer: "L0" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Given the tracked tree, when prod's inventory, overlay, IaC and readiness record are + inspected, then prod is authored as three bare-metal nodes with node-local storage, + bootstrapped by dev's own code retargeted by variables, receiving only + environment-scoped overrides on the same branch — so there is nowhere to develop + against prod — and it exists only because the roadmap phase was opened. +spec: + steps: + - name: then-prod-is-bare-metal-and-carries-no-development-of-its-own + description: >- + Run the offline assertion script over the tracked tree (same script ci/l0.sh and + conformance/run.sh --layer l0 execute). + try: + - script: + content: bash ./check.sh diff --git a/conformance/tests/l0-static/prod-baremetal-not-dev-surface/check.sh b/conformance/tests/l0-static/prod-baremetal-not-dev-surface/check.sh new file mode 100755 index 0000000..1fd041a --- /dev/null +++ b/conformance/tests/l0-static/prod-baremetal-not-dev-surface/check.sh @@ -0,0 +1,528 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 static check (offline, no cluster) — prod-baremetal-not-dev-surface +# Serves: R-ENV-03 ("prod is a 3× mini-PC bare-metal k3s cluster with local storage; it +# MUST NOT be an active development surface until the roadmap declares it ready." +# — spec docs/00-vision-and-scope.md. Detail in docs/02: prod substrate "3× mini-PC +# bare metal", "3 physical nodes", storage "local SSD/NVMe presented to Longhorn", +# "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation +# step differs", and "**Never develop against prod.** Prod hardware is a target, not +# a sandbox (R-ENV-03)". The roadmap gate is roadmap/phases.md Phase 7: entry +# "operator declares prod hardware ready"; note "Do not begin building on prod +# hardware before this phase is explicitly opened".) +# +# HOW A "MUST NOT" IS MADE TESTABLE +# -------------------------------- +# The second clause is a governance rule, and the same discipline applies as to R-TEST-06 +# (tests/l0-static/spec-immutability-enforced): "nobody developed against prod today" +# cannot be observed, but the PROPERTY THAT MAKES DEVELOPING AGAINST PROD IMPOSSIBLE can +# be. In this repo that property is structural — prod receives the base configuration plus +# environment-scoped overrides and NOTHING ELSE, so there is no place to put prod-only work +# even if someone wanted to. This check asserts that, together with the first clause's +# authored facts (three nodes, no VM IaC behind them, node-local storage, the same +# bootstrap code as dev). +# +# The first clause's HARDWARE facts (that the three machines really are mini-PCs, really +# are bare metal, really hold local NVMe) are operator-attested, not machine-checkable — +# see the gap list. What IS checkable is that the repository describes prod that way and +# that no IaC creates prod nodes as virtual machines. +# +# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted), +# WHEN the prod inventory, the prod overlay, the IaC module and the roadmap record are +# inspected — and, where offline execution is possible, the provisioning chain is +# actually retargeted at prod with `--list-hosts`, +# THEN (1) prod resolves to EXACTLY THREE nodes, reached directly on their own sshd +# (no NAT indirection, i.e. real machines on the LAN); +# (2) NO IaC resource creates any prod node: the VM module's generated node names are +# disjoint from the prod host set and no prod hostname appears anywhere under +# infra/tofu — prod is imaged by the operator (docs/02 stage 1), not instantiated; +# (3) prod's storage is NODE-LOCAL: the Longhorn data path for prod is an absolute +# local path, and the replica count equals the node count (one replica per node's +# own disk). A distinct bulk/media tier is explicitly permitted (R-STATE-04) and +# is not counted against this; +# (4) prod runs THE SAME bootstrap code as dev: the same playbooks/roles, retargeted +# by variables only — every prod group_vars key is an existing role/global +# variable (an invented prod-only knob would be a fork), and no prod-only role +# or playbook exists; +# (5) prod is NOT a development surface: clusters/prod carries only the base plus +# sync.yaml, every patch it applies targets an ENVIRONMENT-SCOPED object +# (cluster-vars, a Flux Kustomization's sops decryption, or the Git source), +# prod's Git ref is the SAME ref dev tracks (no prod-only branch to develop on), +# and no prod-only manifest exists anywhere in the rendered configuration; +# (6) prod artifacts exist only WITH a declaration of readiness: the spec roadmap +# carries the phase that opens prod build (checked when the spec repo is mounted), +# and the repository records that the operator opened it. +# +# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them): +# - The HARDWARE. That the three nodes are mini-PCs, are bare metal, and present local +# SSD/NVMe is operator-attested in infra/PROD-STANDUP.md / PROD-PROVISIONING.md; the +# node-health and BIOS surveys that establish it write to gitignored artifacts, so +# nothing tracked can be asserted against. Cluster-side, the Kubernetes API exposes no +# "am I virtual" fact either — a bare-metal check would need a privileged host-path +# probe of DMI data, which is a heavier and more invasive test than the claim warrants. +# - The LIVE prod cluster's shape (three Ready nodes, all control-plane+etcd, Longhorn +# disks at the local path). That is cluster state and belongs to R-PLAT-01's +# (currently uncovered) HA test; asserting it here would also make this test +# environment-specific, since the same suite must pass against dev. +# - That prod's HISTORY contains no development. This is the working tree at HEAD. +# - Whether the roadmap SHOULD have opened prod (the entry-gate judgement). Only that +# prod work is accompanied by the operator's recorded declaration. +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +PYTHON="${PYTHON:-python3}" +: "${SPEC_REPO:=${ROOT}/../home-network-k3s}" +export SPEC_REPO + +cd "${ROOT}" +"${PYTHON}" - "${ROOT}" <<'PY' +import os +import re +import shutil +import subprocess +import sys +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover + print("check FAIL: PyYAML is required (same dependency as catalog/render.py)", + file=sys.stderr) + raise SystemExit(1) + +ROOT = Path(sys.argv[1]) +FAILURES: list[str] = [] +NOTES: list[str] = [] +CHECKS = 0 + + +def check(cond, msg): + global CHECKS + CHECKS += 1 + if not cond: + FAILURES.append(msg) + + +def note(msg): + NOTES.append(msg) + + +# A LOCAL DEPENDENCY the repo deliberately does not carry — the vault passphrase (kept +# outside the repo, R-SEC-01) or the Galaxy collections (gitignored; installed from +# collections/requirements.yml). Their absence is an ENVIRONMENT gap, not a repo defect, +# and is the ONLY tolerated reason the driven sub-assertions may not run. Deliberately +# narrow: every other non-zero exit is a real failure. The parse assertions always run. +ENV_ONLY_FAILURE = re.compile( + r"vault password|vault-id|Attempting to decrypt" + r"|No module named 'ansible_collections|couldn't resolve module/action", re.I) + +ANS = ROOT / "infra" / "ansible" +INV = ANS / "inventory" / "hosts.ini" +GV = ANS / "inventory" / "group_vars" +TOFU = ROOT / "infra" / "tofu" +check(INV.is_file(), "infra/ansible/inventory/hosts.ini is missing — prod is not described anywhere") +if not INV.is_file(): + print("CHECK FAIL (prod-baremetal-not-dev-surface): no inventory", file=sys.stderr) + raise SystemExit(1) + +# --------------------------------------------------------------- inventory parsing +ini_groups: dict[str, list[dict]] = {} +children: dict[str, list[str]] = {} +cur = None +for line in INV.read_text().splitlines(): + s = line.strip() + if not s or s.startswith("#"): + continue + m = re.match(r"^\[([^\]]+)\]$", s) + if m: + cur = m.group(1) + continue + if cur is None or cur.endswith(":vars"): + continue + if cur.endswith(":children"): + children.setdefault(cur.split(":")[0], []).append(s.split()[0]) + continue + parts = s.split() + host = {"name": parts[0]} + for kv in parts[1:]: + if "=" in kv: + k, v = kv.split("=", 1) + host[k] = v + ini_groups.setdefault(cur, []).append(host) + + +def hosts_in(group: str, seen=None) -> list[dict]: + seen = seen or set() + if group in seen: + return [] + seen.add(group) + out = list(ini_groups.get(group, [])) + for child in children.get(group, []): + out += hosts_in(child, seen) + return out + + +def load_gv(group: str) -> dict: + d: dict = {} + for p in [GV / f"{group}.yml", GV / group / "vars.yml"]: + if p.is_file(): + try: + d.update(yaml.safe_load(p.read_text()) or {}) + except Exception: + pass + return d + + +# WHICH GROUP IS PROD? Derived, never named here: it is the group whose group_vars +# RETARGETS the k3s roles (k3s_primary_group / k3s_servers_group) away from the global +# dev default. That is exactly the seam docs/02 + infra/ansible/README describe for prod. +ALL_VARS = load_gv("all") +dev_primary_group = str(ALL_VARS.get("k3s_primary_group", "")) +candidates = [] +for entry in sorted(GV.iterdir()) if GV.is_dir() else []: + g = entry.name[:-4] if entry.name.endswith(".yml") else entry.name + if g == "all": + continue + v = load_gv(g) + if v.get("k3s_primary_group") and str(v["k3s_primary_group"]) != dev_primary_group: + candidates.append((g, v)) +check(len(candidates) == 1, + f"expected exactly ONE environment whose group_vars retargets the k3s roles (prod); found " + f"{[c[0] for c in candidates]} — R-ENV-03 describes a single prod estate reached through " + "the same roles as dev") +if not candidates: + print("CHECK FAIL (prod-baremetal-not-dev-surface): no prod group_vars found", file=sys.stderr) + for f in FAILURES: + print(f" - {f}", file=sys.stderr) + raise SystemExit(1) +PROD_GROUP, PROD_VARS = candidates[0] +PROD_PRIMARY = str(PROD_VARS["k3s_primary_group"]) +prod_hosts = hosts_in(PROD_GROUP) +prod_names = sorted(h["name"] for h in prod_hosts) + +# ------------------------------------------------- (1) exactly three, directly reachable +check(len(prod_names) == 3, + f"prod group {PROD_GROUP!r} resolves to {len(prod_names)} host(s) {prod_names} — R-ENV-03 " + "says prod is a 3× mini-PC cluster") +dev_hosts = hosts_in(str(ALL_VARS.get("k3s_servers_group", "k3s_servers"))) +for h in prod_hosts: + # Real machines on the LAN answer their own sshd. Dev's hosts are reached through a + # NAT port-forward (ansible_port 225x -> a VM's :22), which is precisely the + # indirection a hypervisor-hosted estate needs and bare metal does not. + check(str(h.get("ansible_port", "22")) == "22", + f"prod node {h['name']} is reached on port {h.get('ansible_port')} — a forwarded port " + "implies an indirection (NAT to a virtual estate), not a machine on the LAN") + check(h.get("k3s_node_ip") == h.get("ansible_host"), + f"prod node {h['name']} has ansible_host={h.get('ansible_host')} but " + f"k3s_node_ip={h.get('k3s_node_ip')} — a split address is the dev NAT pattern; a " + "bare-metal node's SSH address IS its cluster address") + +# ------------------------------------------------------ (2) nothing creates prod as a VM +TF = "\n".join(p.read_text(errors="replace") for p in sorted(TOFU.glob("*.tf"))) \ + if TOFU.is_dir() else "" +check(len(TF) > 500, "infra/tofu/*.tf is empty — cannot show that it does NOT create prod") + + +def tf_default(var: str): + m = re.search(r'variable\s+"' + re.escape(var) + r'"\s*\{(.*?)\n\}', TF, re.S) + if not m: + return None + d = re.search(r"default\s*=\s*(.+)", m.group(1)) + return d.group(1).strip() if d else None + + +prefix = (tf_default("node_name_prefix") or "").strip('"') +try: + node_count = int((tf_default("node_count") or "0").strip()) +except ValueError: + node_count = 0 +iac_names = {f"{prefix}{i + 1}" for i in range(node_count)} +check(len(iac_names) > 0, "the IaC module generates no node names — cannot compare with prod") +check(iac_names.isdisjoint(set(prod_names)), + f"the IaC VM module would create {sorted(iac_names & set(prod_names))}, which are PROD nodes " + "— prod is bare metal that the operator images (docs/02 stage 1), never instantiated by the " + "hypervisor module") +for name in prod_names: + check(name not in TF, + f"prod node {name} is named inside infra/tofu — prod hardware must not be described by " + "the VM module at all") +# The prod host addresses must not appear as IaC node addresses either (a renamed VM is +# still a VM). +tf_ips = set(re.findall(r'"(\d+\.\d+\.\d+\.\d+)"', tf_default("node_ips") or "")) +prod_ips = {str(h.get("ansible_host")) for h in prod_hosts} +check(tf_ips.isdisjoint(prod_ips), + f"the IaC module addresses {sorted(tf_ips & prod_ips)}, which are prod node addresses") + +# --------------------------------------------------------------- (3) node-local storage +# Every environment's value for a cluster-vars knob (base default + each overlay patch). +def env_values(key: str) -> dict[str, str]: + vals: dict[str, str] = {} + base = ROOT / "clusters" / "base" / "cluster-vars.yaml" + if base.is_file(): + for d in yaml.safe_load_all(base.read_text()): + if isinstance(d, dict) and d.get("kind") == "ConfigMap" \ + and (d.get("metadata") or {}).get("name") == "cluster-vars": + v = (d.get("data") or {}).get(key) + if v is not None: + vals["base"] = str(v) + for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")): + env = overlay.parent.name + if env == "base": + continue + try: + k = yaml.safe_load(overlay.read_text()) or {} + except Exception: + continue + for p in (k.get("patches") or []): + tgt = p.get("target") or {} + if tgt.get("kind") != "ConfigMap" or tgt.get("name") != "cluster-vars": + continue + try: + ops = yaml.safe_load(p.get("patch") or "") or [] + except Exception: + continue + for op in ops if isinstance(ops, list) else []: + if isinstance(op, dict) and str(op.get("path", "")) == f"/data/{key}": + vals[env] = str(op.get("value")) + return vals + + +# The prod OVERLAY is the one whose CLUSTER_ENV is not the base default. +cluster_envs = env_values("CLUSTER_ENV") +prod_overlays = [e for e, v in cluster_envs.items() + if e != "base" and v != cluster_envs.get("base")] +check(len(prod_overlays) >= 1, + "no clusters/ overlay sets its own CLUSTER_ENV — prod has no overlay of its own") +prod_overlay = None +for e in prod_overlays: + if PROD_GROUP.replace("k3s_", "") in cluster_envs[e] or cluster_envs[e] in PROD_GROUP: + prod_overlay = e +check(prod_overlay is not None, + f"no overlay's CLUSTER_ENV corresponds to the prod inventory group {PROD_GROUP!r} — the " + "cluster config and the machine inventory disagree about which estate is prod") + +data_paths = env_values("LONGHORN_DATA_PATH") +replicas = env_values("LONGHORN_DEFAULT_REPLICA_COUNT") +if prod_overlay: + p = data_paths.get(prod_overlay, data_paths.get("base", "")) + check(p.startswith("/"), + f"prod's LONGHORN_DATA_PATH is {p!r} — R-ENV-03's 'local storage' requires an absolute " + "node-local filesystem path") + check(not re.match(r"^/(net|nfs|mnt/nfs)|:", p), + f"prod's LONGHORN_DATA_PATH {p!r} looks like a network mount, not local SSD/NVMe") + r = replicas.get(prod_overlay, replicas.get("base", "")) + check(r.isdigit() and int(r) == len(prod_names), + f"prod's LONGHORN_DEFAULT_REPLICA_COUNT is {r!r} for {len(prod_names)} node(s) — " + "'local storage' on a 3-node estate means one replica per node's own disk") +# The DEFAULT storage class must be the local (Longhorn) driver. A separate bulk/media tier +# on network storage is explicitly a different requirement (R-STATE-04, docs/04) and is not +# counted here. +default_scs = [] +for f in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z", "*.yaml", "*.yml"], + capture_output=True, text=True, check=True).stdout.split("\0"): + if not f: + continue + try: + for d in yaml.safe_load_all((ROOT / f).read_text(errors="replace")): + if isinstance(d, dict) and d.get("kind") == "StorageClass" and \ + str(((d.get("metadata") or {}).get("annotations") or {}) + .get("storageclass.kubernetes.io/is-default-class", "")) == "true": + default_scs.append((f, str(d.get("provisioner", "")))) + except Exception: + continue +check(len(default_scs) == 1, + f"expected exactly one default StorageClass, found {default_scs} — the tier prod's workloads " + "land on by default must be unambiguous") +for f, prov in default_scs: + check(prov == "driver.longhorn.io", + f"{f}: the default StorageClass provisioner is {prov!r}, not Longhorn — prod's default " + "tier would not be its own local disks") + +# ------------------------------------- (4) prod runs the SAME bootstrap code, retargeted +PLAYBOOKS = ANS / "playbooks" +ROLES = ANS / "roles" +prod_token = PROD_GROUP.replace("k3s_", "") +for d, what in ((ROLES, "role"), (PLAYBOOKS, "playbook")): + for entry in sorted(d.iterdir()) if d.is_dir() else []: + check(prod_token not in entry.name.lower(), + f"infra/ansible/{d.name}/{entry.name} is a prod-specific {what} — docs/02 requires " + "the SAME Ansible bootstrap for prod, differing only in the machine-creation step") +# Every prod group_vars key must be a variable the shared code already knows: a role default, +# a global, or an ansible built-in. An invented key is a fork of the role by another name. +known: set[str] = set(ALL_VARS) +for rd in sorted(ROLES.iterdir()) if ROLES.is_dir() else []: + for p in (rd / "defaults").glob("*.yml") if (rd / "defaults").is_dir() else []: + try: + known |= set((yaml.safe_load(p.read_text()) or {}).keys()) + except Exception: + pass +check(len(known) > 20, f"only {len(known)} shared variables found — (4) would prove nothing") +for k in sorted(PROD_VARS): + check(k.startswith("ansible_") or k in known, + f"prod group_vars sets {k!r}, which is not a role default or a global — prod-only " + "configuration that the dev path never exercises is developing against prod") +# ...and it must retarget, not re-implement: the group it names must exist in the inventory. +check(PROD_PRIMARY in ini_groups or PROD_PRIMARY in children, + f"prod group_vars names primary group {PROD_PRIMARY!r}, which the inventory does not define") + +# ---------------------------------------- (5) prod is not a development surface (overlay) +if prod_overlay: + odir = ROOT / "clusters" / prod_overlay + files = sorted(p.name for p in odir.iterdir() if p.is_file()) + check(files == ["kustomization.yaml", "sync.yaml"], + f"clusters/{prod_overlay}/ contains {files} — a prod overlay may hold only its " + "kustomization and its own root sync; anything else is configuration authored FOR PROD") + k = yaml.safe_load((odir / "kustomization.yaml").read_text()) or {} + check(sorted(str(r) for r in (k.get("resources") or [])) == ["../base", "sync.yaml"], + f"clusters/{prod_overlay} resources are {k.get('resources')} — prod must render the base " + "plus its own sync only (docs/02: 'apply the prod overlay (no base edits)')") + # Every patch must be an ENVIRONMENT-SCOPED override, never a change to what runs. + ALLOWED = {("ConfigMap", "cluster-vars"), ("GitRepository", None)} + for p in (k.get("patches") or []): + tgt = p.get("target") or {} + body = p.get("patch") or "" + kind = tgt.get("kind") + if kind is None: # inline patch: kind comes from the body + try: + doc = yaml.safe_load(body) or {} + except Exception: + doc = {} + kind = doc.get("kind") + name = (doc.get("metadata") or {}).get("name") + if kind == "Kustomization": + # Flux Kustomization patches are allowed ONLY to wire sops decryption — + # anything else (path, interval, images, values) changes what prod RUNS. + keys = set((doc.get("spec") or {}).keys()) + check(keys <= {"decryption"}, + f"clusters/{prod_overlay}: patch of Kustomization/{name} touches {sorted(keys)} " + "— only `decryption` (the per-cluster sops key) is an environment-scoped " + "override; changing anything else is developing against prod") + continue + check((kind, None) in ALLOWED or (kind, name) in ALLOWED, + f"clusters/{prod_overlay}: patch targets {kind}/{name}, which is not an " + "environment-scoped object — prod must not carry configuration of its own") + else: + check((kind, tgt.get("name")) in ALLOWED or (kind, None) in ALLOWED, + f"clusters/{prod_overlay}: patch targets {kind}/{tgt.get('name')}, which is not " + "an environment-scoped object — prod must not carry configuration of its own") + + # Same Git ref as the other environment(s): a prod-only branch is a development line + # that only prod runs, which is exactly what R-ENV-03 forbids. + def git_ref(overlay: str): + kk = yaml.safe_load((ROOT / "clusters" / overlay / "kustomization.yaml").read_text()) or {} + for p in (kk.get("patches") or []): + body = p.get("patch") or "" + try: + doc = yaml.safe_load(body) or {} + except Exception: + continue + if doc.get("kind") == "GitRepository": + return (doc.get("spec") or {}).get("ref") + return None + + others = [o.name for o in (ROOT / "clusters").iterdir() + if o.is_dir() and o.name not in ("base", prod_overlay)] + prod_ref = git_ref(prod_overlay) + check(prod_ref is not None, + f"clusters/{prod_overlay} does not pin a Git ref — prod would follow whatever the base " + "points at, with no record of what it runs") + for o in others: + check(git_ref(o) == prod_ref, + f"clusters/{o} tracks {git_ref(o)} but prod tracks {prod_ref} — a prod-only ref is a " + "development line only prod runs (R-ENV-03: prod is a target, not a sandbox)") + +# No prod-only manifest anywhere in the rendered configuration. +tracked = [f for f in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"], + capture_output=True, text=True, check=True + ).stdout.split("\0") if f] +RENDERED_DIRS = ("platform/", "platform-config", "apps/", "catalog/", "clusters/base/") +for f in tracked: + if f.startswith(RENDERED_DIRS) and re.search(rf"(^|[/_.-]){re.escape(prod_token)}([/_.-]|$)", + f, re.I): + check(False, + f"{f} is a prod-specific file inside the shared configuration — every environment " + "renders these paths, so prod-only content here is development against prod") +check(len(tracked) > 50, f"git ls-files returned only {len(tracked)} paths — not a real checkout") + +# --------------------------- (6) prod work exists only with a declaration of readiness +# In-repo: the operator's recorded decision to open the prod phase. +log = ROOT / "DECISIONS-LOG.md" +check(log.is_file() and re.search(r"PHASE\s*7\b.*OPEN", log.read_text(errors="replace"), re.I), + "DECISIONS-LOG.md carries no record of the operator OPENING the prod phase, yet prod " + "inventory/overlay exist — R-ENV-03 allows prod build only once the roadmap declares it ready") +# In the spec (read-only, may not be mounted): the roadmap phase that opens prod. +spec = Path(os.environ.get("SPEC_REPO", "")) +phases = spec / "roadmap" / "phases.md" +if phases.is_file(): + txt = phases.read_text(errors="replace") + check(re.search(r"##\s*Phase\s*7\b", txt) is not None and + re.search(r"prod", txt, re.I) is not None, + "the spec roadmap has no Phase 7 opening prod build — the declaration R-ENV-03 defers " + "to does not exist") + check(re.search(r"explicitly opened|declares prod hardware ready|operator declares", txt, re.I) + is not None, + "the spec roadmap's prod phase states no explicit entry declaration — the gate R-ENV-03 " + "names would be unenforceable") +else: + note(f"spec repo not mounted at {spec} — the roadmap half of (6) was NOT checked " + "(the in-repo declaration still was)") + +# --------------------------- (4b) DRIVE the retarget offline: same chain, prod hosts +if shutil.which("ansible-playbook") is None: + note("ansible-playbook is not installed here (e.g. the in-cluster chainsaw image), so the " + "prod retarget was PARSED but not DRIVEN; the merge gate's container has it") +else: + sec_groups = [g for g in children.get(PROD_GROUP, []) if g != PROD_PRIMARY] + args = ["--list-hosts", "playbooks/site.yml", + "-e", f"k3s_hosts={PROD_GROUP}", + "-e", f"k3s_primary_hosts={PROD_PRIMARY}"] + if sec_groups: + args += ["-e", f"k3s_secondary_hosts={sec_groups[0]}"] + lh = subprocess.run(["ansible-playbook", *args], cwd=ANS, capture_output=True, text=True) + out = (lh.stderr or "") + (lh.stdout or "") + if lh.returncode != 0 and ENV_ONLY_FAILURE.search(out): + # See the same narrow, self-policing skip in dev-nodes-provisioned-as-code. + note("ansible-playbook lacks a LOCAL dependency kept outside the repo by design (the vault " + "passphrase and/or the gitignored Galaxy collections), so the prod retarget was not " + "driven; every parse assertion above still ran") + else: + check(lh.returncode == 0, + f"retargeting the SHARED chain at prod failed: {out.strip()[:400]} — docs/02 requires " + "the same Ansible bootstrap to run for prod") + seen: set[str] = set() + in_hosts = False + for line in lh.stdout.splitlines(): + if re.match(r"\s*hosts \(\d+\):", line): + in_hosts = True + continue + if re.search(r"play #\d+", line): + in_hosts = False + continue + if in_hosts and line.strip(): + seen.add(line.strip()) + check(seen == set(prod_names), + f"the shared chain retargeted at prod resolves to {sorted(seen)}, not the three prod " + f"nodes {prod_names} — prod would be bootstrapped by something other than dev's path") + +# --------------------------------------------------------------------------- verdict +check(CHECKS >= 25, + f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass " + "would be meaningless") + +if FAILURES: + print(f"CHECK FAIL (prod-baremetal-not-dev-surface): {len(FAILURES)} of {CHECKS} assertions failed", + file=sys.stderr) + for f in FAILURES: + print(f" - {f}", file=sys.stderr) + raise SystemExit(1) +for n in NOTES: + print(f" note: {n}") +print(f"check OK: prod-baremetal-not-dev-surface — {CHECKS} assertions; prod group {PROD_GROUP!r} = " + f"{prod_names} (3 nodes, none created by IaC), overlay clusters/{prod_overlay} renders the base " + f"plus environment-scoped patches only, local Longhorn path " + f"{data_paths.get(prod_overlay, data_paths.get('base'))!r} with " + f"{replicas.get(prod_overlay, replicas.get('base'))} replica(s). R-ENV-03's AUTHORED clauses " + "only — the hardware facts (mini-PC, bare metal, local NVMe) are operator-attested and the " + "live cluster's shape is NOT asserted here (see the header's gap list).") +PY diff --git a/conformance/tests/l0-static/tls-lifecycle-automated/chainsaw-test.yaml b/conformance/tests/l0-static/tls-lifecycle-automated/chainsaw-test.yaml new file mode 100644 index 0000000..f137268 --- /dev/null +++ b/conformance/tests/l0-static/tls-lifecycle-automated/chainsaw-test.yaml @@ -0,0 +1,69 @@ +# ============================================================================= +# L0 STATIC — certificate lifecycle needs no manual step (R-TLS-02) [EXECUTABLE] +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# Spec: docs/00 R-TLS-02 ("Certificates MUST be automatically issued, renewed, and +# revoked without manual steps"); mechanism in docs/08 + ADR-0006. +# +# R-TLS-02 is one adverbial clause plus three runtime verbs. This test asserts the +# clause — "without manual steps" — which in a GitOps platform is a property of the +# REPOSITORY: a human step in the certificate path leaves evidence in git (committed +# cert material, a Gateway serving a Secret nothing declares, an issuer with no +# automated solver, a credential never delivered, auto-approval switched off, or CRs +# nothing reconciles). The three verbs are observed by the companion L2 test. +# +# ENVIRONMENT-AGNOSTIC (R-ENV-02): the issuer a Certificate names is `${ACME_CLUSTER_ISSUER}`, +# so the check resolves it against the BASE cluster-vars default AND every overlay's value and +# requires each resolved issuer to exist. dev's issuer and prod's issuer are both verified from +# one authored assertion, with no hostname, address or environment name written down. +# +# GIVEN the repository as git tracks it, +# WHEN every serving-TLS reference, Certificate, Issuer and the cert-manager release +# is inspected offline, +# THEN every Gateway listener that terminates TLS serves a Secret a tracked Certificate +# produces; no certificate material is committed anywhere; every Certificate's +# issuer resolves — in EVERY environment — to a tracked ACME issuer with an +# automated dns01/http01 solver whose credential is SOPS-delivered from git; +# cert-manager's automatic request approval is not disabled; and every one of +# those objects is rendered by a Flux Kustomization, so nothing in the chain +# depends on a human running kubectl. +# +# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git + PyYAML, no +# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and +# this Chainsaw Test wraps the SAME script. +# +# KNOWN, DELIBERATE GAPS (named in check.sh's header too — this test claims NONE of them): +# * ISSUED / RENEWED as runtime events: L2 (l2-behavioral/cert-lifecycle-automated). +# * REVOKED: not proven at any layer today. cert-manager performs no ACME revocation, +# and the spec's own reading (docs/08: "removing/rotating a certificate is a +# declarative change") needs `enableCertificateOwnerRef`, which the tracked +# cert-manager HelmRelease does not set — so a removed Certificate leaves its +# serving Secret behind. The L2 test asserts the withdrawal behaviour and is +# EXPECTED TO FAIL until that knob is set; escalated, not softened (R-TEST-06). +# * That the live cluster is serving the tracked Certificate (cluster state). +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: tls-lifecycle-automated + labels: + conformance.home.k3s/R-TLS-02: "true" + annotations: + conformance.home.k3s/requirements: "R-TLS-02" + conformance.home.k3s/layer: "L0" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Given the tracked tree, when the certificate path is inspected offline, then no + manual step exists anywhere in it: serving Secrets are cert-manager-produced, no + cert material is committed, every environment's issuer is a tracked ACME issuer + with an automated solver and a git-delivered credential, auto-approval is on, and + the whole chain is Flux-reconciled. +spec: + steps: + - name: then-no-manual-step-exists-in-the-certificate-path + description: >- + Run the offline assertion script over the tracked tree (same script ci/l0.sh + and conformance/run.sh --layer l0 execute). + try: + - script: + content: bash ./check.sh diff --git a/conformance/tests/l0-static/tls-lifecycle-automated/check.sh b/conformance/tests/l0-static/tls-lifecycle-automated/check.sh new file mode 100755 index 0000000..294b4b1 --- /dev/null +++ b/conformance/tests/l0-static/tls-lifecycle-automated/check.sh @@ -0,0 +1,478 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 static check (offline, no cluster) — tls-lifecycle-automated +# Serves: R-TLS-02 ("Certificates MUST be automatically **issued, renewed, and revoked** +# without manual steps." — spec docs/00-vision-and-scope.md; mechanism in +# docs/08-tls-and-certificates.md, ADR-0006.) +# +# WHICH CLAUSE THIS LAYER CAN PROVE — AND WHICH IT CANNOT +# ------------------------------------------------------ +# R-TLS-02 has one adverbial clause ("without manual steps") and three verbs +# ("issued", "renewed", "revoked"). In a GitOps platform the *without manual steps* +# clause IS a repository property, and it is the half that can be proven with no +# cluster: if a human had to mint, install, or approve a serving certificate, the +# evidence of that human step would be IN THE REPO — a committed `tls.crt`, a Gateway +# pointing at a Secret nothing declares, an issuer that no automated solver can +# satisfy, a per-cert credential handed over out of band, or an admission path that +# waits for a manual CertificateRequest approval. This check asserts that none of +# those exist and that the automated path is completely wired instead. +# +# The three VERBS are runtime events. They are asserted by the companion L2 test +# (conformance/tests/l2-behavioral/cert-lifecycle-automated), which observes an +# issuance, an unattended renewal, and the withdrawal of a removed certificate. This +# check does NOT claim them; see "WHAT THIS DOES *NOT* PROVE" below. +# +# GIVEN the repository as git tracks it, +# WHEN every serving-TLS reference, Certificate, Issuer and the cert-manager release +# are inspected offline, +# THEN (1) at least one cert-manager `Certificate` is authored (vacuity floor: a repo +# with no Certificate cannot pass this check by having nothing to check); +# (2) every Gateway HTTPS listener that TERMINATES TLS references ONLY Secrets +# that a tracked Certificate declares as its `secretName` — so the serving +# credential is produced by the controller, never dropped in by hand; +# (3) NO certificate material is tracked anywhere: no Secret manifest of type +# kubernetes.io/tls, no tls.crt/tls.key/ca.crt key in any committed Secret +# (plain OR SOPS-encrypted), and no PEM certificate body in any tracked +# YAML — i.e. issuance is not, and cannot silently become, a human step; +# (4) every Certificate's `issuerRef` resolves to a TRACKED issuer IN EVERY +# ENVIRONMENT — the `${ACME_CLUSTER_ISSUER}` token is resolved against the +# base cluster-vars default AND every overlay's value, so an environment +# whose issuer does not exist is caught here rather than at prod bring-up; +# (5) every issuer so referenced is an ACME issuer with a NON-EMPTY solver list +# whose every solver is an automated challenge type (dns01 / http01) — the +# same code path serves issuance AND renewal, so a solver a human must +# complete would defeat both; +# (6) each dns01 solver's provider credential is referenced BY NAME and that +# Secret is delivered SOPS-encrypted from git via a kustomization — the +# operator's one-time token hand-off (docs/08 OQ-4), never a per-cert step; +# (7) cert-manager's automatic CertificateRequest approval is NOT disabled +# (`disableAutoApproval`), and any `approveSignerNames` narrowing still +# covers the ClusterIssuers actually used — a request that waits for a +# human approver is the definition of a manual step; +# (8) the whole chain is reconciled from git — cert-manager's HelmRelease, the +# ClusterIssuers and the Certificate are each included in a kustomization +# that a Flux Kustomization renders — so a deleted Certificate is +# re-created and re-issued with no human action; +# (9) no Certificate defeats renewal arithmetically (`renewBefore` >= `duration` +# would mean the renewal instant never arrives). +# +# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them): +# - ISSUED: that an issuance actually happens. Asserted at L2 (step 2 of +# l2-behavioral/cert-lifecycle-automated) and, for the real ACME/DNS-01 path, +# evidenced by l2-behavioral/trusted-tls serving a publicly-trusted cert. +# - RENEWED: that a renewal actually happens unattended. docs/08's acceptance gate +# is explicit that this must be "observed across a renewal" — L2 step 3 does that +# by fast-forwarding a certificate's renewal window; nothing static can. +# - REVOKED: NOT PROVEN AT ANY LAYER TODAY, and not asserted here. Two distinct +# readings, neither established: (a) ACME revocation — cert-manager does not +# revoke certificates at all, so no repo state could show it; (b) the spec's own +# reading in docs/08 ("removing/rotating a certificate is a declarative change") +# — withdrawal of a REMOVED certificate's serving material requires cert-manager's +# `enableCertificateOwnerRef`, which the tracked HelmRelease does NOT set (chart +# default false). L2 step 4 asserts the withdrawal behaviour and is expected to +# FAIL until that knob is set; escalated to the PM rather than softened (R-TEST-06). +# - That the tracked Certificate is the one the live Gateway is actually serving +# (cluster state), or that the ACME account/DNS credential is still valid. +# - Certificate EXPIRY ALERTING, which docs/08 delegates to R-OBS-04 (see +# l3-fault-injection/alerting). +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +PYTHON="${PYTHON:-python3}" + +cd "${ROOT}" +"${PYTHON}" - "${ROOT}" <<'PY' +import re +import subprocess +import sys +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover + print("check FAIL: PyYAML is required (same dependency as catalog/render.py)", + file=sys.stderr) + raise SystemExit(1) + +ROOT = Path(sys.argv[1]) +FAILURES: list[str] = [] +CHECKS = 0 + + +def check(cond, msg): + global CHECKS + CHECKS += 1 + if not cond: + FAILURES.append(msg) + + +def tracked() -> list[str]: + out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"], + capture_output=True, text=True, check=True).stdout + return [p for p in out.split("\0") if p] + + +FILES = tracked() +check(len(FILES) > 50, f"git ls-files returned only {len(FILES)} paths — not a real checkout") +YAML_FILES = [f for f in FILES if re.search(r"\.(ya?ml)$", f)] + + +def docs_of(rel: str): + """Parsed YAML documents of a tracked file, or [] when it is not parseable YAML.""" + try: + return [d for d in yaml.safe_load_all((ROOT / rel).read_text(errors="replace")) + if isinstance(d, dict)] + except Exception: + return [] + + +DOCS: list[tuple[str, dict]] = [(rel, d) for rel in YAML_FILES for d in docs_of(rel)] + + +def of_kind(kind: str, group_prefix: str = "") -> list[tuple[str, dict]]: + out = [] + for rel, d in DOCS: + if d.get("kind") != kind: + continue + if group_prefix and not str(d.get("apiVersion", "")).startswith(group_prefix): + continue + out.append((rel, d)) + return out + + +def ns_of(d: dict, default: str = "default") -> str: + return (d.get("metadata") or {}).get("namespace") or default + + +def name_of(d: dict) -> str: + return (d.get("metadata") or {}).get("name") or "" + + +# ------------------------------------------------------------ cluster-vars resolution +# Every environment's value for a ${TOKEN}: the base default plus each overlay's JSON6902 +# `- op: replace / path: /data/KEY / value: ...` patch on the cluster-vars ConfigMap. This +# is the same mechanism flux-postbuild-substitution audits; here we need the VALUES, so an +# issuer name can be resolved per environment instead of being asserted for dev only. +def env_values(key: str) -> dict[str, str]: + vals: dict[str, str] = {} + base = ROOT / "clusters" / "base" / "cluster-vars.yaml" + if base.is_file(): + for d in yaml.safe_load_all(base.read_text()): + if isinstance(d, dict) and d.get("kind") == "ConfigMap" \ + and name_of(d) == "cluster-vars": + v = (d.get("data") or {}).get(key) + if v is not None: + vals["base"] = str(v) + for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")): + env = overlay.parent.name + if env == "base": + continue + try: + k = yaml.safe_load(overlay.read_text()) or {} + except Exception: + continue + for p in (k.get("patches") or []): + tgt = p.get("target") or {} + if tgt.get("kind") != "ConfigMap" or tgt.get("name") != "cluster-vars": + continue + try: + ops = yaml.safe_load(p.get("patch") or "") or [] + except Exception: + continue + for op in ops if isinstance(ops, list) else []: + if isinstance(op, dict) and str(op.get("path", "")) == f"/data/{key}": + vals[env] = str(op.get("value")) + return vals + + +TOKEN_RE = re.compile(r"^\$\{([A-Z0-9_]+)\}$") + + +def resolve_per_env(value: str) -> dict[str, str]: + """A literal resolves to itself in every environment; a ${TOKEN} to each env's value.""" + m = TOKEN_RE.match(value.strip()) + if not m: + return {"(literal)": value.strip()} + return env_values(m.group(1)) + + +# --------------------------------------------- what Flux actually renders (reachability) +# A manifest is DELIVERED only if it is reachable from a Flux Kustomization's `path` through +# the kustomize resource graph: /kustomization.yaml -> resources -> (file | directory +# -> that directory's kustomization.yaml -> ...). Membership is deliberately computed by +# walking that graph rather than by asking "is this file's DIRECTORY mentioned somewhere": +# the loose form silently passes when an entry is deleted from the very kustomization.yaml +# that the directory reference renders (verified — it made two assertions here no-ops). +KUST_RESOURCES: dict[str, list[str]] = {} +for rel in [f for f in YAML_FILES if f.endswith("kustomization.yaml")]: + refs: list[str] = [] + for d in docs_of(rel): + for r in (d.get("resources") or []) + (d.get("components") or []): + if not isinstance(r, str) or "://" in r: # skip remote bases + continue + try: + p = (ROOT / rel).parent.joinpath(r).resolve() + refs.append(str(p.relative_to(ROOT))) + except Exception: + continue + KUST_RESOURCES[rel] = refs + +FLUX_PATHS: set[str] = set() +for _rel, _d in DOCS: + if _d.get("kind") == "Kustomization" and \ + str(_d.get("apiVersion", "")).startswith("kustomize.toolkit.fluxcd.io/"): + p = str((_d.get("spec") or {}).get("path", "")).lstrip("./").rstrip("/") + if p: + FLUX_PATHS.add(p) + + +def _walk(entry: str, seen: set[str], out: set[str]) -> None: + if entry in seen: + return + seen.add(entry) + kust = entry if entry.endswith("kustomization.yaml") else f"{entry}/kustomization.yaml" + if kust in KUST_RESOURCES: # a directory (or a kustomization) + for r in KUST_RESOURCES[kust]: + _walk(r, seen, out) + return + out.add(entry) # a leaf manifest + + +RENDERED: set[str] = set() +for _p in FLUX_PATHS: + _walk(_p, set(), RENDERED) + + +def delivered(rel: str) -> bool: + """True iff Flux renders this file from one of its Kustomization paths.""" + return rel in RENDERED + + +# --------------------------------------------------- (1) there is something to check +CERTS = of_kind("Certificate", "cert-manager.io/") +check(len(CERTS) >= 1, + "no cert-manager.io Certificate is authored anywhere in the tracked tree — every " + "assertion below would be vacuous, so this is a FAILURE, not a pass (R-TLS-02)") + +ISSUERS = of_kind("ClusterIssuer", "cert-manager.io/") + of_kind("Issuer", "cert-manager.io/") +check(len(ISSUERS) >= 1, + "no cert-manager ClusterIssuer/Issuer is authored — nothing could issue automatically") + +# ---------------------------- (2) every terminating listener serves a controller cert +CERT_SECRETS: dict[tuple[str, str], str] = { # (namespace, secretName) -> file + (ns_of(d, "default"), str((d.get("spec") or {}).get("secretName", ""))): rel + for rel, d in CERTS +} +listeners_checked = 0 +for rel, gw in of_kind("Gateway", "gateway.networking.k8s.io/"): + gw_ns = ns_of(gw) + for lis in ((gw.get("spec") or {}).get("listeners") or []): + tls = lis.get("tls") or {} + if str(tls.get("mode", "")) != "Terminate": + continue + listeners_checked += 1 + refs = tls.get("certificateRefs") or [] + check(len(refs) > 0, + f"{rel}: Gateway {gw_ns}/{name_of(gw)} listener {lis.get('name')!r} terminates " + "TLS with NO certificateRefs — it can only serve a certificate someone puts " + "there by hand") + for r in refs: + kind = str(r.get("kind", "Secret")) + key = (str(r.get("namespace") or gw_ns), str(r.get("name"))) + check(kind == "Secret", + f"{rel}: listener {lis.get('name')!r} certificateRef kind is {kind!r}; only " + "Secret refs can be traced to a cert-manager Certificate") + check(key in CERT_SECRETS, + f"{rel}: listener {lis.get('name')!r} serves Secret {key[0]}/{key[1]}, which " + "NO tracked cert-manager Certificate declares as its secretName — that " + "credential would have to be created by hand (R-TLS-02 'without manual steps')") +check(listeners_checked > 0, + "no Gateway listener with tls.mode=Terminate found in the tracked tree — there is no " + "serving-TLS reference to trace, so (2) proved nothing") + +# --------------------------------- (3) no certificate material is tracked, anywhere +TLS_KEYS = {"tls.crt", "tls.key", "ca.crt", "tls-combined.pem"} +for rel, d in DOCS: + if d.get("kind") != "Secret": + continue + stype = str(d.get("type", "")) + check(stype != "kubernetes.io/tls", + f"{rel}: a Secret of type kubernetes.io/tls is COMMITTED (metadata.name=" + f"{name_of(d)}) — a serving certificate authored by a human, not issued") + for field in ("data", "stringData"): + for k in (d.get(field) or {}): + check(str(k) not in TLS_KEYS, + f"{rel}: committed Secret {name_of(d)} carries {field}.{k} — certificate " + "material in git means a manual issuance step exists (R-TLS-02)") + +PEM_RE = re.compile(r"-----BEGIN (?:[A-Z ]+ )?CERTIFICATE-----") +for rel in YAML_FILES: + body = (ROOT / rel).read_text(errors="replace") + # Scoped to YAML manifests deliberately: prose/docs may legitimately SHOW a PEM header + # in an example, and a .md file is not applied to any cluster. A manifest cannot. + check(not PEM_RE.search(body), + f"{rel}: contains a PEM CERTIFICATE body — a hand-minted certificate in a manifest") + +# --------------------------- (4)+(5)+(6) the issuer chain is automated in EVERY env +# The cert-manager release's namespace IS the controller's `--cluster-resource-namespace` +# (chart default), i.e. where a ClusterIssuer's solver credential must live. Derived from the +# HelmRelease, never written down here, so this holds if the release namespace ever changes. +cm_releases = [(rel, d) for rel, d in of_kind("HelmRelease") if name_of(d) == "cert-manager"] +CM_NS = ns_of(cm_releases[0][1], "cert-manager") if cm_releases else "cert-manager" + +ISSUER_INDEX = {(d.get("kind"), name_of(d)): (rel, d) for rel, d in ISSUERS} +solver_creds: list[tuple[str, str, str, str]] = [] # (issuer, ns, secretName, ref field) +issuers_verified: set[str] = set() +for rel, cert in CERTS: + spec = cert.get("spec") or {} + ref = spec.get("issuerRef") or {} + kind = str(ref.get("kind", "Issuer")) + per_env = resolve_per_env(str(ref.get("name", ""))) + check(len(per_env) > 0, + f"{rel}: Certificate {name_of(cert)} issuerRef.name is a token with NO value in " + "clusters/base/cluster-vars.yaml or any overlay — it would render empty and never issue") + for env, iname in sorted(per_env.items()): + hit = ISSUER_INDEX.get((kind, iname)) + check(hit is not None, + f"{rel}: Certificate {name_of(cert)} resolves issuerRef to {kind}/{iname} in " + f"environment {env!r}, but no such issuer is tracked — that environment cannot " + "issue anything without a hand-created issuer (R-TLS-02, R-ENV-02)") + if hit is None: + continue + irel, issuer = hit + issuers_verified.add(iname) + ispec = issuer.get("spec") or {} + acme = ispec.get("acme") + check(isinstance(acme, dict), + f"{irel}: {kind}/{iname} (used by {name_of(cert)} in env {env!r}) is not an ACME " + f"issuer — its issuance/renewal path is {sorted(ispec)}; docs/08/ADR-0006 mandate " + "ACME so both are unattended") + if not isinstance(acme, dict): + continue + check(str(acme.get("server", "")).startswith("https://"), + f"{irel}: {kind}/{iname} has no https ACME directory endpoint") + check(bool((acme.get("privateKeySecretRef") or {}).get("name")), + f"{irel}: {kind}/{iname} declares no privateKeySecretRef — the ACME account key " + "would have to be provisioned by hand") + solvers = acme.get("solvers") or [] + check(len(solvers) > 0, + f"{irel}: {kind}/{iname} declares NO solvers — every order would sit unsolved " + "until a human completed the challenge") + for s in solvers: + types = [t for t in ("dns01", "http01") if t in s] + check(len(types) == 1, + f"{irel}: {kind}/{iname} has a solver with challenge types {sorted(s)} — only " + "the automated dns01/http01 solvers satisfy 'without manual steps'") + for provider, cfg in (s.get("dns01") or {}).items(): + if not isinstance(cfg, dict): + continue + for field, val in cfg.items(): + if isinstance(val, dict) and str(field).endswith(("SecretRef", "secretRef")): + n = str(val.get("name", "")) + check(bool(n), + f"{irel}: {kind}/{iname} dns01 {provider}.{field} has no secret name") + if n: + # A ClusterIssuer resolves secret refs in the controller's + # cluster-resource-namespace; a namespaced Issuer in its own. + want_ns = CM_NS if kind == "ClusterIssuer" else ns_of(issuer, CM_NS) + solver_creds.append((iname, want_ns, n, f"{provider}.{field}")) + +check(len(issuers_verified) > 0, "no issuer was reachable from any Certificate — (5) proved nothing") + +# (6) the solver credential is delivered, SOPS-encrypted, from git — never per-cert by hand. +check(len(solver_creds) > 0, + "no dns01 solver credential reference found — docs/08 (OQ-4) requires the scoped " + "Cloudflare token to drive DNS-01, so this check would otherwise be vacuous") +# Keyed by (namespace, name): a same-named Secret in ANOTHER namespace is NOT this credential. +# (Verified: keying by name alone let the external-dns copy of `cloudflare-api-token` satisfy +# the cert-manager reference, so deleting cert-manager's own copy passed — a silent no-op.) +SECRET_DOCS = {(ns_of(d, CM_NS), name_of(d)): (rel, d) + for rel, d in DOCS if d.get("kind") == "Secret"} +for iname, sns, sname, field in solver_creds: + hit = SECRET_DOCS.get((sns, sname)) + check(hit is not None, + f"{iname}: dns01 {field} references Secret {sname!r}, which no tracked manifest " + f"delivers into namespace {sns!r} (the issuer's secret-resolution namespace) — the " + "credential would be created by hand in-cluster (R-TLS-02)") + if hit is None: + continue + srel, sdoc = hit + check("sops" in sdoc, + f"{srel}: Secret {sname} is the DNS-01 credential but is not SOPS-encrypted (R-SEC-01)") + check(delivered(srel), + f"{srel}: Secret {sname} is not rendered by any Flux Kustomization path — Flux would " + "never deliver it, so issuance would wait on a manual kubectl apply") + +# ------------------------------- (7) automatic CertificateRequest approval is intact +check(len(cm_releases) == 1, + f"expected exactly one cert-manager HelmRelease, found {[r for r, _ in cm_releases]} — " + "the controller that issues and renews must itself be GitOps-managed") +for rel, hr in cm_releases: + values = ((hr.get("spec") or {}).get("values") or {}) + check(values.get("disableAutoApproval") in (None, False), + f"{rel}: cert-manager values set disableAutoApproval={values.get('disableAutoApproval')!r} " + "— every CertificateRequest would then wait for an approver, which is a manual step") + signers = values.get("approveSignerNames") + if signers is not None: + pats = [str(s) for s in (signers if isinstance(signers, list) else [signers])] + for iname in sorted(issuers_verified): + ok = any(p in ("*", "*/*") or "clusterissuers.cert-manager.io" in p and + (p.endswith("/*") or p.endswith(f"/{iname}")) for p in pats) + check(ok, + f"{rel}: approveSignerNames={pats} does not cover ClusterIssuer {iname} — " + "requests for it would never be auto-approved") + # The chart's own CRD install must be on, else the CRs below cannot even be applied. + check((values.get("crds") or {}).get("enabled") is True, + f"{rel}: cert-manager values do not enable CRD installation — the Certificate CRD " + "would have to be installed by hand") + +# ------------------------------- (8) the whole chain is continuously reconciled from git +check(len(FLUX_PATHS) > 0 and len(RENDERED) > 20, + f"the kustomize resource graph resolved {len(RENDERED)} manifest(s) from {len(FLUX_PATHS)} " + "Flux Kustomization path(s) — too few to be a real graph, so (8) would prove nothing") +for rel, _ in cm_releases + CERTS + ISSUERS: + check(delivered(rel), + f"{rel} is not rendered by any Flux Kustomization path — nothing reconciles it, so a " + "deleted Certificate/issuer/controller would stay deleted until a human re-applied it") + +# --------------------------------------------- (9) renewal is not defeated arithmetically +def to_seconds(v) -> int | None: + m = re.fullmatch(r"\s*(\d+)\s*([smh])\s*", str(v)) + if not m: + return None + return int(m.group(1)) * {"s": 1, "m": 60, "h": 3600}[m.group(2)] + + +renew_pairs = 0 +for rel, cert in CERTS: + spec = cert.get("spec") or {} + dur, rb = to_seconds(spec.get("duration")), to_seconds(spec.get("renewBefore")) + if dur is None or rb is None: + continue + renew_pairs += 1 + check(rb < dur, + f"{rel}: Certificate {name_of(cert)} sets renewBefore={spec.get('renewBefore')} >= " + f"duration={spec.get('duration')} — the renewal instant never arrives") + +# --------------------------------------------------------------------------- verdict +# NO-OP GUARD: this repo has been bitten by checks that degrade to a successful no-op. +# The floor below is a count of assertions that MUST have run for the verdict to mean +# anything; it is not a style rule. +check(CHECKS >= 20, + f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass " + "would be meaningless") + +if FAILURES: + print(f"CHECK FAIL (tls-lifecycle-automated): {len(FAILURES)} of {CHECKS} assertions failed", + file=sys.stderr) + for f in FAILURES: + print(f" - {f}", file=sys.stderr) + raise SystemExit(1) +print(f"check OK: tls-lifecycle-automated — {CHECKS} assertions; {len(CERTS)} Certificate(s), " + f"{len(ISSUERS)} issuer(s), {len(issuers_verified)} issuer(s) reached from a Certificate " + f"across every environment, {listeners_checked} terminating listener(s), " + f"{len(solver_creds)} dns01 credential ref(s), {renew_pairs} duration/renewBefore pair(s). " + "R-TLS-02 'without manual steps' clause only — issued/renewed/revoked are RUNTIME verbs " + "(see l2-behavioral/cert-lifecycle-automated; 'revoked' is NOT proven at any layer).") +PY diff --git a/conformance/tests/l2-behavioral/cert-lifecycle-automated/chainsaw-test.yaml b/conformance/tests/l2-behavioral/cert-lifecycle-automated/chainsaw-test.yaml new file mode 100644 index 0000000..14fca5e --- /dev/null +++ b/conformance/tests/l2-behavioral/cert-lifecycle-automated/chainsaw-test.yaml @@ -0,0 +1,391 @@ +# ============================================================================= +# L2 BEHAVIORAL — certificate lifecycle is automatic (R-TLS-02) +# [RUNNABLE / NOT YET EXECUTED — see STATUS] +# Catalog: docs/14 "Trusted TLS (R-TLS-01/02)" +# Spec: docs/00 R-TLS-02 — "Certificates MUST be automatically **issued, renewed, +# and revoked** without manual steps." +# docs/08 §"Renewal & revocation (R-TLS-02)" — "cert-manager renews well +# before expiry automatically; no human action." / "Revocation: removing/ +# rotating a certificate is a declarative change; cert-manager handles +# issuance of replacements." +# docs/08 acceptance gate — "Certificates renew automatically ahead of expiry +# with no manual step (**observed across a renewal**)." +# Impl: platform/cert-manager/**, platform/gateway/wildcard-certificate.yaml. +# Sibling: tests/l0-static/tls-lifecycle-automated proves the "without manual steps" +# clause statically (in the merge gate). This test observes the three VERBS. +# +# ENVIRONMENT-AGNOSTIC (R-ENV-02): NOTHING about either environment is written down. The +# serving certificate is DISCOVERED — find the Gateway whose listener terminates TLS, follow +# its certificateRefs Secret, and read that Secret's `cert-manager.io/certificate-name` +# annotation to reach the Certificate. No hostname, no namespace list, no issuer name, no +# address: dev proves dev's cert and prod proves prod's from the same authored assertions. +# +# GIVEN cert-manager reconciled by Flux and a Gateway terminating TLS, +# WHEN the serving Certificate's provenance and renewal schedule are read, and a NEW +# Certificate is then requested, left to renew itself, and removed, +# THEN 1. the serving credential was produced by the CONTROLLER, not a human: its +# Secret carries cert-manager's certificate-name annotation, its +# CertificateRequest was submitted by cert-manager's own ServiceAccount and +# was APPROVED automatically, and its issuer is a Ready ACME ClusterIssuer; +# 2. an unattended RENEWAL IS SCHEDULED: status.renewalTime is in the future and +# strictly before status.notAfter, so no human is in the renewal path; +# 3. ISSUED — a newly declared Certificate becomes Ready and its Secret appears +# with certificate material, with no step between declaring and having it; +# 4. RENEWED — that certificate, given a renewal window that opens one minute +# after issuance, REISSUES ITSELF at least twice (status.revision >= 3) and +# the serving bytes actually change, with no human action. This is docs/08's +# "observed across a renewal", fast-forwarded; +# 5. REMOVED/REVOKED — deleting the Certificate withdraws its serving material. +# +# WHY A HERMETIC SELF-SIGNED ISSUER FOR STEPS 3-5, AND WHAT THAT COSTS +# ------------------------------------------------------------------- +# Steps 3-5 need a certificate lifecycle that completes in minutes and repeats on every +# run. Driving that through the real ACME issuer would fire a public Let's Encrypt order +# (and a Cloudflare _acme-challenge write) per conformance run, burning issuance rate +# limits for evidence that is not about ACME: cert-manager's renewal scheduler and +# ownership logic are issuer-independent. So steps 3-5 use a `selfSigned` Issuer created +# in this test's OWN namespace, and the ACME/DNS-01 path is proven elsewhere — step 1-2 +# read the REAL wildcard certificate the Gateway serves, and l2-behavioral/trusted-tls +# proves a publicly-trusted LE certificate is actually served (R-TLS-01/03). +# STATED PLAINLY: steps 3-5 do NOT prove the ACME order path renews; they prove the +# CONTROLLER's issue/renew/withdraw automation runs unattended in this cluster, while +# steps 1-2 prove the real serving certificate is on exactly that automated path. +# +# STATUS: RUNNABLE, NOT YET EXECUTED. Steps 1-2 are read-only. Steps 3-5 CREATE an +# Issuer/Certificate (and DELETE the Certificate they created) inside chainsaw's +# per-test namespace — the same class of mutation every other L2 test here performs, +# self-cleaning via namespace teardown, touching no platform object. WS-TEST authored +# them against live dev/prod read-only inspection under an explicit instruction not to +# mutate either cluster, so THE MUTATING STEPS HAVE NOT BEEN RUN. Nothing about the +# suite gates on them: ci/conformance.sh only runs L2/L3 when CLUSTER_AVAILABLE=true. +# +# EXPECTED FAILURE, DELIBERATELY NOT SOFTENED (R-TEST-06) +# ------------------------------------------------------ +# Step 5 WILL FAIL as the platform is authored today. cert-manager only garbage-collects +# a Certificate's Secret when the controller runs with `enableCertificateOwnerRef: true`; +# platform/cert-manager/helmrelease.yaml does not set it (chart default false), and the +# live wildcard Secret on both clusters has NO ownerReferences (verified read-only, +# 2026-09-05). So a removed Certificate leaves its credential in place, and "revoked +# without manual steps" does not hold. That is a finding about the platform, not a +# defect in this test: the assertion stays as the requirement reads and is escalated to +# the PM. (ACME revocation proper is not implemented by cert-manager at all.) +# +# KNOWN, DELIBERATE GAPS (this test claims none of these): +# * Not proven: that the CA would refuse a revoked certificate — there is no +# revocation request to make, so nothing observable exists to assert. +# * Not proven: renewal of the REAL wildcard cert (its window opens 30 days before a +# 90-day expiry; a conformance run cannot wait, and forcing it would rotate the +# platform's serving credential). Step 2 asserts the schedule; step 4 asserts the +# mechanism that will act on it. +# * Not proven: certificate expiry ALERTING — docs/08 delegates that to R-OBS-04 +# (l3-fault-injection/alerting). +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: cert-lifecycle-automated + labels: + conformance.home.k3s/R-TLS-02: "true" + annotations: + conformance.home.k3s/requirements: "R-TLS-02" + conformance.home.k3s/layer: "L2" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Given cert-manager and a TLS-terminating Gateway, when the serving certificate's + provenance and renewal schedule are read and a new certificate is requested, left + to renew, and removed, then issuance, renewal and withdrawal all happen with no + human step — the serving cert was controller-issued and auto-approved, its + renewal is scheduled ahead of expiry, and a declared certificate issues and + reissues itself unattended. +spec: + timeouts: + apply: 60s + assert: 3m + exec: 10m + steps: + # ---------------------------------------------------------------- READ-ONLY HALF + - name: given-cert-manager-is-reconciled-and-healthy + description: >- + The controller that issues and renews must be running, and must have been + installed by Flux rather than by hand — otherwise "automatic" has no engine. + try: + - script: + content: | + set -u + # Discover cert-manager's Deployments by chart label, in whatever namespace + # the release lives (nothing about the environment is assumed). + sel='app.kubernetes.io/instance=cert-manager' + rows="$(kubectl get deploy -A -l "$sel" \ + -o custom-columns='NS:.metadata.namespace,N:.metadata.name,RDY:.status.readyReplicas' \ + --no-headers 2>/dev/null || true)" + echo "$rows" + [ -n "$rows" ] || { echo "FAIL: no cert-manager Deployment found ($sel)"; exit 1; } + n=0 + echo "$rows" | while read -r ns name rdy; do + [ "${rdy:-0}" -ge 1 ] || { echo "FAIL: $ns/$name has no ready replica"; exit 1; } + done || exit 1 + n="$(echo "$rows" | wc -l)" + # controller + webhook + cainjector: without the webhook a Certificate is + # rejected, without cainjector the webhook has no CA — all three or nothing. + [ "$n" -ge 3 ] || { echo "FAIL: expected >=3 cert-manager Deployments, saw $n"; exit 1; } + # GitOps-managed, not hand-installed: Flux labels what it applies. + ns="$(echo "$rows" | head -1 | awk '{print $1}')" + lbl="$(kubectl get deploy -n "$ns" -l "$sel" \ + -o jsonpath='{.items[0].metadata.labels.helm\.toolkit\.fluxcd\.io/name}' 2>/dev/null || true)" + [ -n "$lbl" ] || { echo "FAIL: cert-manager is not owned by a Flux HelmRelease"; exit 1; } + echo "CERT_MANAGER_HEALTHY_OK ($n deployments, flux HelmRelease $lbl)" + check: + (contains($stdout, 'CERT_MANAGER_HEALTHY_OK')): true + + - name: then-the-serving-certificate-was-issued-by-the-controller + description: >- + Follow the Gateway's terminating listener to the Secret it serves, and prove + that Secret's provenance is cert-manager — annotation, CertificateRequest + requester, automatic approval, and a Ready ACME issuer. + try: + - script: + content: | + set -u + # --- discover the serving Secret from the Gateway itself ------------------- + gw="$(kubectl get gateway -A \ + -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}' \ + 2>/dev/null | head -1)" + [ -n "$gw" ] || { echo "FAIL: no Gateway found — nothing terminates TLS"; exit 1; } + gns="${gw% *}"; gname="${gw#* }" + sec="$(kubectl -n "$gns" get gateway "$gname" \ + -o jsonpath='{.spec.listeners[?(@.tls.mode=="Terminate")].tls.certificateRefs[0].name}' \ + 2>/dev/null | awk '{print $1}')" + [ -n "$sec" ] || { echo "FAIL: Gateway $gns/$gname terminates no TLS listener"; exit 1; } + echo "gateway=$gns/$gname serving-secret=$sec" + + # --- provenance (1): only the controller writes this annotation ------------ + cert="$(kubectl -n "$gns" get secret "$sec" \ + -o jsonpath='{.metadata.annotations.cert-manager\.io/certificate-name}' 2>/dev/null || true)" + [ -n "$cert" ] || { echo "FAIL: Secret $gns/$sec carries no cert-manager.io/certificate-name annotation — it was not produced by cert-manager"; exit 1; } + crt="$(kubectl -n "$gns" get secret "$sec" -o jsonpath='{.data.tls\.crt}' 2>/dev/null || true)" + [ -n "$crt" ] || { echo "FAIL: Secret $gns/$sec holds no tls.crt"; exit 1; } + + # --- provenance (2): the Certificate is Ready and has a revision ----------- + ready="$(kubectl -n "$gns" get certificate "$cert" \ + -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)" + rev="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.status.revision}' 2>/dev/null || true)" + echo "certificate=$gns/$cert ready=$ready revision=$rev" + [ "$ready" = "True" ] || { echo "FAIL: Certificate $cert is not Ready"; exit 1; } + [ -n "$rev" ] && [ "$rev" -ge 1 ] || { echo "FAIL: Certificate $cert has no issued revision"; exit 1; } + + # --- provenance (3): the request was made BY THE CONTROLLER and auto-approved + # spec.username is the identity that created the CertificateRequest. A human + # operator would appear as a user/OIDC identity, not a ServiceAccount. + cr="$(kubectl -n "$gns" get certificaterequest \ + -o jsonpath="{range .items[?(@.metadata.annotations.cert-manager\.io/certificate-name=='${cert}')]}{.metadata.name}{' '}{.spec.username}{'\n'}{end}" \ + 2>/dev/null | tail -1)" + [ -n "$cr" ] || { echo "FAIL: no CertificateRequest for $cert — cannot show who requested it"; exit 1; } + crname="${cr% *}"; requester="${cr#* }" + echo "certificaterequest=$crname requester=$requester" + case "$requester" in + system:serviceaccount:*) : ;; + *) echo "FAIL: CertificateRequest $crname was submitted by '$requester', not a controller ServiceAccount"; exit 1 ;; + esac + appr="$(kubectl -n "$gns" get certificaterequest "$crname" \ + -o jsonpath='{.status.conditions[?(@.type=="Approved")].status}' 2>/dev/null || true)" + [ "$appr" = "True" ] || { echo "FAIL: CertificateRequest $crname is not Approved=True (approval would be a manual step)"; exit 1; } + + # --- provenance (4): an automated ACME issuer, and it is Ready ------------- + ikind="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.spec.issuerRef.kind}')" + iname="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.spec.issuerRef.name}')" + echo "issuer=$ikind/$iname" + if [ "$ikind" = "ClusterIssuer" ]; then + iready="$(kubectl get clusterissuer "$iname" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)" + iacme="$(kubectl get clusterissuer "$iname" -o jsonpath='{.spec.acme.server}' 2>/dev/null || true)" + else + iready="$(kubectl -n "$gns" get issuer "$iname" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)" + iacme="$(kubectl -n "$gns" get issuer "$iname" -o jsonpath='{.spec.acme.server}' 2>/dev/null || true)" + fi + [ "$iready" = "True" ] || { echo "FAIL: issuer $iname is not Ready — renewal would need a human to fix it"; exit 1; } + [ -n "$iacme" ] || { echo "FAIL: issuer $iname is not an ACME issuer"; exit 1; } + echo "SERVING_CERT_CONTROLLER_ISSUED_OK" + check: + (contains($stdout, 'SERVING_CERT_CONTROLLER_ISSUED_OK')): true + + - name: then-an-unattended-renewal-is-already-scheduled + description: >- + cert-manager writes status.renewalTime only after computing when it will renew. + Asserting it is in the future and before expiry proves the renewal is scheduled + and that nothing is waiting on a human (docs/08 "renews well before expiry"). + try: + - script: + content: | + set -u + gw="$(kubectl get gateway -A \ + -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}' | head -1)" + gns="${gw% *}"; gname="${gw#* }" + sec="$(kubectl -n "$gns" get gateway "$gname" \ + -o jsonpath='{.spec.listeners[?(@.tls.mode=="Terminate")].tls.certificateRefs[0].name}' | awk '{print $1}')" + cert="$(kubectl -n "$gns" get secret "$sec" \ + -o jsonpath='{.metadata.annotations.cert-manager\.io/certificate-name}')" + renew="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.status.renewalTime}')" + after="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.status.notAfter}')" + [ -n "$renew" ] || { echo "FAIL: Certificate $cert has no status.renewalTime — no renewal is scheduled"; exit 1; } + [ -n "$after" ] || { echo "FAIL: Certificate $cert has no status.notAfter"; exit 1; } + now_s="$(date -u +%s)" + r_s="$(date -u -d "$renew" +%s)" + a_s="$(date -u -d "$after" +%s)" + echo "now=$(date -u -d "@$now_s" +%FT%TZ) renewalTime=$renew notAfter=$after" + echo "renewal is in $(( (r_s - now_s) / 86400 ))d; expiry in $(( (a_s - now_s) / 86400 ))d" + [ "$r_s" -gt "$now_s" ] || { echo "FAIL: renewalTime is in the PAST — the scheduled renewal did not happen"; exit 1; } + [ "$r_s" -lt "$a_s" ] || { echo "FAIL: renewalTime is not before notAfter — renewal would come too late"; exit 1; } + echo "RENEWAL_SCHEDULED_OK" + check: + (contains($stdout, 'RENEWAL_SCHEDULED_OK')): true + + # ------------------------------------------------- MUTATING HALF (hermetic namespace) + - name: when-a-certificate-is-declared-then-it-is-issued-with-no-human-step + description: >- + Declare an Issuer and a Certificate in this test's own namespace, with a renewal + window that opens one minute after issuance. Nothing else is done to them for the + rest of the test — every later transition is the controller acting on its own. + try: + - apply: + resource: + # selfSigned: no external CA, no rate limit, no DNS write. See the header for + # what this proves (controller automation) and what it does not (the ACME path). + apiVersion: cert-manager.io/v1 + kind: Issuer + metadata: + name: conformance-lifecycle + spec: + selfSigned: {} + - apply: + resource: + apiVersion: cert-manager.io/v1 + kind: Certificate + metadata: + name: conformance-lifecycle + spec: + secretName: conformance-lifecycle-tls + # 1h is cert-manager's minimum duration; renewing 59m before expiry puts the + # renewal instant ~1 minute after each issuance, so an unattended renewal is + # observable inside a test run instead of 30 days from now. + duration: 1h + renewBefore: 3540s + commonName: conformance-lifecycle.invalid + dnsNames: + - conformance-lifecycle.invalid + privateKey: + rotationPolicy: Always + issuerRef: + kind: Issuer + name: conformance-lifecycle + - assert: + resource: + apiVersion: cert-manager.io/v1 + kind: Certificate + metadata: + name: conformance-lifecycle + status: + conditions: + - type: Ready + status: "True" + revision: 1 + - script: + content: | + set -u + crt="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \ + -o jsonpath='{.data.tls\.crt}' 2>/dev/null || true)" + key="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \ + -o jsonpath='{.data.tls\.key}' 2>/dev/null || true)" + [ -n "$crt" ] && [ -n "$key" ] || { echo "FAIL: no certificate material was issued into conformance-lifecycle-tls"; exit 1; } + # Record the serving bytes so the renewal step can prove they CHANGED. + echo "$crt" | cksum > /tmp/conformance-cert-r1.cksum + echo "revision-1 cert fingerprint: $(cat /tmp/conformance-cert-r1.cksum)" + echo "CERT_ISSUED_OK" + check: + (contains($stdout, 'CERT_ISSUED_OK')): true + + - name: then-it-renews-itself-before-expiry-with-no-human-step + description: >- + Nothing touches the Certificate here. Poll until the controller has reissued it + at least twice on its own and the serving bytes differ from revision 1 — docs/08's + "observed across a renewal", and every request auto-approved. + try: + - script: + content: | + set -u + want=3 # revision 1 = issuance; >=3 means TWO unattended renewals + for i in $(seq 1 30); do + rev="$(kubectl -n "$NAMESPACE" get certificate conformance-lifecycle \ + -o jsonpath='{.status.revision}' 2>/dev/null || echo 0)" + echo "poll $i: revision=${rev:-0}" + [ "${rev:-0}" -ge "$want" ] && break + sleep 20 + done + rev="$(kubectl -n "$NAMESPACE" get certificate conformance-lifecycle \ + -o jsonpath='{.status.revision}' 2>/dev/null || echo 0)" + [ "${rev:-0}" -ge "$want" ] || { echo "FAIL: revision reached only ${rev:-0} — cert-manager did not renew unattended"; exit 1; } + + # The serving material must actually have rotated, not just the revision count. + now="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \ + -o jsonpath='{.data.tls\.crt}' | cksum)" + was="$(cat /tmp/conformance-cert-r1.cksum)" + echo "r1=$was now=$now" + [ "$now" != "$was" ] || { echo "FAIL: the Secret still holds the revision-1 certificate — nothing was actually reissued"; exit 1; } + + # Every renewal request was approved by the controller, never by a person. + bad=0 + for cr in $(kubectl -n "$NAMESPACE" get certificaterequest -o name 2>/dev/null); do + st="$(kubectl -n "$NAMESPACE" get "$cr" \ + -o jsonpath='{.status.conditions[?(@.type=="Approved")].status}' 2>/dev/null || true)" + who="$(kubectl -n "$NAMESPACE" get "$cr" -o jsonpath='{.spec.username}' 2>/dev/null || true)" + echo " $cr approved=$st requester=$who" + [ "$st" = "True" ] || bad=1 + case "$who" in system:serviceaccount:*) : ;; *) bad=1 ;; esac + done + [ "$bad" = 0 ] || { echo "FAIL: a CertificateRequest was not auto-approved, or was not submitted by a controller"; exit 1; } + echo "CERT_RENEWED_UNATTENDED_OK (revision=$rev)" + check: + (contains($stdout, 'CERT_RENEWED_UNATTENDED_OK')): true + + - name: then-removing-the-certificate-withdraws-the-serving-material + description: >- + docs/08 reads revocation as "removing/rotating a certificate is a declarative + change". The declarative change is the deletion; withdrawal of the credential + must follow with no manual cleanup. EXPECTED TO FAIL TODAY — cert-manager needs + `enableCertificateOwnerRef: true` for the Secret to be garbage-collected, and the + tracked HelmRelease does not set it. Kept as the requirement reads (R-TEST-06). + try: + - delete: + ref: + apiVersion: cert-manager.io/v1 + kind: Certificate + name: conformance-lifecycle + - script: + content: | + set -u + for i in $(seq 1 15); do + if ! kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls >/dev/null 2>&1; then + echo "CERT_WITHDRAWN_OK (serving material gone after $((i * 4))s)" + exit 0 + fi + sleep 4 + done + owners="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \ + -o jsonpath='{.metadata.ownerReferences}' 2>/dev/null || true)" + echo "FAIL: the Certificate was deleted but Secret conformance-lifecycle-tls still" + echo " holds usable certificate material (ownerReferences=${owners:-none})." + echo " Removing a certificate therefore does NOT withdraw the credential, so" + echo " R-TLS-02's 'revoked ... without manual steps' does not hold. FIX:" + echo " set enableCertificateOwnerRef: true in platform/cert-manager values" + echo " (owner decision — WS-TEST does not weaken the assertion instead)." + exit 1 + check: + (contains($stdout, 'CERT_WITHDRAWN_OK')): true + finally: + # Namespace teardown removes both objects anyway; explicit so a leftover Secret + # from the expected failure above is never left holding key material. + - script: + content: | + kubectl -n "$NAMESPACE" delete secret conformance-lifecycle-tls --ignore-not-found --wait=false || true + kubectl -n "$NAMESPACE" delete certificate conformance-lifecycle --ignore-not-found --wait=false || true + kubectl -n "$NAMESPACE" delete issuer conformance-lifecycle --ignore-not-found --wait=false || true + rm -f /tmp/conformance-cert-r1.cksum || true