DECISIONS-LOG records Phase 7 being opened knowingly while the uncovered set included R-ENV-01/02/03 — the very requirements prod provisioning serves. Two of those three are now covered, plus R-TLS-02. tls-lifecycle-automated R-TLS-02 L0 (in the merge gate) cert-lifecycle-automated R-TLS-02 L2 (authored, NOT executed) dev-nodes-provisioned-as-code R-ENV-01 L0 prod-baremetal-not-dev-surface R-ENV-03 L0 Coverage 42/51 (82.4%) -> 45/51 (88.2%), 6 gaps, 0 safety-critical. Remaining: R-C2-01, R-C2-02, R-NET-04, R-PLAT-01, R-SEC-03, R-STATE-04. Verified through the real runner: --layer l0 7/7 PASS, the three new ones 3/3 PASS, --strict-coverage exit 0 with 0 label/annotation mismatches. *** R-TLS-02 "REVOKED" IS NOT SATISFIED — A REAL GAP, NOT A TEST DEFECT. *** The spec requires certificates be "issued, renewed, AND REVOKED without manual steps", and docs/08 reads revocation as "removing/rotating a certificate is a declarative change". That needs enableCertificateOwnerRef: true; platform/cert-manager/helmrelease.yaml does not set it (chart default false). Verified read-only on BOTH live clusters: gateway/wildcard-home-tls has ZERO ownerReferences, so deleting the Certificate would leave the private key serving indefinitely. The L2 test asserts the requirement AS WRITTEN and will fail until this is decided. It was deliberately NOT softened — weakening a test to match the implementation is how a suite stops meaning anything. NOT FLIPPED UNATTENDED, and the reason is specific rather than caution: the wildcard Certificate lives under ./platform, which runs prune: true. With owner refs ON, any transient prune of that Certificate — exactly what happened to the CDI StorageProfiles hours ago — would delete the SERVING TLS Secret for every host, and reissuance is bounded by Let's Encrypt's duplicate-certificate limit (5/week per identical name set), which could leave a cluster with no trusted cert for a week. Security gap vs a plausible week-long TLS outage is an operator call. NON-VACUITY DEMONSTRATED, which is the part that makes these worth having: 7/7 perturbations fail for R-TLS-02, 8/8 for R-ENV-01, 13/13 for R-ENV-03, and each restored copy passes. Perturbations were applied to a throwaway copy; neither this repo nor the read-only spec repo was modified. TWO CHECKS INITIALLY PASSED WHEN PERTURBED and were rewritten — the exact degrade-to-no-op failure this repo keeps hitting. The first draft asked "is this file's DIRECTORY referenced by a kustomization", which a directory reference always satisfies, and keyed Secrets by name alone, so external-dns's copy of `cloudflare-api-token` satisfied cert-manager's reference. Now a recursive kustomize resource-graph walk from each Flux path, and Secrets keyed by (namespace, name). Both bite. SECURITY NOTE, recorded because this project has already leaked a password once: `ansible-inventory --list` PRINTS DECRYPTED VAULT VALUES — it emitted the prod become-password to stdout during exploration, since the vault passphrase file is mounted into the dev container. Both new checks use only --syntax-check and --list-hosts, and say so in a comment. Confirmed no plaintext value reached disk: vault.yml is still AES256-encrypted and the tree has no assigned value. WHAT IS NOT PROVEN, per requirement: R-TLS-02 "revoked" (above), and issuance + observed renewal await the unexecuted L2. R-ENV-01 cannot prove the running VMs came from this module (needs OpenTofu state) — it proves the four lifecycle stages exist as code and cover every dev node, and that the chain RESOLVES under real ansible-playbook. R-ENV-03 cannot prove the HARDWARE (the K8s API exposes no "am I virtual" fact; mini-PC/bare-metal is operator-attested) — it proves prod is AUTHORED as 3 directly-reached nodes with no IaC creating them, sharing dev's playbooks and roles, with no prod-only manifest, branch or role anywhere. VERIFIED BY ME: coverage delta and the 3/3 L0 pass through the real runner; the zero-ownerReferences finding on both clusters; that the new checks avoid ansible-inventory; that no credential material reached disk. Full CI gate PASS. Refs R-TLS-02, R-ENV-01, R-ENV-03, R-TEST-01..05. Co-Authored-By: Claude Opus 5 <[email protected]>
529 lines
27 KiB
Bash
Executable File
529 lines
27 KiB
Bash
Executable File
#!/usr/bin/env bash
|
||
# =============================================================================
|
||
# L0 static check (offline, no cluster) — prod-baremetal-not-dev-surface
|
||
# Serves: R-ENV-03 ("prod is a 3× mini-PC bare-metal k3s cluster with local storage; it
|
||
# MUST NOT be an active development surface until the roadmap declares it ready."
|
||
# — spec docs/00-vision-and-scope.md. Detail in docs/02: prod substrate "3× mini-PC
|
||
# bare metal", "3 physical nodes", storage "local SSD/NVMe presented to Longhorn",
|
||
# "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation
|
||
# step differs", and "**Never develop against prod.** Prod hardware is a target, not
|
||
# a sandbox (R-ENV-03)". The roadmap gate is roadmap/phases.md Phase 7: entry
|
||
# "operator declares prod hardware ready"; note "Do not begin building on prod
|
||
# hardware before this phase is explicitly opened".)
|
||
#
|
||
# HOW A "MUST NOT" IS MADE TESTABLE
|
||
# --------------------------------
|
||
# The second clause is a governance rule, and the same discipline applies as to R-TEST-06
|
||
# (tests/l0-static/spec-immutability-enforced): "nobody developed against prod today"
|
||
# cannot be observed, but the PROPERTY THAT MAKES DEVELOPING AGAINST PROD IMPOSSIBLE can
|
||
# be. In this repo that property is structural — prod receives the base configuration plus
|
||
# environment-scoped overrides and NOTHING ELSE, so there is no place to put prod-only work
|
||
# even if someone wanted to. This check asserts that, together with the first clause's
|
||
# authored facts (three nodes, no VM IaC behind them, node-local storage, the same
|
||
# bootstrap code as dev).
|
||
#
|
||
# The first clause's HARDWARE facts (that the three machines really are mini-PCs, really
|
||
# are bare metal, really hold local NVMe) are operator-attested, not machine-checkable —
|
||
# see the gap list. What IS checkable is that the repository describes prod that way and
|
||
# that no IaC creates prod nodes as virtual machines.
|
||
#
|
||
# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted),
|
||
# WHEN the prod inventory, the prod overlay, the IaC module and the roadmap record are
|
||
# inspected — and, where offline execution is possible, the provisioning chain is
|
||
# actually retargeted at prod with `--list-hosts`,
|
||
# THEN (1) prod resolves to EXACTLY THREE nodes, reached directly on their own sshd
|
||
# (no NAT indirection, i.e. real machines on the LAN);
|
||
# (2) NO IaC resource creates any prod node: the VM module's generated node names are
|
||
# disjoint from the prod host set and no prod hostname appears anywhere under
|
||
# infra/tofu — prod is imaged by the operator (docs/02 stage 1), not instantiated;
|
||
# (3) prod's storage is NODE-LOCAL: the Longhorn data path for prod is an absolute
|
||
# local path, and the replica count equals the node count (one replica per node's
|
||
# own disk). A distinct bulk/media tier is explicitly permitted (R-STATE-04) and
|
||
# is not counted against this;
|
||
# (4) prod runs THE SAME bootstrap code as dev: the same playbooks/roles, retargeted
|
||
# by variables only — every prod group_vars key is an existing role/global
|
||
# variable (an invented prod-only knob would be a fork), and no prod-only role
|
||
# or playbook exists;
|
||
# (5) prod is NOT a development surface: clusters/prod carries only the base plus
|
||
# sync.yaml, every patch it applies targets an ENVIRONMENT-SCOPED object
|
||
# (cluster-vars, a Flux Kustomization's sops decryption, or the Git source),
|
||
# prod's Git ref is the SAME ref dev tracks (no prod-only branch to develop on),
|
||
# and no prod-only manifest exists anywhere in the rendered configuration;
|
||
# (6) prod artifacts exist only WITH a declaration of readiness: the spec roadmap
|
||
# carries the phase that opens prod build (checked when the spec repo is mounted),
|
||
# and the repository records that the operator opened it.
|
||
#
|
||
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them):
|
||
# - The HARDWARE. That the three nodes are mini-PCs, are bare metal, and present local
|
||
# SSD/NVMe is operator-attested in infra/PROD-STANDUP.md / PROD-PROVISIONING.md; the
|
||
# node-health and BIOS surveys that establish it write to gitignored artifacts, so
|
||
# nothing tracked can be asserted against. Cluster-side, the Kubernetes API exposes no
|
||
# "am I virtual" fact either — a bare-metal check would need a privileged host-path
|
||
# probe of DMI data, which is a heavier and more invasive test than the claim warrants.
|
||
# - The LIVE prod cluster's shape (three Ready nodes, all control-plane+etcd, Longhorn
|
||
# disks at the local path). That is cluster state and belongs to R-PLAT-01's
|
||
# (currently uncovered) HA test; asserting it here would also make this test
|
||
# environment-specific, since the same suite must pass against dev.
|
||
# - That prod's HISTORY contains no development. This is the working tree at HEAD.
|
||
# - Whether the roadmap SHOULD have opened prod (the entry-gate judgement). Only that
|
||
# prod work is accompanied by the operator's recorded declaration.
|
||
# =============================================================================
|
||
set -euo pipefail
|
||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||
PYTHON="${PYTHON:-python3}"
|
||
: "${SPEC_REPO:=${ROOT}/../home-network-k3s}"
|
||
export SPEC_REPO
|
||
|
||
cd "${ROOT}"
|
||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||
import os
|
||
import re
|
||
import shutil
|
||
import subprocess
|
||
import sys
|
||
from pathlib import Path
|
||
|
||
try:
|
||
import yaml
|
||
except ImportError: # pragma: no cover
|
||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||
file=sys.stderr)
|
||
raise SystemExit(1)
|
||
|
||
ROOT = Path(sys.argv[1])
|
||
FAILURES: list[str] = []
|
||
NOTES: list[str] = []
|
||
CHECKS = 0
|
||
|
||
|
||
def check(cond, msg):
|
||
global CHECKS
|
||
CHECKS += 1
|
||
if not cond:
|
||
FAILURES.append(msg)
|
||
|
||
|
||
def note(msg):
|
||
NOTES.append(msg)
|
||
|
||
|
||
# A LOCAL DEPENDENCY the repo deliberately does not carry — the vault passphrase (kept
|
||
# outside the repo, R-SEC-01) or the Galaxy collections (gitignored; installed from
|
||
# collections/requirements.yml). Their absence is an ENVIRONMENT gap, not a repo defect,
|
||
# and is the ONLY tolerated reason the driven sub-assertions may not run. Deliberately
|
||
# narrow: every other non-zero exit is a real failure. The parse assertions always run.
|
||
ENV_ONLY_FAILURE = re.compile(
|
||
r"vault password|vault-id|Attempting to decrypt"
|
||
r"|No module named 'ansible_collections|couldn't resolve module/action", re.I)
|
||
|
||
ANS = ROOT / "infra" / "ansible"
|
||
INV = ANS / "inventory" / "hosts.ini"
|
||
GV = ANS / "inventory" / "group_vars"
|
||
TOFU = ROOT / "infra" / "tofu"
|
||
check(INV.is_file(), "infra/ansible/inventory/hosts.ini is missing — prod is not described anywhere")
|
||
if not INV.is_file():
|
||
print("CHECK FAIL (prod-baremetal-not-dev-surface): no inventory", file=sys.stderr)
|
||
raise SystemExit(1)
|
||
|
||
# --------------------------------------------------------------- inventory parsing
|
||
ini_groups: dict[str, list[dict]] = {}
|
||
children: dict[str, list[str]] = {}
|
||
cur = None
|
||
for line in INV.read_text().splitlines():
|
||
s = line.strip()
|
||
if not s or s.startswith("#"):
|
||
continue
|
||
m = re.match(r"^\[([^\]]+)\]$", s)
|
||
if m:
|
||
cur = m.group(1)
|
||
continue
|
||
if cur is None or cur.endswith(":vars"):
|
||
continue
|
||
if cur.endswith(":children"):
|
||
children.setdefault(cur.split(":")[0], []).append(s.split()[0])
|
||
continue
|
||
parts = s.split()
|
||
host = {"name": parts[0]}
|
||
for kv in parts[1:]:
|
||
if "=" in kv:
|
||
k, v = kv.split("=", 1)
|
||
host[k] = v
|
||
ini_groups.setdefault(cur, []).append(host)
|
||
|
||
|
||
def hosts_in(group: str, seen=None) -> list[dict]:
|
||
seen = seen or set()
|
||
if group in seen:
|
||
return []
|
||
seen.add(group)
|
||
out = list(ini_groups.get(group, []))
|
||
for child in children.get(group, []):
|
||
out += hosts_in(child, seen)
|
||
return out
|
||
|
||
|
||
def load_gv(group: str) -> dict:
|
||
d: dict = {}
|
||
for p in [GV / f"{group}.yml", GV / group / "vars.yml"]:
|
||
if p.is_file():
|
||
try:
|
||
d.update(yaml.safe_load(p.read_text()) or {})
|
||
except Exception:
|
||
pass
|
||
return d
|
||
|
||
|
||
# WHICH GROUP IS PROD? Derived, never named here: it is the group whose group_vars
|
||
# RETARGETS the k3s roles (k3s_primary_group / k3s_servers_group) away from the global
|
||
# dev default. That is exactly the seam docs/02 + infra/ansible/README describe for prod.
|
||
ALL_VARS = load_gv("all")
|
||
dev_primary_group = str(ALL_VARS.get("k3s_primary_group", ""))
|
||
candidates = []
|
||
for entry in sorted(GV.iterdir()) if GV.is_dir() else []:
|
||
g = entry.name[:-4] if entry.name.endswith(".yml") else entry.name
|
||
if g == "all":
|
||
continue
|
||
v = load_gv(g)
|
||
if v.get("k3s_primary_group") and str(v["k3s_primary_group"]) != dev_primary_group:
|
||
candidates.append((g, v))
|
||
check(len(candidates) == 1,
|
||
f"expected exactly ONE environment whose group_vars retargets the k3s roles (prod); found "
|
||
f"{[c[0] for c in candidates]} — R-ENV-03 describes a single prod estate reached through "
|
||
"the same roles as dev")
|
||
if not candidates:
|
||
print("CHECK FAIL (prod-baremetal-not-dev-surface): no prod group_vars found", file=sys.stderr)
|
||
for f in FAILURES:
|
||
print(f" - {f}", file=sys.stderr)
|
||
raise SystemExit(1)
|
||
PROD_GROUP, PROD_VARS = candidates[0]
|
||
PROD_PRIMARY = str(PROD_VARS["k3s_primary_group"])
|
||
prod_hosts = hosts_in(PROD_GROUP)
|
||
prod_names = sorted(h["name"] for h in prod_hosts)
|
||
|
||
# ------------------------------------------------- (1) exactly three, directly reachable
|
||
check(len(prod_names) == 3,
|
||
f"prod group {PROD_GROUP!r} resolves to {len(prod_names)} host(s) {prod_names} — R-ENV-03 "
|
||
"says prod is a 3× mini-PC cluster")
|
||
dev_hosts = hosts_in(str(ALL_VARS.get("k3s_servers_group", "k3s_servers")))
|
||
for h in prod_hosts:
|
||
# Real machines on the LAN answer their own sshd. Dev's hosts are reached through a
|
||
# NAT port-forward (ansible_port 225x -> a VM's :22), which is precisely the
|
||
# indirection a hypervisor-hosted estate needs and bare metal does not.
|
||
check(str(h.get("ansible_port", "22")) == "22",
|
||
f"prod node {h['name']} is reached on port {h.get('ansible_port')} — a forwarded port "
|
||
"implies an indirection (NAT to a virtual estate), not a machine on the LAN")
|
||
check(h.get("k3s_node_ip") == h.get("ansible_host"),
|
||
f"prod node {h['name']} has ansible_host={h.get('ansible_host')} but "
|
||
f"k3s_node_ip={h.get('k3s_node_ip')} — a split address is the dev NAT pattern; a "
|
||
"bare-metal node's SSH address IS its cluster address")
|
||
|
||
# ------------------------------------------------------ (2) nothing creates prod as a VM
|
||
TF = "\n".join(p.read_text(errors="replace") for p in sorted(TOFU.glob("*.tf"))) \
|
||
if TOFU.is_dir() else ""
|
||
check(len(TF) > 500, "infra/tofu/*.tf is empty — cannot show that it does NOT create prod")
|
||
|
||
|
||
def tf_default(var: str):
|
||
m = re.search(r'variable\s+"' + re.escape(var) + r'"\s*\{(.*?)\n\}', TF, re.S)
|
||
if not m:
|
||
return None
|
||
d = re.search(r"default\s*=\s*(.+)", m.group(1))
|
||
return d.group(1).strip() if d else None
|
||
|
||
|
||
prefix = (tf_default("node_name_prefix") or "").strip('"')
|
||
try:
|
||
node_count = int((tf_default("node_count") or "0").strip())
|
||
except ValueError:
|
||
node_count = 0
|
||
iac_names = {f"{prefix}{i + 1}" for i in range(node_count)}
|
||
check(len(iac_names) > 0, "the IaC module generates no node names — cannot compare with prod")
|
||
check(iac_names.isdisjoint(set(prod_names)),
|
||
f"the IaC VM module would create {sorted(iac_names & set(prod_names))}, which are PROD nodes "
|
||
"— prod is bare metal that the operator images (docs/02 stage 1), never instantiated by the "
|
||
"hypervisor module")
|
||
for name in prod_names:
|
||
check(name not in TF,
|
||
f"prod node {name} is named inside infra/tofu — prod hardware must not be described by "
|
||
"the VM module at all")
|
||
# The prod host addresses must not appear as IaC node addresses either (a renamed VM is
|
||
# still a VM).
|
||
tf_ips = set(re.findall(r'"(\d+\.\d+\.\d+\.\d+)"', tf_default("node_ips") or ""))
|
||
prod_ips = {str(h.get("ansible_host")) for h in prod_hosts}
|
||
check(tf_ips.isdisjoint(prod_ips),
|
||
f"the IaC module addresses {sorted(tf_ips & prod_ips)}, which are prod node addresses")
|
||
|
||
# --------------------------------------------------------------- (3) node-local storage
|
||
# Every environment's value for a cluster-vars knob (base default + each overlay patch).
|
||
def env_values(key: str) -> dict[str, str]:
|
||
vals: dict[str, str] = {}
|
||
base = ROOT / "clusters" / "base" / "cluster-vars.yaml"
|
||
if base.is_file():
|
||
for d in yaml.safe_load_all(base.read_text()):
|
||
if isinstance(d, dict) and d.get("kind") == "ConfigMap" \
|
||
and (d.get("metadata") or {}).get("name") == "cluster-vars":
|
||
v = (d.get("data") or {}).get(key)
|
||
if v is not None:
|
||
vals["base"] = str(v)
|
||
for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
|
||
env = overlay.parent.name
|
||
if env == "base":
|
||
continue
|
||
try:
|
||
k = yaml.safe_load(overlay.read_text()) or {}
|
||
except Exception:
|
||
continue
|
||
for p in (k.get("patches") or []):
|
||
tgt = p.get("target") or {}
|
||
if tgt.get("kind") != "ConfigMap" or tgt.get("name") != "cluster-vars":
|
||
continue
|
||
try:
|
||
ops = yaml.safe_load(p.get("patch") or "") or []
|
||
except Exception:
|
||
continue
|
||
for op in ops if isinstance(ops, list) else []:
|
||
if isinstance(op, dict) and str(op.get("path", "")) == f"/data/{key}":
|
||
vals[env] = str(op.get("value"))
|
||
return vals
|
||
|
||
|
||
# The prod OVERLAY is the one whose CLUSTER_ENV is not the base default.
|
||
cluster_envs = env_values("CLUSTER_ENV")
|
||
prod_overlays = [e for e, v in cluster_envs.items()
|
||
if e != "base" and v != cluster_envs.get("base")]
|
||
check(len(prod_overlays) >= 1,
|
||
"no clusters/ overlay sets its own CLUSTER_ENV — prod has no overlay of its own")
|
||
prod_overlay = None
|
||
for e in prod_overlays:
|
||
if PROD_GROUP.replace("k3s_", "") in cluster_envs[e] or cluster_envs[e] in PROD_GROUP:
|
||
prod_overlay = e
|
||
check(prod_overlay is not None,
|
||
f"no overlay's CLUSTER_ENV corresponds to the prod inventory group {PROD_GROUP!r} — the "
|
||
"cluster config and the machine inventory disagree about which estate is prod")
|
||
|
||
data_paths = env_values("LONGHORN_DATA_PATH")
|
||
replicas = env_values("LONGHORN_DEFAULT_REPLICA_COUNT")
|
||
if prod_overlay:
|
||
p = data_paths.get(prod_overlay, data_paths.get("base", ""))
|
||
check(p.startswith("/"),
|
||
f"prod's LONGHORN_DATA_PATH is {p!r} — R-ENV-03's 'local storage' requires an absolute "
|
||
"node-local filesystem path")
|
||
check(not re.match(r"^/(net|nfs|mnt/nfs)|:", p),
|
||
f"prod's LONGHORN_DATA_PATH {p!r} looks like a network mount, not local SSD/NVMe")
|
||
r = replicas.get(prod_overlay, replicas.get("base", ""))
|
||
check(r.isdigit() and int(r) == len(prod_names),
|
||
f"prod's LONGHORN_DEFAULT_REPLICA_COUNT is {r!r} for {len(prod_names)} node(s) — "
|
||
"'local storage' on a 3-node estate means one replica per node's own disk")
|
||
# The DEFAULT storage class must be the local (Longhorn) driver. A separate bulk/media tier
|
||
# on network storage is explicitly a different requirement (R-STATE-04, docs/04) and is not
|
||
# counted here.
|
||
default_scs = []
|
||
for f in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z", "*.yaml", "*.yml"],
|
||
capture_output=True, text=True, check=True).stdout.split("\0"):
|
||
if not f:
|
||
continue
|
||
try:
|
||
for d in yaml.safe_load_all((ROOT / f).read_text(errors="replace")):
|
||
if isinstance(d, dict) and d.get("kind") == "StorageClass" and \
|
||
str(((d.get("metadata") or {}).get("annotations") or {})
|
||
.get("storageclass.kubernetes.io/is-default-class", "")) == "true":
|
||
default_scs.append((f, str(d.get("provisioner", ""))))
|
||
except Exception:
|
||
continue
|
||
check(len(default_scs) == 1,
|
||
f"expected exactly one default StorageClass, found {default_scs} — the tier prod's workloads "
|
||
"land on by default must be unambiguous")
|
||
for f, prov in default_scs:
|
||
check(prov == "driver.longhorn.io",
|
||
f"{f}: the default StorageClass provisioner is {prov!r}, not Longhorn — prod's default "
|
||
"tier would not be its own local disks")
|
||
|
||
# ------------------------------------- (4) prod runs the SAME bootstrap code, retargeted
|
||
PLAYBOOKS = ANS / "playbooks"
|
||
ROLES = ANS / "roles"
|
||
prod_token = PROD_GROUP.replace("k3s_", "")
|
||
for d, what in ((ROLES, "role"), (PLAYBOOKS, "playbook")):
|
||
for entry in sorted(d.iterdir()) if d.is_dir() else []:
|
||
check(prod_token not in entry.name.lower(),
|
||
f"infra/ansible/{d.name}/{entry.name} is a prod-specific {what} — docs/02 requires "
|
||
"the SAME Ansible bootstrap for prod, differing only in the machine-creation step")
|
||
# Every prod group_vars key must be a variable the shared code already knows: a role default,
|
||
# a global, or an ansible built-in. An invented key is a fork of the role by another name.
|
||
known: set[str] = set(ALL_VARS)
|
||
for rd in sorted(ROLES.iterdir()) if ROLES.is_dir() else []:
|
||
for p in (rd / "defaults").glob("*.yml") if (rd / "defaults").is_dir() else []:
|
||
try:
|
||
known |= set((yaml.safe_load(p.read_text()) or {}).keys())
|
||
except Exception:
|
||
pass
|
||
check(len(known) > 20, f"only {len(known)} shared variables found — (4) would prove nothing")
|
||
for k in sorted(PROD_VARS):
|
||
check(k.startswith("ansible_") or k in known,
|
||
f"prod group_vars sets {k!r}, which is not a role default or a global — prod-only "
|
||
"configuration that the dev path never exercises is developing against prod")
|
||
# ...and it must retarget, not re-implement: the group it names must exist in the inventory.
|
||
check(PROD_PRIMARY in ini_groups or PROD_PRIMARY in children,
|
||
f"prod group_vars names primary group {PROD_PRIMARY!r}, which the inventory does not define")
|
||
|
||
# ---------------------------------------- (5) prod is not a development surface (overlay)
|
||
if prod_overlay:
|
||
odir = ROOT / "clusters" / prod_overlay
|
||
files = sorted(p.name for p in odir.iterdir() if p.is_file())
|
||
check(files == ["kustomization.yaml", "sync.yaml"],
|
||
f"clusters/{prod_overlay}/ contains {files} — a prod overlay may hold only its "
|
||
"kustomization and its own root sync; anything else is configuration authored FOR PROD")
|
||
k = yaml.safe_load((odir / "kustomization.yaml").read_text()) or {}
|
||
check(sorted(str(r) for r in (k.get("resources") or [])) == ["../base", "sync.yaml"],
|
||
f"clusters/{prod_overlay} resources are {k.get('resources')} — prod must render the base "
|
||
"plus its own sync only (docs/02: 'apply the prod overlay (no base edits)')")
|
||
# Every patch must be an ENVIRONMENT-SCOPED override, never a change to what runs.
|
||
ALLOWED = {("ConfigMap", "cluster-vars"), ("GitRepository", None)}
|
||
for p in (k.get("patches") or []):
|
||
tgt = p.get("target") or {}
|
||
body = p.get("patch") or ""
|
||
kind = tgt.get("kind")
|
||
if kind is None: # inline patch: kind comes from the body
|
||
try:
|
||
doc = yaml.safe_load(body) or {}
|
||
except Exception:
|
||
doc = {}
|
||
kind = doc.get("kind")
|
||
name = (doc.get("metadata") or {}).get("name")
|
||
if kind == "Kustomization":
|
||
# Flux Kustomization patches are allowed ONLY to wire sops decryption —
|
||
# anything else (path, interval, images, values) changes what prod RUNS.
|
||
keys = set((doc.get("spec") or {}).keys())
|
||
check(keys <= {"decryption"},
|
||
f"clusters/{prod_overlay}: patch of Kustomization/{name} touches {sorted(keys)} "
|
||
"— only `decryption` (the per-cluster sops key) is an environment-scoped "
|
||
"override; changing anything else is developing against prod")
|
||
continue
|
||
check((kind, None) in ALLOWED or (kind, name) in ALLOWED,
|
||
f"clusters/{prod_overlay}: patch targets {kind}/{name}, which is not an "
|
||
"environment-scoped object — prod must not carry configuration of its own")
|
||
else:
|
||
check((kind, tgt.get("name")) in ALLOWED or (kind, None) in ALLOWED,
|
||
f"clusters/{prod_overlay}: patch targets {kind}/{tgt.get('name')}, which is not "
|
||
"an environment-scoped object — prod must not carry configuration of its own")
|
||
|
||
# Same Git ref as the other environment(s): a prod-only branch is a development line
|
||
# that only prod runs, which is exactly what R-ENV-03 forbids.
|
||
def git_ref(overlay: str):
|
||
kk = yaml.safe_load((ROOT / "clusters" / overlay / "kustomization.yaml").read_text()) or {}
|
||
for p in (kk.get("patches") or []):
|
||
body = p.get("patch") or ""
|
||
try:
|
||
doc = yaml.safe_load(body) or {}
|
||
except Exception:
|
||
continue
|
||
if doc.get("kind") == "GitRepository":
|
||
return (doc.get("spec") or {}).get("ref")
|
||
return None
|
||
|
||
others = [o.name for o in (ROOT / "clusters").iterdir()
|
||
if o.is_dir() and o.name not in ("base", prod_overlay)]
|
||
prod_ref = git_ref(prod_overlay)
|
||
check(prod_ref is not None,
|
||
f"clusters/{prod_overlay} does not pin a Git ref — prod would follow whatever the base "
|
||
"points at, with no record of what it runs")
|
||
for o in others:
|
||
check(git_ref(o) == prod_ref,
|
||
f"clusters/{o} tracks {git_ref(o)} but prod tracks {prod_ref} — a prod-only ref is a "
|
||
"development line only prod runs (R-ENV-03: prod is a target, not a sandbox)")
|
||
|
||
# No prod-only manifest anywhere in the rendered configuration.
|
||
tracked = [f for f in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
|
||
capture_output=True, text=True, check=True
|
||
).stdout.split("\0") if f]
|
||
RENDERED_DIRS = ("platform/", "platform-config", "apps/", "catalog/", "clusters/base/")
|
||
for f in tracked:
|
||
if f.startswith(RENDERED_DIRS) and re.search(rf"(^|[/_.-]){re.escape(prod_token)}([/_.-]|$)",
|
||
f, re.I):
|
||
check(False,
|
||
f"{f} is a prod-specific file inside the shared configuration — every environment "
|
||
"renders these paths, so prod-only content here is development against prod")
|
||
check(len(tracked) > 50, f"git ls-files returned only {len(tracked)} paths — not a real checkout")
|
||
|
||
# --------------------------- (6) prod work exists only with a declaration of readiness
|
||
# In-repo: the operator's recorded decision to open the prod phase.
|
||
log = ROOT / "DECISIONS-LOG.md"
|
||
check(log.is_file() and re.search(r"PHASE\s*7\b.*OPEN", log.read_text(errors="replace"), re.I),
|
||
"DECISIONS-LOG.md carries no record of the operator OPENING the prod phase, yet prod "
|
||
"inventory/overlay exist — R-ENV-03 allows prod build only once the roadmap declares it ready")
|
||
# In the spec (read-only, may not be mounted): the roadmap phase that opens prod.
|
||
spec = Path(os.environ.get("SPEC_REPO", ""))
|
||
phases = spec / "roadmap" / "phases.md"
|
||
if phases.is_file():
|
||
txt = phases.read_text(errors="replace")
|
||
check(re.search(r"##\s*Phase\s*7\b", txt) is not None and
|
||
re.search(r"prod", txt, re.I) is not None,
|
||
"the spec roadmap has no Phase 7 opening prod build — the declaration R-ENV-03 defers "
|
||
"to does not exist")
|
||
check(re.search(r"explicitly opened|declares prod hardware ready|operator declares", txt, re.I)
|
||
is not None,
|
||
"the spec roadmap's prod phase states no explicit entry declaration — the gate R-ENV-03 "
|
||
"names would be unenforceable")
|
||
else:
|
||
note(f"spec repo not mounted at {spec} — the roadmap half of (6) was NOT checked "
|
||
"(the in-repo declaration still was)")
|
||
|
||
# --------------------------- (4b) DRIVE the retarget offline: same chain, prod hosts
|
||
if shutil.which("ansible-playbook") is None:
|
||
note("ansible-playbook is not installed here (e.g. the in-cluster chainsaw image), so the "
|
||
"prod retarget was PARSED but not DRIVEN; the merge gate's container has it")
|
||
else:
|
||
sec_groups = [g for g in children.get(PROD_GROUP, []) if g != PROD_PRIMARY]
|
||
args = ["--list-hosts", "playbooks/site.yml",
|
||
"-e", f"k3s_hosts={PROD_GROUP}",
|
||
"-e", f"k3s_primary_hosts={PROD_PRIMARY}"]
|
||
if sec_groups:
|
||
args += ["-e", f"k3s_secondary_hosts={sec_groups[0]}"]
|
||
lh = subprocess.run(["ansible-playbook", *args], cwd=ANS, capture_output=True, text=True)
|
||
out = (lh.stderr or "") + (lh.stdout or "")
|
||
if lh.returncode != 0 and ENV_ONLY_FAILURE.search(out):
|
||
# See the same narrow, self-policing skip in dev-nodes-provisioned-as-code.
|
||
note("ansible-playbook lacks a LOCAL dependency kept outside the repo by design (the vault "
|
||
"passphrase and/or the gitignored Galaxy collections), so the prod retarget was not "
|
||
"driven; every parse assertion above still ran")
|
||
else:
|
||
check(lh.returncode == 0,
|
||
f"retargeting the SHARED chain at prod failed: {out.strip()[:400]} — docs/02 requires "
|
||
"the same Ansible bootstrap to run for prod")
|
||
seen: set[str] = set()
|
||
in_hosts = False
|
||
for line in lh.stdout.splitlines():
|
||
if re.match(r"\s*hosts \(\d+\):", line):
|
||
in_hosts = True
|
||
continue
|
||
if re.search(r"play #\d+", line):
|
||
in_hosts = False
|
||
continue
|
||
if in_hosts and line.strip():
|
||
seen.add(line.strip())
|
||
check(seen == set(prod_names),
|
||
f"the shared chain retargeted at prod resolves to {sorted(seen)}, not the three prod "
|
||
f"nodes {prod_names} — prod would be bootstrapped by something other than dev's path")
|
||
|
||
# --------------------------------------------------------------------------- verdict
|
||
check(CHECKS >= 25,
|
||
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
|
||
"would be meaningless")
|
||
|
||
if FAILURES:
|
||
print(f"CHECK FAIL (prod-baremetal-not-dev-surface): {len(FAILURES)} of {CHECKS} assertions failed",
|
||
file=sys.stderr)
|
||
for f in FAILURES:
|
||
print(f" - {f}", file=sys.stderr)
|
||
raise SystemExit(1)
|
||
for n in NOTES:
|
||
print(f" note: {n}")
|
||
print(f"check OK: prod-baremetal-not-dev-surface — {CHECKS} assertions; prod group {PROD_GROUP!r} = "
|
||
f"{prod_names} (3 nodes, none created by IaC), overlay clusters/{prod_overlay} renders the base "
|
||
f"plus environment-scoped patches only, local Longhorn path "
|
||
f"{data_paths.get(prod_overlay, data_paths.get('base'))!r} with "
|
||
f"{replicas.get(prod_overlay, replicas.get('base'))} replica(s). R-ENV-03's AUTHORED clauses "
|
||
"only — the hardware facts (mini-PC, bare metal, local NVMe) are operator-attested and the "
|
||
"live cluster's shape is NOT asserted here (see the header's gap list).")
|
||
PY
|