692 lines
36 KiB
Bash
Executable File
692 lines
36 KiB
Bash
Executable File
#!/usr/bin/env bash
|
||
# =============================================================================
|
||
# L0 static check (offline, no cluster) — prod-baremetal-not-dev-surface
|
||
# Serves: R-ENV-03 ("prod is a 3× mini-PC bare-metal k3s cluster with local storage; it
|
||
# MUST NOT be an active development surface until the roadmap declares it ready."
|
||
# — spec docs/00-vision-and-scope.md. Detail in docs/02: prod substrate "3× mini-PC
|
||
# bare metal", "3 physical nodes", storage "local SSD/NVMe presented to Longhorn",
|
||
# "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation
|
||
# step differs", and "**Never develop against prod.** Prod hardware is a target, not
|
||
# a sandbox (R-ENV-03)". The roadmap gate is roadmap/phases.md Phase 7: entry
|
||
# "operator declares prod hardware ready"; note "Do not begin building on prod
|
||
# hardware before this phase is explicitly opened".)
|
||
#
|
||
# HOW A "MUST NOT" IS MADE TESTABLE
|
||
# --------------------------------
|
||
# The second clause is a governance rule, and the same discipline applies as to R-TEST-06
|
||
# (tests/l0-static/spec-immutability-enforced): "nobody developed against prod today"
|
||
# cannot be observed, but the PROPERTY THAT MAKES DEVELOPING AGAINST PROD IMPOSSIBLE can
|
||
# be. In this repo that property is structural — prod receives the base configuration plus
|
||
# environment-scoped overrides and NOTHING ELSE, so there is no place to put prod-only work
|
||
# even if someone wanted to. This check asserts that, together with the first clause's
|
||
# authored facts (three nodes, no VM IaC behind them, node-local storage, the same
|
||
# bootstrap code as dev).
|
||
#
|
||
# The first clause's HARDWARE facts (that the three machines really are mini-PCs, really
|
||
# are bare metal, really hold local NVMe) are operator-attested, not machine-checkable —
|
||
# see the gap list. What IS checkable is that the repository describes prod that way and
|
||
# that no IaC creates prod nodes as virtual machines.
|
||
#
|
||
# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted),
|
||
# WHEN the prod inventory, the prod overlay, the IaC module and the roadmap record are
|
||
# inspected — and, where offline execution is possible, the provisioning chain is
|
||
# actually retargeted at prod with `--list-hosts`,
|
||
# THEN (1) prod resolves to EXACTLY THREE nodes, reached directly on their own sshd
|
||
# (no NAT indirection, i.e. real machines on the LAN);
|
||
# (2) NO IaC resource creates any prod node: the VM module's generated node names are
|
||
# disjoint from the prod host set and no prod hostname appears anywhere under
|
||
# infra/tofu — prod is imaged by the operator (docs/02 stage 1), not instantiated;
|
||
# (3) prod's storage is NODE-LOCAL: the Longhorn data path for prod is an absolute
|
||
# local path, and the replica count equals the node count (one replica per node's
|
||
# own disk). A distinct bulk/media tier is explicitly permitted (R-STATE-04) and
|
||
# is not counted against this;
|
||
# (4) prod runs THE SAME bootstrap code as dev: the same playbooks/roles, retargeted
|
||
# by variables only — every prod group_vars key is an existing role/global
|
||
# variable (an invented prod-only knob would be a fork), and no prod-only role
|
||
# or playbook exists;
|
||
# (5) prod is NOT a development surface: clusters/prod carries only the base plus
|
||
# sync.yaml, every patch it applies targets an ENVIRONMENT-SCOPED object
|
||
# (cluster-vars, a Flux Kustomization's sops decryption, or the Git source),
|
||
# prod's Git ref is the SAME ref dev tracks (no prod-only branch to develop on),
|
||
# and no prod-only manifest exists anywhere in the rendered configuration — a file
|
||
# whose NAME carries the environment token is excused ONLY when the tree proves it
|
||
# is an OVERLAY-SELECTED half that every environment applies (see the four
|
||
# conditions at "THE ONE STRUCTURAL EXCEPTION" below); anything else still fails;
|
||
# (6) prod artifacts exist only WITH a declaration of readiness: the spec roadmap
|
||
# carries the phase that opens prod build (checked when the spec repo is mounted),
|
||
# and the repository records that the operator opened it.
|
||
#
|
||
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them):
|
||
# - The HARDWARE. That the three nodes are mini-PCs, are bare metal, and present local
|
||
# SSD/NVMe is operator-attested in infra/PROD-STANDUP.md / PROD-PROVISIONING.md; the
|
||
# node-health and BIOS surveys that establish it write to gitignored artifacts, so
|
||
# nothing tracked can be asserted against. Cluster-side, the Kubernetes API exposes no
|
||
# "am I virtual" fact either — a bare-metal check would need a privileged host-path
|
||
# probe of DMI data, which is a heavier and more invasive test than the claim warrants.
|
||
# - The LIVE prod cluster's shape (three Ready nodes, all control-plane+etcd, Longhorn
|
||
# disks at the local path). That is cluster state and belongs to R-PLAT-01's
|
||
# (currently uncovered) HA test; asserting it here would also make this test
|
||
# environment-specific, since the same suite must pass against dev.
|
||
# - That prod's HISTORY contains no development. This is the working tree at HEAD.
|
||
# - Whether the roadmap SHOULD have opened prod (the entry-gate judgement). Only that
|
||
# prod work is accompanied by the operator's recorded declaration.
|
||
# =============================================================================
|
||
set -euo pipefail
|
||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||
PYTHON="${PYTHON:-python3}"
|
||
: "${SPEC_REPO:=${ROOT}/../home-network-k3s}"
|
||
export SPEC_REPO
|
||
|
||
cd "${ROOT}"
|
||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||
import os
|
||
import re
|
||
import shutil
|
||
import subprocess
|
||
import sys
|
||
from pathlib import Path
|
||
|
||
try:
|
||
import yaml
|
||
except ImportError: # pragma: no cover
|
||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||
file=sys.stderr)
|
||
raise SystemExit(1)
|
||
|
||
ROOT = Path(sys.argv[1])
|
||
FAILURES: list[str] = []
|
||
NOTES: list[str] = []
|
||
CHECKS = 0
|
||
|
||
|
||
def check(cond, msg):
|
||
global CHECKS
|
||
CHECKS += 1
|
||
if not cond:
|
||
FAILURES.append(msg)
|
||
|
||
|
||
def note(msg):
|
||
NOTES.append(msg)
|
||
|
||
|
||
# A LOCAL DEPENDENCY the repo deliberately does not carry — the vault passphrase (kept
|
||
# outside the repo, R-SEC-01) or the Galaxy collections (gitignored; installed from
|
||
# collections/requirements.yml). Their absence is an ENVIRONMENT gap, not a repo defect,
|
||
# and is the ONLY tolerated reason the driven sub-assertions may not run. Deliberately
|
||
# narrow: every other non-zero exit is a real failure. The parse assertions always run.
|
||
ENV_ONLY_FAILURE = re.compile(
|
||
r"vault password|vault-id|Attempting to decrypt"
|
||
r"|No module named 'ansible_collections|couldn't resolve module/action", re.I)
|
||
|
||
ANS = ROOT / "infra" / "ansible"
|
||
INV = ANS / "inventory" / "hosts.ini"
|
||
GV = ANS / "inventory" / "group_vars"
|
||
TOFU = ROOT / "infra" / "tofu"
|
||
check(INV.is_file(), "infra/ansible/inventory/hosts.ini is missing — prod is not described anywhere")
|
||
if not INV.is_file():
|
||
print("CHECK FAIL (prod-baremetal-not-dev-surface): no inventory", file=sys.stderr)
|
||
raise SystemExit(1)
|
||
|
||
# --------------------------------------------------------------- inventory parsing
|
||
ini_groups: dict[str, list[dict]] = {}
|
||
children: dict[str, list[str]] = {}
|
||
cur = None
|
||
for line in INV.read_text().splitlines():
|
||
s = line.strip()
|
||
if not s or s.startswith("#"):
|
||
continue
|
||
m = re.match(r"^\[([^\]]+)\]$", s)
|
||
if m:
|
||
cur = m.group(1)
|
||
continue
|
||
if cur is None or cur.endswith(":vars"):
|
||
continue
|
||
if cur.endswith(":children"):
|
||
children.setdefault(cur.split(":")[0], []).append(s.split()[0])
|
||
continue
|
||
parts = s.split()
|
||
host = {"name": parts[0]}
|
||
for kv in parts[1:]:
|
||
if "=" in kv:
|
||
k, v = kv.split("=", 1)
|
||
host[k] = v
|
||
ini_groups.setdefault(cur, []).append(host)
|
||
|
||
|
||
def hosts_in(group: str, seen=None) -> list[dict]:
|
||
seen = seen or set()
|
||
if group in seen:
|
||
return []
|
||
seen.add(group)
|
||
out = list(ini_groups.get(group, []))
|
||
for child in children.get(group, []):
|
||
out += hosts_in(child, seen)
|
||
return out
|
||
|
||
|
||
def load_gv(group: str) -> dict:
|
||
d: dict = {}
|
||
for p in [GV / f"{group}.yml", GV / group / "vars.yml"]:
|
||
if p.is_file():
|
||
try:
|
||
d.update(yaml.safe_load(p.read_text()) or {})
|
||
except Exception:
|
||
pass
|
||
return d
|
||
|
||
|
||
# WHICH GROUP IS PROD? Derived, never named here: it is the group whose group_vars
|
||
# RETARGETS the k3s roles (k3s_primary_group / k3s_servers_group) away from the global
|
||
# dev default. That is exactly the seam docs/02 + infra/ansible/README describe for prod.
|
||
ALL_VARS = load_gv("all")
|
||
dev_primary_group = str(ALL_VARS.get("k3s_primary_group", ""))
|
||
candidates = []
|
||
for entry in sorted(GV.iterdir()) if GV.is_dir() else []:
|
||
g = entry.name[:-4] if entry.name.endswith(".yml") else entry.name
|
||
if g == "all":
|
||
continue
|
||
v = load_gv(g)
|
||
if v.get("k3s_primary_group") and str(v["k3s_primary_group"]) != dev_primary_group:
|
||
candidates.append((g, v))
|
||
check(len(candidates) == 1,
|
||
f"expected exactly ONE environment whose group_vars retargets the k3s roles (prod); found "
|
||
f"{[c[0] for c in candidates]} — R-ENV-03 describes a single prod estate reached through "
|
||
"the same roles as dev")
|
||
if not candidates:
|
||
print("CHECK FAIL (prod-baremetal-not-dev-surface): no prod group_vars found", file=sys.stderr)
|
||
for f in FAILURES:
|
||
print(f" - {f}", file=sys.stderr)
|
||
raise SystemExit(1)
|
||
PROD_GROUP, PROD_VARS = candidates[0]
|
||
PROD_PRIMARY = str(PROD_VARS["k3s_primary_group"])
|
||
prod_hosts = hosts_in(PROD_GROUP)
|
||
prod_names = sorted(h["name"] for h in prod_hosts)
|
||
|
||
# ------------------------------------------------- (1) exactly three, directly reachable
|
||
check(len(prod_names) == 3,
|
||
f"prod group {PROD_GROUP!r} resolves to {len(prod_names)} host(s) {prod_names} — R-ENV-03 "
|
||
"says prod is a 3× mini-PC cluster")
|
||
dev_hosts = hosts_in(str(ALL_VARS.get("k3s_servers_group", "k3s_servers")))
|
||
for h in prod_hosts:
|
||
# Real machines on the LAN answer their own sshd. Dev's hosts are reached through a
|
||
# NAT port-forward (ansible_port 225x -> a VM's :22), which is precisely the
|
||
# indirection a hypervisor-hosted estate needs and bare metal does not.
|
||
check(str(h.get("ansible_port", "22")) == "22",
|
||
f"prod node {h['name']} is reached on port {h.get('ansible_port')} — a forwarded port "
|
||
"implies an indirection (NAT to a virtual estate), not a machine on the LAN")
|
||
check(h.get("k3s_node_ip") == h.get("ansible_host"),
|
||
f"prod node {h['name']} has ansible_host={h.get('ansible_host')} but "
|
||
f"k3s_node_ip={h.get('k3s_node_ip')} — a split address is the dev NAT pattern; a "
|
||
"bare-metal node's SSH address IS its cluster address")
|
||
|
||
# ------------------------------------------------------ (2) nothing creates prod as a VM
|
||
TF = "\n".join(p.read_text(errors="replace") for p in sorted(TOFU.glob("*.tf"))) \
|
||
if TOFU.is_dir() else ""
|
||
check(len(TF) > 500, "infra/tofu/*.tf is empty — cannot show that it does NOT create prod")
|
||
|
||
|
||
def tf_default(var: str):
|
||
m = re.search(r'variable\s+"' + re.escape(var) + r'"\s*\{(.*?)\n\}', TF, re.S)
|
||
if not m:
|
||
return None
|
||
d = re.search(r"default\s*=\s*(.+)", m.group(1))
|
||
return d.group(1).strip() if d else None
|
||
|
||
|
||
prefix = (tf_default("node_name_prefix") or "").strip('"')
|
||
try:
|
||
node_count = int((tf_default("node_count") or "0").strip())
|
||
except ValueError:
|
||
node_count = 0
|
||
iac_names = {f"{prefix}{i + 1}" for i in range(node_count)}
|
||
check(len(iac_names) > 0, "the IaC module generates no node names — cannot compare with prod")
|
||
check(iac_names.isdisjoint(set(prod_names)),
|
||
f"the IaC VM module would create {sorted(iac_names & set(prod_names))}, which are PROD nodes "
|
||
"— prod is bare metal that the operator images (docs/02 stage 1), never instantiated by the "
|
||
"hypervisor module")
|
||
for name in prod_names:
|
||
check(name not in TF,
|
||
f"prod node {name} is named inside infra/tofu — prod hardware must not be described by "
|
||
"the VM module at all")
|
||
# The prod host addresses must not appear as IaC node addresses either (a renamed VM is
|
||
# still a VM).
|
||
tf_ips = set(re.findall(r'"(\d+\.\d+\.\d+\.\d+)"', tf_default("node_ips") or ""))
|
||
prod_ips = {str(h.get("ansible_host")) for h in prod_hosts}
|
||
check(tf_ips.isdisjoint(prod_ips),
|
||
f"the IaC module addresses {sorted(tf_ips & prod_ips)}, which are prod node addresses")
|
||
|
||
# --------------------------------------------------------------- (3) node-local storage
|
||
# Every environment's value for a cluster-vars knob (base default + each overlay patch).
|
||
def env_values(key: str) -> dict[str, str]:
|
||
vals: dict[str, str] = {}
|
||
base = ROOT / "clusters" / "base" / "cluster-vars.yaml"
|
||
if base.is_file():
|
||
for d in yaml.safe_load_all(base.read_text()):
|
||
if isinstance(d, dict) and d.get("kind") == "ConfigMap" \
|
||
and (d.get("metadata") or {}).get("name") == "cluster-vars":
|
||
v = (d.get("data") or {}).get(key)
|
||
if v is not None:
|
||
vals["base"] = str(v)
|
||
for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
|
||
env = overlay.parent.name
|
||
if env == "base":
|
||
continue
|
||
try:
|
||
k = yaml.safe_load(overlay.read_text()) or {}
|
||
except Exception:
|
||
continue
|
||
for p in (k.get("patches") or []):
|
||
tgt = p.get("target") or {}
|
||
if tgt.get("kind") != "ConfigMap" or tgt.get("name") != "cluster-vars":
|
||
continue
|
||
try:
|
||
ops = yaml.safe_load(p.get("patch") or "") or []
|
||
except Exception:
|
||
continue
|
||
for op in ops if isinstance(ops, list) else []:
|
||
if isinstance(op, dict) and str(op.get("path", "")) == f"/data/{key}":
|
||
vals[env] = str(op.get("value"))
|
||
return vals
|
||
|
||
|
||
# The prod OVERLAY is the one whose CLUSTER_ENV is not the base default.
|
||
cluster_envs = env_values("CLUSTER_ENV")
|
||
prod_overlays = [e for e, v in cluster_envs.items()
|
||
if e != "base" and v != cluster_envs.get("base")]
|
||
check(len(prod_overlays) >= 1,
|
||
"no clusters/ overlay sets its own CLUSTER_ENV — prod has no overlay of its own")
|
||
prod_overlay = None
|
||
for e in prod_overlays:
|
||
if PROD_GROUP.replace("k3s_", "") in cluster_envs[e] or cluster_envs[e] in PROD_GROUP:
|
||
prod_overlay = e
|
||
check(prod_overlay is not None,
|
||
f"no overlay's CLUSTER_ENV corresponds to the prod inventory group {PROD_GROUP!r} — the "
|
||
"cluster config and the machine inventory disagree about which estate is prod")
|
||
|
||
data_paths = env_values("LONGHORN_DATA_PATH")
|
||
replicas = env_values("LONGHORN_DEFAULT_REPLICA_COUNT")
|
||
if prod_overlay:
|
||
p = data_paths.get(prod_overlay, data_paths.get("base", ""))
|
||
check(p.startswith("/"),
|
||
f"prod's LONGHORN_DATA_PATH is {p!r} — R-ENV-03's 'local storage' requires an absolute "
|
||
"node-local filesystem path")
|
||
check(not re.match(r"^/(net|nfs|mnt/nfs)|:", p),
|
||
f"prod's LONGHORN_DATA_PATH {p!r} looks like a network mount, not local SSD/NVMe")
|
||
r = replicas.get(prod_overlay, replicas.get("base", ""))
|
||
check(r.isdigit() and int(r) == len(prod_names),
|
||
f"prod's LONGHORN_DEFAULT_REPLICA_COUNT is {r!r} for {len(prod_names)} node(s) — "
|
||
"'local storage' on a 3-node estate means one replica per node's own disk")
|
||
# The DEFAULT storage class must be the local (Longhorn) driver. A separate bulk/media tier
|
||
# on network storage is explicitly a different requirement (R-STATE-04, docs/04) and is not
|
||
# counted here.
|
||
default_scs = []
|
||
for f in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z", "*.yaml", "*.yml"],
|
||
capture_output=True, text=True, check=True).stdout.split("\0"):
|
||
if not f:
|
||
continue
|
||
try:
|
||
for d in yaml.safe_load_all((ROOT / f).read_text(errors="replace")):
|
||
if isinstance(d, dict) and d.get("kind") == "StorageClass" and \
|
||
str(((d.get("metadata") or {}).get("annotations") or {})
|
||
.get("storageclass.kubernetes.io/is-default-class", "")) == "true":
|
||
default_scs.append((f, str(d.get("provisioner", ""))))
|
||
except Exception:
|
||
continue
|
||
check(len(default_scs) == 1,
|
||
f"expected exactly one default StorageClass, found {default_scs} — the tier prod's workloads "
|
||
"land on by default must be unambiguous")
|
||
for f, prov in default_scs:
|
||
check(prov == "driver.longhorn.io",
|
||
f"{f}: the default StorageClass provisioner is {prov!r}, not Longhorn — prod's default "
|
||
"tier would not be its own local disks")
|
||
|
||
# ------------------------------------- (4) prod runs the SAME bootstrap code, retargeted
|
||
PLAYBOOKS = ANS / "playbooks"
|
||
ROLES = ANS / "roles"
|
||
prod_token = PROD_GROUP.replace("k3s_", "")
|
||
for d, what in ((ROLES, "role"), (PLAYBOOKS, "playbook")):
|
||
for entry in sorted(d.iterdir()) if d.is_dir() else []:
|
||
check(prod_token not in entry.name.lower(),
|
||
f"infra/ansible/{d.name}/{entry.name} is a prod-specific {what} — docs/02 requires "
|
||
"the SAME Ansible bootstrap for prod, differing only in the machine-creation step")
|
||
# Every prod group_vars key must be a variable the shared code already knows: a role default,
|
||
# a global, or an ansible built-in. An invented key is a fork of the role by another name.
|
||
known: set[str] = set(ALL_VARS)
|
||
for rd in sorted(ROLES.iterdir()) if ROLES.is_dir() else []:
|
||
for p in (rd / "defaults").glob("*.yml") if (rd / "defaults").is_dir() else []:
|
||
try:
|
||
known |= set((yaml.safe_load(p.read_text()) or {}).keys())
|
||
except Exception:
|
||
pass
|
||
check(len(known) > 20, f"only {len(known)} shared variables found — (4) would prove nothing")
|
||
for k in sorted(PROD_VARS):
|
||
check(k.startswith("ansible_") or k in known,
|
||
f"prod group_vars sets {k!r}, which is not a role default or a global — prod-only "
|
||
"configuration that the dev path never exercises is developing against prod")
|
||
# ...and it must retarget, not re-implement: the group it names must exist in the inventory.
|
||
check(PROD_PRIMARY in ini_groups or PROD_PRIMARY in children,
|
||
f"prod group_vars names primary group {PROD_PRIMARY!r}, which the inventory does not define")
|
||
|
||
# ---------------------------------------- (5) prod is not a development surface (overlay)
|
||
if prod_overlay:
|
||
odir = ROOT / "clusters" / prod_overlay
|
||
files = sorted(p.name for p in odir.iterdir() if p.is_file())
|
||
check(files == ["kustomization.yaml", "sync.yaml"],
|
||
f"clusters/{prod_overlay}/ contains {files} — a prod overlay may hold only its "
|
||
"kustomization and its own root sync; anything else is configuration authored FOR PROD")
|
||
k = yaml.safe_load((odir / "kustomization.yaml").read_text()) or {}
|
||
check(sorted(str(r) for r in (k.get("resources") or [])) == ["../base", "sync.yaml"],
|
||
f"clusters/{prod_overlay} resources are {k.get('resources')} — prod must render the base "
|
||
"plus its own sync only (docs/02: 'apply the prod overlay (no base edits)')")
|
||
# Every patch must be an ENVIRONMENT-SCOPED override, never a change to what runs.
|
||
ALLOWED = {("ConfigMap", "cluster-vars"), ("GitRepository", None)}
|
||
for p in (k.get("patches") or []):
|
||
tgt = p.get("target") or {}
|
||
body = p.get("patch") or ""
|
||
kind = tgt.get("kind")
|
||
if kind is None: # inline patch: kind comes from the body
|
||
try:
|
||
doc = yaml.safe_load(body) or {}
|
||
except Exception:
|
||
doc = {}
|
||
kind = doc.get("kind")
|
||
name = (doc.get("metadata") or {}).get("name")
|
||
if kind == "Kustomization":
|
||
# Flux Kustomization patches are allowed ONLY to wire sops decryption —
|
||
# anything else (path, interval, images, values) changes what prod RUNS.
|
||
keys = set((doc.get("spec") or {}).keys())
|
||
check(keys <= {"decryption"},
|
||
f"clusters/{prod_overlay}: patch of Kustomization/{name} touches {sorted(keys)} "
|
||
"— only `decryption` (the per-cluster sops key) is an environment-scoped "
|
||
"override; changing anything else is developing against prod")
|
||
continue
|
||
check((kind, None) in ALLOWED or (kind, name) in ALLOWED,
|
||
f"clusters/{prod_overlay}: patch targets {kind}/{name}, which is not an "
|
||
"environment-scoped object — prod must not carry configuration of its own")
|
||
else:
|
||
check((kind, tgt.get("name")) in ALLOWED or (kind, None) in ALLOWED,
|
||
f"clusters/{prod_overlay}: patch targets {kind}/{tgt.get('name')}, which is not "
|
||
"an environment-scoped object — prod must not carry configuration of its own")
|
||
|
||
# Same Git ref as the other environment(s): a prod-only branch is a development line
|
||
# that only prod runs, which is exactly what R-ENV-03 forbids.
|
||
def git_ref(overlay: str):
|
||
kk = yaml.safe_load((ROOT / "clusters" / overlay / "kustomization.yaml").read_text()) or {}
|
||
for p in (kk.get("patches") or []):
|
||
body = p.get("patch") or ""
|
||
try:
|
||
doc = yaml.safe_load(body) or {}
|
||
except Exception:
|
||
continue
|
||
if doc.get("kind") == "GitRepository":
|
||
return (doc.get("spec") or {}).get("ref")
|
||
return None
|
||
|
||
others = [o.name for o in (ROOT / "clusters").iterdir()
|
||
if o.is_dir() and o.name not in ("base", prod_overlay)]
|
||
prod_ref = git_ref(prod_overlay)
|
||
check(prod_ref is not None,
|
||
f"clusters/{prod_overlay} does not pin a Git ref — prod would follow whatever the base "
|
||
"points at, with no record of what it runs")
|
||
for o in others:
|
||
check(git_ref(o) == prod_ref,
|
||
f"clusters/{o} tracks {git_ref(o)} but prod tracks {prod_ref} — a prod-only ref is a "
|
||
"development line only prod runs (R-ENV-03: prod is a target, not a sandbox)")
|
||
|
||
# No prod-only manifest anywhere in the rendered configuration.
|
||
tracked = [f for f in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
|
||
capture_output=True, text=True, check=True
|
||
).stdout.split("\0") if f]
|
||
RENDERED_DIRS = ("platform/", "platform-config", "apps/", "catalog/", "clusters/base/")
|
||
|
||
# THE ONE STRUCTURAL EXCEPTION: AN OVERLAY-SELECTED ENVIRONMENT HALF.
|
||
# ------------------------------------------------------------------
|
||
# A filename carrying an environment token is a HEURISTIC for the harm R-ENV-03 names
|
||
# (content only prod runs, in a path every environment renders). It is not the harm
|
||
# itself. There is exactly one shape where a `-prod` name is the OPPOSITE of a prod-only
|
||
# fork: a credential that is TWO Secret objects, BOTH applied to BOTH clusters as base
|
||
# resources, where a cluster-vars knob decides only which one the shared consumer
|
||
# REFERENCES (clusters/base/cluster-vars.yaml, BACKUP_CREDENTIALS_SUFFIX: dev backs up to
|
||
# MinIO, prod to Cloudflare R2, and one shared Secret meant either environment's rotation
|
||
# broke the other). Nothing is forked: every environment renders the identical manifests
|
||
# and the identical object set, which is precisely the R-ENV-02 contract.
|
||
#
|
||
# So the exception is not a path allowlist — it is the PROPERTY that makes that shape
|
||
# safe, asserted from the tree, and a file is excused ONLY when all of it holds:
|
||
#
|
||
# (A) it declares nothing but Secrets whose names end in the environment suffix — the
|
||
# only kind of object this repo has a selection mechanism for;
|
||
# (1) it is listed in its own directory's kustomization.yaml, so EVERY environment
|
||
# applies it, and every environment's value of the selector resolves to an object
|
||
# that some resource of that same kustomization declares (same namespace) — no
|
||
# environment is left selecting a Secret that does not exist;
|
||
# (2) nothing references the suffixed name literally: every reference is a substituted
|
||
# expression `<base-name>${VAR}`, and the suffixed FILE is named only by the
|
||
# kustomization that applies it — so no shared manifest hardcodes prod's half;
|
||
# (3) that VAR is defined for the base AND for every overlay (StrictPostBuildSubstitutions
|
||
# fails on an UNDEFINED variable, so an overlay that forgot it would break that
|
||
# cluster's reconciliation), and prod's value is the one that selects THIS file.
|
||
#
|
||
# References are read from PARSED YAML VALUES, never from raw text: the mechanism is
|
||
# documented by name in comments beside the resources (cluster-vars says so explicitly),
|
||
# and a name in a comment references nothing. Anything that fails any condition still
|
||
# fails, naming the condition it broke.
|
||
ENV_SUFFIX = f"-{prod_token}"
|
||
# Every environment that renders the shared configuration.
|
||
ENVS = ["base"] + sorted(p.parent.name
|
||
for p in (ROOT / "clusters").glob("*/kustomization.yaml")
|
||
if p.parent.name != "base")
|
||
|
||
|
||
def _docs(rel: str):
|
||
p = ROOT / rel
|
||
if not p.is_file():
|
||
return None
|
||
try:
|
||
return [d for d in yaml.safe_load_all(p.read_text(errors="replace"))
|
||
if isinstance(d, dict)]
|
||
except Exception:
|
||
return None
|
||
|
||
|
||
def _strings(node):
|
||
if isinstance(node, str):
|
||
yield node
|
||
elif isinstance(node, dict):
|
||
for k, v in node.items():
|
||
yield from _strings(k)
|
||
yield from _strings(v)
|
||
elif isinstance(node, (list, tuple)):
|
||
for v in node:
|
||
yield from _strings(v)
|
||
|
||
|
||
RENDERED_YAML = [f for f in tracked
|
||
if f.startswith(RENDERED_DIRS) and f.endswith((".yaml", ".yml"))]
|
||
VALUE_STRINGS = {f: list(_strings(_docs(f) or [])) for f in RENDERED_YAML}
|
||
|
||
|
||
def overlay_selected(f: str) -> list[str]:
|
||
"""[] if f is an overlay-SELECTED environment half; else the conditions it breaks."""
|
||
docs = _docs(f)
|
||
objs = [((d.get("kind")), (d.get("metadata") or {}).get("name"),
|
||
(d.get("metadata") or {}).get("namespace")) for d in (docs or [])]
|
||
if not objs or any(k != "Secret" or not isinstance(n, str) or not n.endswith(ENV_SUFFIX)
|
||
for k, n, _ in objs):
|
||
return [f"(A) it does not consist solely of Secret objects whose names end in "
|
||
f"{ENV_SUFFIX!r} — an overlay can select which SECRET a shared manifest "
|
||
f"references, and nothing else, so any other prod-suffixed object in a "
|
||
f"shared path is prod-only content"]
|
||
|
||
reasons: list[str] = []
|
||
d = str(Path(f).parent)
|
||
kfile = f"{d}/kustomization.yaml"
|
||
kdocs = _docs(kfile) or []
|
||
res = [str(r) for r in ((kdocs[0].get("resources") if kdocs else None) or [])]
|
||
if Path(f).name not in res:
|
||
reasons.append(f"(1) {kfile} does not list it in `resources`, so it is not applied "
|
||
f"in every environment — one environment would carry an object the "
|
||
f"other does not")
|
||
|
||
# Every Secret the applying kustomization declares: name -> namespace.
|
||
declared: dict[str, object] = {}
|
||
for r in res:
|
||
for dd in (_docs(f"{d}/{r}") or []):
|
||
if dd.get("kind") == "Secret":
|
||
md = dd.get("metadata") or {}
|
||
declared[str(md.get("name"))] = md.get("namespace")
|
||
|
||
for _, name, ns in objs:
|
||
base_name = name[:-len(ENV_SUFFIX)]
|
||
pat = re.escape(base_name) + r"\$\{([A-Za-z_][A-Za-z0-9_]*)\}"
|
||
sel = sorted({m for ss in VALUE_STRINGS.values() for s in ss
|
||
for m in re.findall(pat, s)})
|
||
# (2) literal references to the suffixed OBJECT — anywhere but this file, except
|
||
# the applying kustomization naming this FILE (which is how it gets applied).
|
||
lit = sorted({f2 for f2, ss in VALUE_STRINGS.items() for s in ss
|
||
if name in s and f2 != f
|
||
and not (f2 == kfile and s == Path(f).name)})
|
||
if lit:
|
||
reasons.append(f"(2) {', '.join(lit)} names the Secret {name!r} literally — a "
|
||
f"shared manifest that hardcodes prod's half makes every "
|
||
f"environment use prod's credential; reference "
|
||
f"{base_name}${{...}} instead")
|
||
# (2) literal references to the suffixed FILE from anywhere but its kustomization.
|
||
stray = sorted({f2 for f2, ss in VALUE_STRINGS.items() for s in ss
|
||
if Path(f).name in s and f2 not in (f, kfile)})
|
||
if stray:
|
||
reasons.append(f"(2) {', '.join(stray)} names the file {Path(f).name!r} "
|
||
f"literally — only {kfile} may apply it")
|
||
if not sel:
|
||
reasons.append(f"(2) no manifest references {base_name}${{...}}: nothing selects "
|
||
f"between the halves, so this file is prod-only content that "
|
||
f"only prod could ever consume")
|
||
continue
|
||
if len(sel) > 1:
|
||
reasons.append(f"(2) {base_name} is selected by more than one variable {sel} — "
|
||
f"they can drift apart and leave one consumer on the other "
|
||
f"environment's credential")
|
||
var = sel[0]
|
||
vals = env_values(var)
|
||
missing = [e for e in ENVS if e not in vals]
|
||
if missing:
|
||
reasons.append(f"(3) ${{{var}}} is not defined for {missing} (defined for "
|
||
f"{sorted(vals)}) — StrictPostBuildSubstitutions fails on an "
|
||
f"undefined variable, so that environment would stop "
|
||
f"reconciling, or silently inherit the other's credential")
|
||
for e, v in sorted(vals.items()):
|
||
want = base_name + v
|
||
if want not in declared:
|
||
reasons.append(f"(1) environment {e!r} selects Secret {want!r} via "
|
||
f"${{{var}}}={v!r}, which no resource of {kfile} declares — "
|
||
f"that environment renders a dangling reference")
|
||
elif declared[want] != ns:
|
||
reasons.append(f"(1) environment {e!r} selects Secret {want!r}, declared in "
|
||
f"namespace {declared[want]!r}, but this half is in {ns!r} — "
|
||
f"the halves are not interchangeable")
|
||
if prod_overlay and prod_overlay in vals and base_name + vals[prod_overlay] != name:
|
||
reasons.append(f"(3) the prod overlay's ${{{var}}}={vals[prod_overlay]!r} selects "
|
||
f"{base_name + vals[prod_overlay]!r}, not this file's {name!r} — "
|
||
f"prod's half is applied but no environment selects it")
|
||
return reasons
|
||
|
||
|
||
flagged = [f for f in tracked
|
||
if f.startswith(RENDERED_DIRS)
|
||
and re.search(rf"(^|[/_.-]){re.escape(prod_token)}([/_.-]|$)", f, re.I)]
|
||
for f in flagged:
|
||
why = overlay_selected(f)
|
||
check(not why,
|
||
f"{f} is a prod-specific file inside the shared configuration — every environment "
|
||
f"renders these paths, so prod-only content here is development against prod. It is "
|
||
f"not an overlay-selected environment half either: " + " AND ".join(why))
|
||
if not why:
|
||
note(f"{f} carries the {prod_token!r} token but is an OVERLAY-SELECTED half: both "
|
||
f"halves are applied in every environment and only the cluster-vars selector "
|
||
f"differs (R-ENV-02), so it is not prod-only content")
|
||
check(len(tracked) > 50, f"git ls-files returned only {len(tracked)} paths — not a real checkout")
|
||
|
||
# --------------------------- (6) prod work exists only with a declaration of readiness
|
||
# In-repo: the operator's recorded decision to open the prod phase.
|
||
log = ROOT / "DECISIONS-LOG.md"
|
||
check(log.is_file() and re.search(r"PHASE\s*7\b.*OPEN", log.read_text(errors="replace"), re.I),
|
||
"DECISIONS-LOG.md carries no record of the operator OPENING the prod phase, yet prod "
|
||
"inventory/overlay exist — R-ENV-03 allows prod build only once the roadmap declares it ready")
|
||
# In the spec (read-only, may not be mounted): the roadmap phase that opens prod.
|
||
spec = Path(os.environ.get("SPEC_REPO", ""))
|
||
phases = spec / "roadmap" / "phases.md"
|
||
if phases.is_file():
|
||
txt = phases.read_text(errors="replace")
|
||
check(re.search(r"##\s*Phase\s*7\b", txt) is not None and
|
||
re.search(r"prod", txt, re.I) is not None,
|
||
"the spec roadmap has no Phase 7 opening prod build — the declaration R-ENV-03 defers "
|
||
"to does not exist")
|
||
check(re.search(r"explicitly opened|declares prod hardware ready|operator declares", txt, re.I)
|
||
is not None,
|
||
"the spec roadmap's prod phase states no explicit entry declaration — the gate R-ENV-03 "
|
||
"names would be unenforceable")
|
||
else:
|
||
note(f"spec repo not mounted at {spec} — the roadmap half of (6) was NOT checked "
|
||
"(the in-repo declaration still was)")
|
||
|
||
# --------------------------- (4b) DRIVE the retarget offline: same chain, prod hosts
|
||
if shutil.which("ansible-playbook") is None:
|
||
note("ansible-playbook is not installed here (e.g. the in-cluster chainsaw image), so the "
|
||
"prod retarget was PARSED but not DRIVEN; the merge gate's container has it")
|
||
else:
|
||
sec_groups = [g for g in children.get(PROD_GROUP, []) if g != PROD_PRIMARY]
|
||
args = ["--list-hosts", "playbooks/site.yml",
|
||
"-e", f"k3s_hosts={PROD_GROUP}",
|
||
"-e", f"k3s_primary_hosts={PROD_PRIMARY}"]
|
||
if sec_groups:
|
||
args += ["-e", f"k3s_secondary_hosts={sec_groups[0]}"]
|
||
lh = subprocess.run(["ansible-playbook", *args], cwd=ANS, capture_output=True, text=True)
|
||
out = (lh.stderr or "") + (lh.stdout or "")
|
||
if lh.returncode != 0 and ENV_ONLY_FAILURE.search(out):
|
||
# See the same narrow, self-policing skip in dev-nodes-provisioned-as-code.
|
||
note("ansible-playbook lacks a LOCAL dependency kept outside the repo by design (the vault "
|
||
"passphrase and/or the gitignored Galaxy collections), so the prod retarget was not "
|
||
"driven; every parse assertion above still ran")
|
||
else:
|
||
check(lh.returncode == 0,
|
||
f"retargeting the SHARED chain at prod failed: {out.strip()[:400]} — docs/02 requires "
|
||
"the same Ansible bootstrap to run for prod")
|
||
seen: set[str] = set()
|
||
in_hosts = False
|
||
for line in lh.stdout.splitlines():
|
||
if re.match(r"\s*hosts \(\d+\):", line):
|
||
in_hosts = True
|
||
continue
|
||
if re.search(r"play #\d+", line):
|
||
in_hosts = False
|
||
continue
|
||
if in_hosts and line.strip():
|
||
seen.add(line.strip())
|
||
check(seen == set(prod_names),
|
||
f"the shared chain retargeted at prod resolves to {sorted(seen)}, not the three prod "
|
||
f"nodes {prod_names} — prod would be bootstrapped by something other than dev's path")
|
||
|
||
# --------------------------------------------------------------------------- verdict
|
||
check(CHECKS >= 25,
|
||
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
|
||
"would be meaningless")
|
||
|
||
if FAILURES:
|
||
print(f"CHECK FAIL (prod-baremetal-not-dev-surface): {len(FAILURES)} of {CHECKS} assertions failed",
|
||
file=sys.stderr)
|
||
for f in FAILURES:
|
||
print(f" - {f}", file=sys.stderr)
|
||
raise SystemExit(1)
|
||
for n in NOTES:
|
||
print(f" note: {n}")
|
||
print(f"check OK: prod-baremetal-not-dev-surface — {CHECKS} assertions; prod group {PROD_GROUP!r} = "
|
||
f"{prod_names} (3 nodes, none created by IaC), overlay clusters/{prod_overlay} renders the base "
|
||
f"plus environment-scoped patches only, local Longhorn path "
|
||
f"{data_paths.get(prod_overlay, data_paths.get('base'))!r} with "
|
||
f"{replicas.get(prod_overlay, replicas.get('base'))} replica(s). R-ENV-03's AUTHORED clauses "
|
||
"only — the hardware facts (mini-PC, bare metal, local NVMe) are operator-attested and the "
|
||
"live cluster's shape is NOT asserted here (see the header's gap list).")
|
||
PY
|