Files
home-network-k3s-impl/conformance/tests/l0-static/prod-baremetal-not-dev-surface/check.sh
T
jakman85andClaude Opus 5 6a8953e288 feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.

Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.

WHAT THIS FIXES

1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
   ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
   anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
   to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
   volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
   database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
   the hour, and evaluates to 3 firing series on prod today.
   Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
   nextcloud-db's exporter cannot log in to its own database (permission denied for
   database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
   for it — the obvious metric would have been blind to one database in three.

2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
   by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
   job="kubelet" (the apiserver copy of the same series reports namespace=default),
   and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
   available=0 forever and would otherwise have fired critical permanently.

3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
   firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
   a node carries the egress-gateway label, with dev's behaviour unchanged today.

4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
   LITERALLY and the import still reports success, then sits at Done=True and never
   re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
   $(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
   green. The CronJob is the level-triggered correction; two alerts make its silence
   observable. Repaired live already — this stops it recurring.

Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 00:59:22 -05:00

529 lines
27 KiB
Bash
Executable File
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — prod-baremetal-not-dev-surface
# Serves: R-ENV-03 ("prod is a 3× mini-PC bare-metal k3s cluster with local storage; it
# MUST NOT be an active development surface until the roadmap declares it ready."
# — spec docs/00-vision-and-scope.md. Detail in docs/02: prod substrate "3× mini-PC
# bare metal", "3 physical nodes", storage "local SSD/NVMe presented to Longhorn",
# "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation
# step differs", and "**Never develop against prod.** Prod hardware is a target, not
# a sandbox (R-ENV-03)". The roadmap gate is roadmap/phases.md Phase 7: entry
# "operator declares prod hardware ready"; note "Do not begin building on prod
# hardware before this phase is explicitly opened".)
#
# HOW A "MUST NOT" IS MADE TESTABLE
# --------------------------------
# The second clause is a governance rule, and the same discipline applies as to R-TEST-06
# (tests/l0-static/spec-immutability-enforced): "nobody developed against prod today"
# cannot be observed, but the PROPERTY THAT MAKES DEVELOPING AGAINST PROD IMPOSSIBLE can
# be. In this repo that property is structural — prod receives the base configuration plus
# environment-scoped overrides and NOTHING ELSE, so there is no place to put prod-only work
# even if someone wanted to. This check asserts that, together with the first clause's
# authored facts (three nodes, no VM IaC behind them, node-local storage, the same
# bootstrap code as dev).
#
# The first clause's HARDWARE facts (that the three machines really are mini-PCs, really
# are bare metal, really hold local NVMe) are operator-attested, not machine-checkable —
# see the gap list. What IS checkable is that the repository describes prod that way and
# that no IaC creates prod nodes as virtual machines.
#
# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted),
# WHEN the prod inventory, the prod overlay, the IaC module and the roadmap record are
# inspected — and, where offline execution is possible, the provisioning chain is
# actually retargeted at prod with `--list-hosts`,
# THEN (1) prod resolves to EXACTLY THREE nodes, reached directly on their own sshd
# (no NAT indirection, i.e. real machines on the LAN);
# (2) NO IaC resource creates any prod node: the VM module's generated node names are
# disjoint from the prod host set and no prod hostname appears anywhere under
# infra/tofu — prod is imaged by the operator (docs/02 stage 1), not instantiated;
# (3) prod's storage is NODE-LOCAL: the Longhorn data path for prod is an absolute
# local path, and the replica count equals the node count (one replica per node's
# own disk). A distinct bulk/media tier is explicitly permitted (R-STATE-04) and
# is not counted against this;
# (4) prod runs THE SAME bootstrap code as dev: the same playbooks/roles, retargeted
# by variables only — every prod group_vars key is an existing role/global
# variable (an invented prod-only knob would be a fork), and no prod-only role
# or playbook exists;
# (5) prod is NOT a development surface: clusters/prod carries only the base plus
# sync.yaml, every patch it applies targets an ENVIRONMENT-SCOPED object
# (cluster-vars, a Flux Kustomization's sops decryption, or the Git source),
# prod's Git ref is the SAME ref dev tracks (no prod-only branch to develop on),
# and no prod-only manifest exists anywhere in the rendered configuration;
# (6) prod artifacts exist only WITH a declaration of readiness: the spec roadmap
# carries the phase that opens prod build (checked when the spec repo is mounted),
# and the repository records that the operator opened it.
#
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them):
# - The HARDWARE. That the three nodes are mini-PCs, are bare metal, and present local
# SSD/NVMe is operator-attested in infra/PROD-STANDUP.md / PROD-PROVISIONING.md; the
# node-health and BIOS surveys that establish it write to gitignored artifacts, so
# nothing tracked can be asserted against. Cluster-side, the Kubernetes API exposes no
# "am I virtual" fact either — a bare-metal check would need a privileged host-path
# probe of DMI data, which is a heavier and more invasive test than the claim warrants.
# - The LIVE prod cluster's shape (three Ready nodes, all control-plane+etcd, Longhorn
# disks at the local path). That is cluster state and belongs to R-PLAT-01's
# (currently uncovered) HA test; asserting it here would also make this test
# environment-specific, since the same suite must pass against dev.
# - That prod's HISTORY contains no development. This is the working tree at HEAD.
# - Whether the roadmap SHOULD have opened prod (the entry-gate judgement). Only that
# prod work is accompanied by the operator's recorded declaration.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
: "${SPEC_REPO:=${ROOT}/../home-network-k3s}"
export SPEC_REPO
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
import os
import re
import shutil
import subprocess
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
FAILURES: list[str] = []
NOTES: list[str] = []
CHECKS = 0
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
def note(msg):
NOTES.append(msg)
# A LOCAL DEPENDENCY the repo deliberately does not carry — the vault passphrase (kept
# outside the repo, R-SEC-01) or the Galaxy collections (gitignored; installed from
# collections/requirements.yml). Their absence is an ENVIRONMENT gap, not a repo defect,
# and is the ONLY tolerated reason the driven sub-assertions may not run. Deliberately
# narrow: every other non-zero exit is a real failure. The parse assertions always run.
ENV_ONLY_FAILURE = re.compile(
r"vault password|vault-id|Attempting to decrypt"
r"|No module named 'ansible_collections|couldn't resolve module/action", re.I)
ANS = ROOT / "infra" / "ansible"
INV = ANS / "inventory" / "hosts.ini"
GV = ANS / "inventory" / "group_vars"
TOFU = ROOT / "infra" / "tofu"
check(INV.is_file(), "infra/ansible/inventory/hosts.ini is missing — prod is not described anywhere")
if not INV.is_file():
print("CHECK FAIL (prod-baremetal-not-dev-surface): no inventory", file=sys.stderr)
raise SystemExit(1)
# --------------------------------------------------------------- inventory parsing
ini_groups: dict[str, list[dict]] = {}
children: dict[str, list[str]] = {}
cur = None
for line in INV.read_text().splitlines():
s = line.strip()
if not s or s.startswith("#"):
continue
m = re.match(r"^\[([^\]]+)\]$", s)
if m:
cur = m.group(1)
continue
if cur is None or cur.endswith(":vars"):
continue
if cur.endswith(":children"):
children.setdefault(cur.split(":")[0], []).append(s.split()[0])
continue
parts = s.split()
host = {"name": parts[0]}
for kv in parts[1:]:
if "=" in kv:
k, v = kv.split("=", 1)
host[k] = v
ini_groups.setdefault(cur, []).append(host)
def hosts_in(group: str, seen=None) -> list[dict]:
seen = seen or set()
if group in seen:
return []
seen.add(group)
out = list(ini_groups.get(group, []))
for child in children.get(group, []):
out += hosts_in(child, seen)
return out
def load_gv(group: str) -> dict:
d: dict = {}
for p in [GV / f"{group}.yml", GV / group / "vars.yml"]:
if p.is_file():
try:
d.update(yaml.safe_load(p.read_text()) or {})
except Exception:
pass
return d
# WHICH GROUP IS PROD? Derived, never named here: it is the group whose group_vars
# RETARGETS the k3s roles (k3s_primary_group / k3s_servers_group) away from the global
# dev default. That is exactly the seam docs/02 + infra/ansible/README describe for prod.
ALL_VARS = load_gv("all")
dev_primary_group = str(ALL_VARS.get("k3s_primary_group", ""))
candidates = []
for entry in sorted(GV.iterdir()) if GV.is_dir() else []:
g = entry.name[:-4] if entry.name.endswith(".yml") else entry.name
if g == "all":
continue
v = load_gv(g)
if v.get("k3s_primary_group") and str(v["k3s_primary_group"]) != dev_primary_group:
candidates.append((g, v))
check(len(candidates) == 1,
f"expected exactly ONE environment whose group_vars retargets the k3s roles (prod); found "
f"{[c[0] for c in candidates]} — R-ENV-03 describes a single prod estate reached through "
"the same roles as dev")
if not candidates:
print("CHECK FAIL (prod-baremetal-not-dev-surface): no prod group_vars found", file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
PROD_GROUP, PROD_VARS = candidates[0]
PROD_PRIMARY = str(PROD_VARS["k3s_primary_group"])
prod_hosts = hosts_in(PROD_GROUP)
prod_names = sorted(h["name"] for h in prod_hosts)
# ------------------------------------------------- (1) exactly three, directly reachable
check(len(prod_names) == 3,
f"prod group {PROD_GROUP!r} resolves to {len(prod_names)} host(s) {prod_names} — R-ENV-03 "
"says prod is a 3× mini-PC cluster")
dev_hosts = hosts_in(str(ALL_VARS.get("k3s_servers_group", "k3s_servers")))
for h in prod_hosts:
# Real machines on the LAN answer their own sshd. Dev's hosts are reached through a
# NAT port-forward (ansible_port 225x -> a VM's :22), which is precisely the
# indirection a hypervisor-hosted estate needs and bare metal does not.
check(str(h.get("ansible_port", "22")) == "22",
f"prod node {h['name']} is reached on port {h.get('ansible_port')} — a forwarded port "
"implies an indirection (NAT to a virtual estate), not a machine on the LAN")
check(h.get("k3s_node_ip") == h.get("ansible_host"),
f"prod node {h['name']} has ansible_host={h.get('ansible_host')} but "
f"k3s_node_ip={h.get('k3s_node_ip')} — a split address is the dev NAT pattern; a "
"bare-metal node's SSH address IS its cluster address")
# ------------------------------------------------------ (2) nothing creates prod as a VM
TF = "\n".join(p.read_text(errors="replace") for p in sorted(TOFU.glob("*.tf"))) \
if TOFU.is_dir() else ""
check(len(TF) > 500, "infra/tofu/*.tf is empty — cannot show that it does NOT create prod")
def tf_default(var: str):
m = re.search(r'variable\s+"' + re.escape(var) + r'"\s*\{(.*?)\n\}', TF, re.S)
if not m:
return None
d = re.search(r"default\s*=\s*(.+)", m.group(1))
return d.group(1).strip() if d else None
prefix = (tf_default("node_name_prefix") or "").strip('"')
try:
node_count = int((tf_default("node_count") or "0").strip())
except ValueError:
node_count = 0
iac_names = {f"{prefix}{i + 1}" for i in range(node_count)}
check(len(iac_names) > 0, "the IaC module generates no node names — cannot compare with prod")
check(iac_names.isdisjoint(set(prod_names)),
f"the IaC VM module would create {sorted(iac_names & set(prod_names))}, which are PROD nodes "
"— prod is bare metal that the operator images (docs/02 stage 1), never instantiated by the "
"hypervisor module")
for name in prod_names:
check(name not in TF,
f"prod node {name} is named inside infra/tofu — prod hardware must not be described by "
"the VM module at all")
# The prod host addresses must not appear as IaC node addresses either (a renamed VM is
# still a VM).
tf_ips = set(re.findall(r'"(\d+\.\d+\.\d+\.\d+)"', tf_default("node_ips") or ""))
prod_ips = {str(h.get("ansible_host")) for h in prod_hosts}
check(tf_ips.isdisjoint(prod_ips),
f"the IaC module addresses {sorted(tf_ips & prod_ips)}, which are prod node addresses")
# --------------------------------------------------------------- (3) node-local storage
# Every environment's value for a cluster-vars knob (base default + each overlay patch).
def env_values(key: str) -> dict[str, str]:
vals: dict[str, str] = {}
base = ROOT / "clusters" / "base" / "cluster-vars.yaml"
if base.is_file():
for d in yaml.safe_load_all(base.read_text()):
if isinstance(d, dict) and d.get("kind") == "ConfigMap" \
and (d.get("metadata") or {}).get("name") == "cluster-vars":
v = (d.get("data") or {}).get(key)
if v is not None:
vals["base"] = str(v)
for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
env = overlay.parent.name
if env == "base":
continue
try:
k = yaml.safe_load(overlay.read_text()) or {}
except Exception:
continue
for p in (k.get("patches") or []):
tgt = p.get("target") or {}
if tgt.get("kind") != "ConfigMap" or tgt.get("name") != "cluster-vars":
continue
try:
ops = yaml.safe_load(p.get("patch") or "") or []
except Exception:
continue
for op in ops if isinstance(ops, list) else []:
if isinstance(op, dict) and str(op.get("path", "")) == f"/data/{key}":
vals[env] = str(op.get("value"))
return vals
# The prod OVERLAY is the one whose CLUSTER_ENV is not the base default.
cluster_envs = env_values("CLUSTER_ENV")
prod_overlays = [e for e, v in cluster_envs.items()
if e != "base" and v != cluster_envs.get("base")]
check(len(prod_overlays) >= 1,
"no clusters/ overlay sets its own CLUSTER_ENV — prod has no overlay of its own")
prod_overlay = None
for e in prod_overlays:
if PROD_GROUP.replace("k3s_", "") in cluster_envs[e] or cluster_envs[e] in PROD_GROUP:
prod_overlay = e
check(prod_overlay is not None,
f"no overlay's CLUSTER_ENV corresponds to the prod inventory group {PROD_GROUP!r} — the "
"cluster config and the machine inventory disagree about which estate is prod")
data_paths = env_values("LONGHORN_DATA_PATH")
replicas = env_values("LONGHORN_DEFAULT_REPLICA_COUNT")
if prod_overlay:
p = data_paths.get(prod_overlay, data_paths.get("base", ""))
check(p.startswith("/"),
f"prod's LONGHORN_DATA_PATH is {p!r} — R-ENV-03's 'local storage' requires an absolute "
"node-local filesystem path")
check(not re.match(r"^/(net|nfs|mnt/nfs)|:", p),
f"prod's LONGHORN_DATA_PATH {p!r} looks like a network mount, not local SSD/NVMe")
r = replicas.get(prod_overlay, replicas.get("base", ""))
check(r.isdigit() and int(r) == len(prod_names),
f"prod's LONGHORN_DEFAULT_REPLICA_COUNT is {r!r} for {len(prod_names)} node(s) — "
"'local storage' on a 3-node estate means one replica per node's own disk")
# The DEFAULT storage class must be the local (Longhorn) driver. A separate bulk/media tier
# on network storage is explicitly a different requirement (R-STATE-04, docs/04) and is not
# counted here.
default_scs = []
for f in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z", "*.yaml", "*.yml"],
capture_output=True, text=True, check=True).stdout.split("\0"):
if not f:
continue
try:
for d in yaml.safe_load_all((ROOT / f).read_text(errors="replace")):
if isinstance(d, dict) and d.get("kind") == "StorageClass" and \
str(((d.get("metadata") or {}).get("annotations") or {})
.get("storageclass.kubernetes.io/is-default-class", "")) == "true":
default_scs.append((f, str(d.get("provisioner", ""))))
except Exception:
continue
check(len(default_scs) == 1,
f"expected exactly one default StorageClass, found {default_scs} — the tier prod's workloads "
"land on by default must be unambiguous")
for f, prov in default_scs:
check(prov == "driver.longhorn.io",
f"{f}: the default StorageClass provisioner is {prov!r}, not Longhorn — prod's default "
"tier would not be its own local disks")
# ------------------------------------- (4) prod runs the SAME bootstrap code, retargeted
PLAYBOOKS = ANS / "playbooks"
ROLES = ANS / "roles"
prod_token = PROD_GROUP.replace("k3s_", "")
for d, what in ((ROLES, "role"), (PLAYBOOKS, "playbook")):
for entry in sorted(d.iterdir()) if d.is_dir() else []:
check(prod_token not in entry.name.lower(),
f"infra/ansible/{d.name}/{entry.name} is a prod-specific {what} — docs/02 requires "
"the SAME Ansible bootstrap for prod, differing only in the machine-creation step")
# Every prod group_vars key must be a variable the shared code already knows: a role default,
# a global, or an ansible built-in. An invented key is a fork of the role by another name.
known: set[str] = set(ALL_VARS)
for rd in sorted(ROLES.iterdir()) if ROLES.is_dir() else []:
for p in (rd / "defaults").glob("*.yml") if (rd / "defaults").is_dir() else []:
try:
known |= set((yaml.safe_load(p.read_text()) or {}).keys())
except Exception:
pass
check(len(known) > 20, f"only {len(known)} shared variables found — (4) would prove nothing")
for k in sorted(PROD_VARS):
check(k.startswith("ansible_") or k in known,
f"prod group_vars sets {k!r}, which is not a role default or a global — prod-only "
"configuration that the dev path never exercises is developing against prod")
# ...and it must retarget, not re-implement: the group it names must exist in the inventory.
check(PROD_PRIMARY in ini_groups or PROD_PRIMARY in children,
f"prod group_vars names primary group {PROD_PRIMARY!r}, which the inventory does not define")
# ---------------------------------------- (5) prod is not a development surface (overlay)
if prod_overlay:
odir = ROOT / "clusters" / prod_overlay
files = sorted(p.name for p in odir.iterdir() if p.is_file())
check(files == ["kustomization.yaml", "sync.yaml"],
f"clusters/{prod_overlay}/ contains {files} — a prod overlay may hold only its "
"kustomization and its own root sync; anything else is configuration authored FOR PROD")
k = yaml.safe_load((odir / "kustomization.yaml").read_text()) or {}
check(sorted(str(r) for r in (k.get("resources") or [])) == ["../base", "sync.yaml"],
f"clusters/{prod_overlay} resources are {k.get('resources')} — prod must render the base "
"plus its own sync only (docs/02: 'apply the prod overlay (no base edits)')")
# Every patch must be an ENVIRONMENT-SCOPED override, never a change to what runs.
ALLOWED = {("ConfigMap", "cluster-vars"), ("GitRepository", None)}
for p in (k.get("patches") or []):
tgt = p.get("target") or {}
body = p.get("patch") or ""
kind = tgt.get("kind")
if kind is None: # inline patch: kind comes from the body
try:
doc = yaml.safe_load(body) or {}
except Exception:
doc = {}
kind = doc.get("kind")
name = (doc.get("metadata") or {}).get("name")
if kind == "Kustomization":
# Flux Kustomization patches are allowed ONLY to wire sops decryption —
# anything else (path, interval, images, values) changes what prod RUNS.
keys = set((doc.get("spec") or {}).keys())
check(keys <= {"decryption"},
f"clusters/{prod_overlay}: patch of Kustomization/{name} touches {sorted(keys)} "
"— only `decryption` (the per-cluster sops key) is an environment-scoped "
"override; changing anything else is developing against prod")
continue
check((kind, None) in ALLOWED or (kind, name) in ALLOWED,
f"clusters/{prod_overlay}: patch targets {kind}/{name}, which is not an "
"environment-scoped object — prod must not carry configuration of its own")
else:
check((kind, tgt.get("name")) in ALLOWED or (kind, None) in ALLOWED,
f"clusters/{prod_overlay}: patch targets {kind}/{tgt.get('name')}, which is not "
"an environment-scoped object — prod must not carry configuration of its own")
# Same Git ref as the other environment(s): a prod-only branch is a development line
# that only prod runs, which is exactly what R-ENV-03 forbids.
def git_ref(overlay: str):
kk = yaml.safe_load((ROOT / "clusters" / overlay / "kustomization.yaml").read_text()) or {}
for p in (kk.get("patches") or []):
body = p.get("patch") or ""
try:
doc = yaml.safe_load(body) or {}
except Exception:
continue
if doc.get("kind") == "GitRepository":
return (doc.get("spec") or {}).get("ref")
return None
others = [o.name for o in (ROOT / "clusters").iterdir()
if o.is_dir() and o.name not in ("base", prod_overlay)]
prod_ref = git_ref(prod_overlay)
check(prod_ref is not None,
f"clusters/{prod_overlay} does not pin a Git ref — prod would follow whatever the base "
"points at, with no record of what it runs")
for o in others:
check(git_ref(o) == prod_ref,
f"clusters/{o} tracks {git_ref(o)} but prod tracks {prod_ref} — a prod-only ref is a "
"development line only prod runs (R-ENV-03: prod is a target, not a sandbox)")
# No prod-only manifest anywhere in the rendered configuration.
tracked = [f for f in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
capture_output=True, text=True, check=True
).stdout.split("\0") if f]
RENDERED_DIRS = ("platform/", "platform-config", "apps/", "catalog/", "clusters/base/")
for f in tracked:
if f.startswith(RENDERED_DIRS) and re.search(rf"(^|[/_.-]){re.escape(prod_token)}([/_.-]|$)",
f, re.I):
check(False,
f"{f} is a prod-specific file inside the shared configuration — every environment "
"renders these paths, so prod-only content here is development against prod")
check(len(tracked) > 50, f"git ls-files returned only {len(tracked)} paths — not a real checkout")
# --------------------------- (6) prod work exists only with a declaration of readiness
# In-repo: the operator's recorded decision to open the prod phase.
log = ROOT / "DECISIONS-LOG.md"
check(log.is_file() and re.search(r"PHASE\s*7\b.*OPEN", log.read_text(errors="replace"), re.I),
"DECISIONS-LOG.md carries no record of the operator OPENING the prod phase, yet prod "
"inventory/overlay exist — R-ENV-03 allows prod build only once the roadmap declares it ready")
# In the spec (read-only, may not be mounted): the roadmap phase that opens prod.
spec = Path(os.environ.get("SPEC_REPO", ""))
phases = spec / "roadmap" / "phases.md"
if phases.is_file():
txt = phases.read_text(errors="replace")
check(re.search(r"##\s*Phase\s*7\b", txt) is not None and
re.search(r"prod", txt, re.I) is not None,
"the spec roadmap has no Phase 7 opening prod build — the declaration R-ENV-03 defers "
"to does not exist")
check(re.search(r"explicitly opened|declares prod hardware ready|operator declares", txt, re.I)
is not None,
"the spec roadmap's prod phase states no explicit entry declaration — the gate R-ENV-03 "
"names would be unenforceable")
else:
note(f"spec repo not mounted at {spec} — the roadmap half of (6) was NOT checked "
"(the in-repo declaration still was)")
# --------------------------- (4b) DRIVE the retarget offline: same chain, prod hosts
if shutil.which("ansible-playbook") is None:
note("ansible-playbook is not installed here (e.g. the in-cluster chainsaw image), so the "
"prod retarget was PARSED but not DRIVEN; the merge gate's container has it")
else:
sec_groups = [g for g in children.get(PROD_GROUP, []) if g != PROD_PRIMARY]
args = ["--list-hosts", "playbooks/site.yml",
"-e", f"k3s_hosts={PROD_GROUP}",
"-e", f"k3s_primary_hosts={PROD_PRIMARY}"]
if sec_groups:
args += ["-e", f"k3s_secondary_hosts={sec_groups[0]}"]
lh = subprocess.run(["ansible-playbook", *args], cwd=ANS, capture_output=True, text=True)
out = (lh.stderr or "") + (lh.stdout or "")
if lh.returncode != 0 and ENV_ONLY_FAILURE.search(out):
# See the same narrow, self-policing skip in dev-nodes-provisioned-as-code.
note("ansible-playbook lacks a LOCAL dependency kept outside the repo by design (the vault "
"passphrase and/or the gitignored Galaxy collections), so the prod retarget was not "
"driven; every parse assertion above still ran")
else:
check(lh.returncode == 0,
f"retargeting the SHARED chain at prod failed: {out.strip()[:400]} — docs/02 requires "
"the same Ansible bootstrap to run for prod")
seen: set[str] = set()
in_hosts = False
for line in lh.stdout.splitlines():
if re.match(r"\s*hosts \(\d+\):", line):
in_hosts = True
continue
if re.search(r"play #\d+", line):
in_hosts = False
continue
if in_hosts and line.strip():
seen.add(line.strip())
check(seen == set(prod_names),
f"the shared chain retargeted at prod resolves to {sorted(seen)}, not the three prod "
f"nodes {prod_names} — prod would be bootstrapped by something other than dev's path")
# --------------------------------------------------------------------------- verdict
check(CHECKS >= 25,
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
"would be meaningless")
if FAILURES:
print(f"CHECK FAIL (prod-baremetal-not-dev-surface): {len(FAILURES)} of {CHECKS} assertions failed",
file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
for n in NOTES:
print(f" note: {n}")
print(f"check OK: prod-baremetal-not-dev-surface — {CHECKS} assertions; prod group {PROD_GROUP!r} = "
f"{prod_names} (3 nodes, none created by IaC), overlay clusters/{prod_overlay} renders the base "
f"plus environment-scoped patches only, local Longhorn path "
f"{data_paths.get(prod_overlay, data_paths.get('base'))!r} with "
f"{replicas.get(prod_overlay, replicas.get('base'))} replica(s). R-ENV-03's AUTHORED clauses "
"only — the hardware facts (mini-PC, bare metal, local NVMe) are operator-attested and the "
"live cluster's shape is NOT asserted here (see the header's gap list).")
PY