Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing

Browse Source 
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.

Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.

WHAT THIS FIXES

1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
   ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
   anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
   to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
   volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
   database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
   the hour, and evaluates to 3 firing series on prod today.
   Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
   nextcloud-db's exporter cannot log in to its own database (permission denied for
   database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
   for it — the obvious metric would have been blind to one database in three.

2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
   by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
   job="kubelet" (the apiserver copy of the same series reports namespace=default),
   and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
   available=0 forever and would otherwise have fired critical permanently.

3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
   firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
   a node carries the egress-gateway label, with dev's behaviour unchanged today.

4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
   LITERALLY and the import still reports success, then sits at Done=True and never
   re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
   $(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
   green. The CronJob is the level-triggered correction; two alerts make its silence
   observable. Repaired live already — this stops it recurring.

Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-07 00:59:22 -05:00
1 parent d1a6628a97
commit 6a8953e288
46 files changed
+7898 -71
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

conformance/tests/l0-static/prod-baremetal-not-dev-surface/chainsaw-test.yaml
+77
View File
@@ -0,0 +1,77 @@
# =============================================================================
# L0 STATIC — prod is bare metal, and is not a development surface (R-ENV-03) [EXECUTABLE]
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
# Spec: docs/00 R-ENV-03 ("prod is a 3× mini-PC bare-metal k3s cluster with local
# storage; it MUST NOT be an active development surface until the roadmap
# declares it ready"); docs/02 (prod = 3 physical nodes, local SSD/NVMe to
# Longhorn, "the *same* Ansible bootstrap runs; only the machine-creation step
# differs", "Never develop against prod"); roadmap/phases.md Phase 7.
#
# HOW A "MUST NOT" IS MADE TESTABLE — the same discipline as R-TEST-06's test: a
# governance rule is verified through the property that ENFORCES it, never by observing
# that nobody misbehaved today. Here that property is structural: prod receives the base
# configuration plus environment-scoped overrides and nothing else, from the same branch
# and the same Ansible code as dev, so there is no place to put prod-only work at all.
#
# ENVIRONMENT-AGNOSTIC: the prod inventory group is DERIVED (it is the group whose
# group_vars retargets the k3s roles away from the global default), the prod overlay is
# derived from CLUSTER_ENV, and the storage/replica expectations are compared against the
# node count that inventory resolves to. No hostname, address or environment name is
# written into the test.
#
# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted),
# WHEN the prod inventory, overlay, IaC module and readiness record are inspected — and
# the shared provisioning chain is actually retargeted at prod with --list-hosts,
# THEN prod resolves to exactly three nodes reached directly on their own sshd; no IaC
# resource creates any of them (bare metal is imaged, not instantiated); prod's
# Longhorn path is node-local with one replica per node and the default
# StorageClass is the local driver; prod runs the same playbooks/roles as dev,
# retargeted only by variables the shared code already defines; the prod overlay
# carries only the base plus environment-scoped patches on the same Git ref dev
# tracks, with no prod-only manifest anywhere in the shared configuration; and prod
# artifacts exist only alongside the operator's recorded opening of the prod phase.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git, PyYAML, and —
# where present — ansible-playbook in --list-hosts mode, which opens no connection);
# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test
# wraps the SAME script.
#
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
# * THE HARDWARE. That the three machines are mini-PCs, are bare metal, and hold local
# SSD/NVMe is operator-attested (infra/PROD-STANDUP.md, PROD-PROVISIONING.md); the
# node-health/BIOS surveys that establish it write to gitignored artifacts, and the
# Kubernetes API exposes no "am I virtual" fact, so nothing here asserts it.
# * The LIVE prod cluster's shape (three Ready nodes, all control-plane+etcd, Longhorn
# disks at the local path) — cluster state, belonging to R-PLAT-01's still-uncovered
# HA test; asserting it here would also make this test environment-specific, since
# the same suite must pass against dev.
# * Git HISTORY: this is the working tree at HEAD.
# * Whether the roadmap SHOULD have opened prod — only that prod work is accompanied by
# the recorded declaration.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: prod-baremetal-not-dev-surface
labels:
conformance.home.k3s/R-ENV-03: "true"
annotations:
conformance.home.k3s/requirements: "R-ENV-03"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Given the tracked tree, when prod's inventory, overlay, IaC and readiness record are
inspected, then prod is authored as three bare-metal nodes with node-local storage,
bootstrapped by dev's own code retargeted by variables, receiving only
environment-scoped overrides on the same branch — so there is nowhere to develop
against prod — and it exists only because the roadmap phase was opened.
spec:
steps:
- name: then-prod-is-bare-metal-and-carries-no-development-of-its-own
description: >-
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
conformance/run.sh --layer l0 execute).
try:
- script:
content: bash ./check.sh
conformance/tests/l0-static/prod-baremetal-not-dev-surface/check.sh
Executable
+528
View File
@@ -0,0 +1,528 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — prod-baremetal-not-dev-surface
# Serves: R-ENV-03 ("prod is a 3× mini-PC bare-metal k3s cluster with local storage; it
# MUST NOT be an active development surface until the roadmap declares it ready."
# — spec docs/00-vision-and-scope.md. Detail in docs/02: prod substrate "3× mini-PC
# bare metal", "3 physical nodes", storage "local SSD/NVMe presented to Longhorn",
# "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation
# step differs", and "**Never develop against prod.** Prod hardware is a target, not
# a sandbox (R-ENV-03)". The roadmap gate is roadmap/phases.md Phase 7: entry
# "operator declares prod hardware ready"; note "Do not begin building on prod
# hardware before this phase is explicitly opened".)
#
# HOW A "MUST NOT" IS MADE TESTABLE
# --------------------------------
# The second clause is a governance rule, and the same discipline applies as to R-TEST-06
# (tests/l0-static/spec-immutability-enforced): "nobody developed against prod today"
# cannot be observed, but the PROPERTY THAT MAKES DEVELOPING AGAINST PROD IMPOSSIBLE can
# be. In this repo that property is structural — prod receives the base configuration plus
# environment-scoped overrides and NOTHING ELSE, so there is no place to put prod-only work
# even if someone wanted to. This check asserts that, together with the first clause's
# authored facts (three nodes, no VM IaC behind them, node-local storage, the same
# bootstrap code as dev).
#
# The first clause's HARDWARE facts (that the three machines really are mini-PCs, really
# are bare metal, really hold local NVMe) are operator-attested, not machine-checkable —
# see the gap list. What IS checkable is that the repository describes prod that way and
# that no IaC creates prod nodes as virtual machines.
#
# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted),
# WHEN the prod inventory, the prod overlay, the IaC module and the roadmap record are
# inspected — and, where offline execution is possible, the provisioning chain is
# actually retargeted at prod with `--list-hosts`,
# THEN (1) prod resolves to EXACTLY THREE nodes, reached directly on their own sshd
# (no NAT indirection, i.e. real machines on the LAN);
# (2) NO IaC resource creates any prod node: the VM module's generated node names are
# disjoint from the prod host set and no prod hostname appears anywhere under
# infra/tofu — prod is imaged by the operator (docs/02 stage 1), not instantiated;
# (3) prod's storage is NODE-LOCAL: the Longhorn data path for prod is an absolute
# local path, and the replica count equals the node count (one replica per node's
# own disk). A distinct bulk/media tier is explicitly permitted (R-STATE-04) and
# is not counted against this;
# (4) prod runs THE SAME bootstrap code as dev: the same playbooks/roles, retargeted
# by variables only — every prod group_vars key is an existing role/global
# variable (an invented prod-only knob would be a fork), and no prod-only role
# or playbook exists;
# (5) prod is NOT a development surface: clusters/prod carries only the base plus
# sync.yaml, every patch it applies targets an ENVIRONMENT-SCOPED object
# (cluster-vars, a Flux Kustomization's sops decryption, or the Git source),
# prod's Git ref is the SAME ref dev tracks (no prod-only branch to develop on),
# and no prod-only manifest exists anywhere in the rendered configuration;
# (6) prod artifacts exist only WITH a declaration of readiness: the spec roadmap
# carries the phase that opens prod build (checked when the spec repo is mounted),
# and the repository records that the operator opened it.
#
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them):
# - The HARDWARE. That the three nodes are mini-PCs, are bare metal, and present local
# SSD/NVMe is operator-attested in infra/PROD-STANDUP.md / PROD-PROVISIONING.md; the
# node-health and BIOS surveys that establish it write to gitignored artifacts, so
# nothing tracked can be asserted against. Cluster-side, the Kubernetes API exposes no
# "am I virtual" fact either — a bare-metal check would need a privileged host-path
# probe of DMI data, which is a heavier and more invasive test than the claim warrants.
# - The LIVE prod cluster's shape (three Ready nodes, all control-plane+etcd, Longhorn
# disks at the local path). That is cluster state and belongs to R-PLAT-01's
# (currently uncovered) HA test; asserting it here would also make this test
# environment-specific, since the same suite must pass against dev.
# - That prod's HISTORY contains no development. This is the working tree at HEAD.
# - Whether the roadmap SHOULD have opened prod (the entry-gate judgement). Only that
# prod work is accompanied by the operator's recorded declaration.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
: "${SPEC_REPO:=${ROOT}/../home-network-k3s}"
export SPEC_REPO
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
import os
import re
import shutil
import subprocess
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
FAILURES: list[str] = []
NOTES: list[str] = []
CHECKS = 0
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
def note(msg):
NOTES.append(msg)
# A LOCAL DEPENDENCY the repo deliberately does not carry — the vault passphrase (kept
# outside the repo, R-SEC-01) or the Galaxy collections (gitignored; installed from
# collections/requirements.yml). Their absence is an ENVIRONMENT gap, not a repo defect,
# and is the ONLY tolerated reason the driven sub-assertions may not run. Deliberately
# narrow: every other non-zero exit is a real failure. The parse assertions always run.
ENV_ONLY_FAILURE = re.compile(
r"vault password|vault-id|Attempting to decrypt"
r"|No module named 'ansible_collections|couldn't resolve module/action", re.I)
ANS = ROOT / "infra" / "ansible"
INV = ANS / "inventory" / "hosts.ini"
GV = ANS / "inventory" / "group_vars"
TOFU = ROOT / "infra" / "tofu"
check(INV.is_file(), "infra/ansible/inventory/hosts.ini is missing — prod is not described anywhere")
if not INV.is_file():
print("CHECK FAIL (prod-baremetal-not-dev-surface): no inventory", file=sys.stderr)
raise SystemExit(1)
# --------------------------------------------------------------- inventory parsing
ini_groups: dict[str, list[dict]] = {}
children: dict[str, list[str]] = {}
cur = None
for line in INV.read_text().splitlines():
s = line.strip()
if not s or s.startswith("#"):
continue
m = re.match(r"^\[([^\]]+)\]$", s)
if m:
cur = m.group(1)
continue
if cur is None or cur.endswith(":vars"):
continue
if cur.endswith(":children"):
children.setdefault(cur.split(":")[0], []).append(s.split()[0])
continue
parts = s.split()
host = {"name": parts[0]}
for kv in parts[1:]:
if "=" in kv:
k, v = kv.split("=", 1)
host[k] = v
ini_groups.setdefault(cur, []).append(host)
def hosts_in(group: str, seen=None) -> list[dict]:
seen = seen or set()
if group in seen:
return []
seen.add(group)
out = list(ini_groups.get(group, []))
for child in children.get(group, []):
out += hosts_in(child, seen)
return out
def load_gv(group: str) -> dict:
d: dict = {}
for p in [GV / f"{group}.yml", GV / group / "vars.yml"]:
if p.is_file():
try:
d.update(yaml.safe_load(p.read_text()) or {})
except Exception:
pass
return d
# WHICH GROUP IS PROD? Derived, never named here: it is the group whose group_vars
# RETARGETS the k3s roles (k3s_primary_group / k3s_servers_group) away from the global
# dev default. That is exactly the seam docs/02 + infra/ansible/README describe for prod.
ALL_VARS = load_gv("all")
dev_primary_group = str(ALL_VARS.get("k3s_primary_group", ""))
candidates = []
for entry in sorted(GV.iterdir()) if GV.is_dir() else []:
g = entry.name[:-4] if entry.name.endswith(".yml") else entry.name
if g == "all":
continue
v = load_gv(g)
if v.get("k3s_primary_group") and str(v["k3s_primary_group"]) != dev_primary_group:
candidates.append((g, v))
check(len(candidates) == 1,
f"expected exactly ONE environment whose group_vars retargets the k3s roles (prod); found "
f"{[c[0] for c in candidates]} — R-ENV-03 describes a single prod estate reached through "
"the same roles as dev")
if not candidates:
print("CHECK FAIL (prod-baremetal-not-dev-surface): no prod group_vars found", file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
PROD_GROUP, PROD_VARS = candidates[0]
PROD_PRIMARY = str(PROD_VARS["k3s_primary_group"])
prod_hosts = hosts_in(PROD_GROUP)
prod_names = sorted(h["name"] for h in prod_hosts)
# ------------------------------------------------- (1) exactly three, directly reachable
check(len(prod_names) == 3,
f"prod group {PROD_GROUP!r} resolves to {len(prod_names)} host(s) {prod_names} — R-ENV-03 "
"says prod is a 3× mini-PC cluster")
dev_hosts = hosts_in(str(ALL_VARS.get("k3s_servers_group", "k3s_servers")))
for h in prod_hosts:
# Real machines on the LAN answer their own sshd. Dev's hosts are reached through a
# NAT port-forward (ansible_port 225x -> a VM's :22), which is precisely the
# indirection a hypervisor-hosted estate needs and bare metal does not.
check(str(h.get("ansible_port", "22")) == "22",
f"prod node {h['name']} is reached on port {h.get('ansible_port')} — a forwarded port "
"implies an indirection (NAT to a virtual estate), not a machine on the LAN")
check(h.get("k3s_node_ip") == h.get("ansible_host"),
f"prod node {h['name']} has ansible_host={h.get('ansible_host')} but "
f"k3s_node_ip={h.get('k3s_node_ip')} — a split address is the dev NAT pattern; a "
"bare-metal node's SSH address IS its cluster address")
# ------------------------------------------------------ (2) nothing creates prod as a VM
TF = "\n".join(p.read_text(errors="replace") for p in sorted(TOFU.glob("*.tf"))) \
if TOFU.is_dir() else ""
check(len(TF) > 500, "infra/tofu/*.tf is empty — cannot show that it does NOT create prod")
def tf_default(var: str):
m = re.search(r'variable\s+"' + re.escape(var) + r'"\s*\{(.*?)\n\}', TF, re.S)
if not m:
return None
d = re.search(r"default\s*=\s*(.+)", m.group(1))
return d.group(1).strip() if d else None
prefix = (tf_default("node_name_prefix") or "").strip('"')
try:
node_count = int((tf_default("node_count") or "0").strip())
except ValueError:
node_count = 0
iac_names = {f"{prefix}{i + 1}" for i in range(node_count)}
check(len(iac_names) > 0, "the IaC module generates no node names — cannot compare with prod")
check(iac_names.isdisjoint(set(prod_names)),
f"the IaC VM module would create {sorted(iac_names & set(prod_names))}, which are PROD nodes "
"— prod is bare metal that the operator images (docs/02 stage 1), never instantiated by the "
"hypervisor module")
for name in prod_names:
check(name not in TF,
f"prod node {name} is named inside infra/tofu — prod hardware must not be described by "
"the VM module at all")
# The prod host addresses must not appear as IaC node addresses either (a renamed VM is
# still a VM).
tf_ips = set(re.findall(r'"(\d+\.\d+\.\d+\.\d+)"', tf_default("node_ips") or ""))
prod_ips = {str(h.get("ansible_host")) for h in prod_hosts}
check(tf_ips.isdisjoint(prod_ips),
f"the IaC module addresses {sorted(tf_ips & prod_ips)}, which are prod node addresses")
# --------------------------------------------------------------- (3) node-local storage
# Every environment's value for a cluster-vars knob (base default + each overlay patch).
def env_values(key: str) -> dict[str, str]:
vals: dict[str, str] = {}
base = ROOT / "clusters" / "base" / "cluster-vars.yaml"
if base.is_file():
for d in yaml.safe_load_all(base.read_text()):
if isinstance(d, dict) and d.get("kind") == "ConfigMap" \
and (d.get("metadata") or {}).get("name") == "cluster-vars":
v = (d.get("data") or {}).get(key)
if v is not None:
vals["base"] = str(v)
for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
env = overlay.parent.name
if env == "base":
continue
try:
k = yaml.safe_load(overlay.read_text()) or {}
except Exception:
continue
for p in (k.get("patches") or []):
tgt = p.get("target") or {}
if tgt.get("kind") != "ConfigMap" or tgt.get("name") != "cluster-vars":
continue
try:
ops = yaml.safe_load(p.get("patch") or "") or []
except Exception:
continue
for op in ops if isinstance(ops, list) else []:
if isinstance(op, dict) and str(op.get("path", "")) == f"/data/{key}":
vals[env] = str(op.get("value"))
return vals
# The prod OVERLAY is the one whose CLUSTER_ENV is not the base default.
cluster_envs = env_values("CLUSTER_ENV")
prod_overlays = [e for e, v in cluster_envs.items()
if e != "base" and v != cluster_envs.get("base")]
check(len(prod_overlays) >= 1,
"no clusters/ overlay sets its own CLUSTER_ENV — prod has no overlay of its own")
prod_overlay = None
for e in prod_overlays:
if PROD_GROUP.replace("k3s_", "") in cluster_envs[e] or cluster_envs[e] in PROD_GROUP:
prod_overlay = e
check(prod_overlay is not None,
f"no overlay's CLUSTER_ENV corresponds to the prod inventory group {PROD_GROUP!r} — the "
"cluster config and the machine inventory disagree about which estate is prod")
data_paths = env_values("LONGHORN_DATA_PATH")
replicas = env_values("LONGHORN_DEFAULT_REPLICA_COUNT")
if prod_overlay:
p = data_paths.get(prod_overlay, data_paths.get("base", ""))
check(p.startswith("/"),
f"prod's LONGHORN_DATA_PATH is {p!r} — R-ENV-03's 'local storage' requires an absolute "
"node-local filesystem path")
check(not re.match(r"^/(net|nfs|mnt/nfs)|:", p),
f"prod's LONGHORN_DATA_PATH {p!r} looks like a network mount, not local SSD/NVMe")
r = replicas.get(prod_overlay, replicas.get("base", ""))
check(r.isdigit() and int(r) == len(prod_names),
f"prod's LONGHORN_DEFAULT_REPLICA_COUNT is {r!r} for {len(prod_names)} node(s) — "
"'local storage' on a 3-node estate means one replica per node's own disk")
# The DEFAULT storage class must be the local (Longhorn) driver. A separate bulk/media tier
# on network storage is explicitly a different requirement (R-STATE-04, docs/04) and is not
# counted here.
default_scs = []
for f in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z", "*.yaml", "*.yml"],
capture_output=True, text=True, check=True).stdout.split("\0"):
if not f:
continue
try:
for d in yaml.safe_load_all((ROOT / f).read_text(errors="replace")):
if isinstance(d, dict) and d.get("kind") == "StorageClass" and \
str(((d.get("metadata") or {}).get("annotations") or {})
.get("storageclass.kubernetes.io/is-default-class", "")) == "true":
default_scs.append((f, str(d.get("provisioner", ""))))
except Exception:
continue
check(len(default_scs) == 1,
f"expected exactly one default StorageClass, found {default_scs} — the tier prod's workloads "
"land on by default must be unambiguous")
for f, prov in default_scs:
check(prov == "driver.longhorn.io",
f"{f}: the default StorageClass provisioner is {prov!r}, not Longhorn — prod's default "
"tier would not be its own local disks")
# ------------------------------------- (4) prod runs the SAME bootstrap code, retargeted
PLAYBOOKS = ANS / "playbooks"
ROLES = ANS / "roles"
prod_token = PROD_GROUP.replace("k3s_", "")
for d, what in ((ROLES, "role"), (PLAYBOOKS, "playbook")):
for entry in sorted(d.iterdir()) if d.is_dir() else []:
check(prod_token not in entry.name.lower(),
f"infra/ansible/{d.name}/{entry.name} is a prod-specific {what} — docs/02 requires "
"the SAME Ansible bootstrap for prod, differing only in the machine-creation step")
# Every prod group_vars key must be a variable the shared code already knows: a role default,
# a global, or an ansible built-in. An invented key is a fork of the role by another name.
known: set[str] = set(ALL_VARS)
for rd in sorted(ROLES.iterdir()) if ROLES.is_dir() else []:
for p in (rd / "defaults").glob("*.yml") if (rd / "defaults").is_dir() else []:
try:
known |= set((yaml.safe_load(p.read_text()) or {}).keys())
except Exception:
pass
check(len(known) > 20, f"only {len(known)} shared variables found — (4) would prove nothing")
for k in sorted(PROD_VARS):
check(k.startswith("ansible_") or k in known,
f"prod group_vars sets {k!r}, which is not a role default or a global — prod-only "
"configuration that the dev path never exercises is developing against prod")
# ...and it must retarget, not re-implement: the group it names must exist in the inventory.
check(PROD_PRIMARY in ini_groups or PROD_PRIMARY in children,
f"prod group_vars names primary group {PROD_PRIMARY!r}, which the inventory does not define")
# ---------------------------------------- (5) prod is not a development surface (overlay)
if prod_overlay:
odir = ROOT / "clusters" / prod_overlay
files = sorted(p.name for p in odir.iterdir() if p.is_file())
check(files == ["kustomization.yaml", "sync.yaml"],
f"clusters/{prod_overlay}/ contains {files} — a prod overlay may hold only its "
"kustomization and its own root sync; anything else is configuration authored FOR PROD")
k = yaml.safe_load((odir / "kustomization.yaml").read_text()) or {}
check(sorted(str(r) for r in (k.get("resources") or [])) == ["../base", "sync.yaml"],
f"clusters/{prod_overlay} resources are {k.get('resources')} — prod must render the base "
"plus its own sync only (docs/02: 'apply the prod overlay (no base edits)')")
# Every patch must be an ENVIRONMENT-SCOPED override, never a change to what runs.
ALLOWED = {("ConfigMap", "cluster-vars"), ("GitRepository", None)}
for p in (k.get("patches") or []):
tgt = p.get("target") or {}
body = p.get("patch") or ""
kind = tgt.get("kind")
if kind is None: # inline patch: kind comes from the body
try:
doc = yaml.safe_load(body) or {}
except Exception:
doc = {}
kind = doc.get("kind")
name = (doc.get("metadata") or {}).get("name")
if kind == "Kustomization":
# Flux Kustomization patches are allowed ONLY to wire sops decryption —
# anything else (path, interval, images, values) changes what prod RUNS.
keys = set((doc.get("spec") or {}).keys())
check(keys <= {"decryption"},
f"clusters/{prod_overlay}: patch of Kustomization/{name} touches {sorted(keys)} "
"— only `decryption` (the per-cluster sops key) is an environment-scoped "
"override; changing anything else is developing against prod")
continue
check((kind, None) in ALLOWED or (kind, name) in ALLOWED,
f"clusters/{prod_overlay}: patch targets {kind}/{name}, which is not an "
"environment-scoped object — prod must not carry configuration of its own")
else:
check((kind, tgt.get("name")) in ALLOWED or (kind, None) in ALLOWED,
f"clusters/{prod_overlay}: patch targets {kind}/{tgt.get('name')}, which is not "
"an environment-scoped object — prod must not carry configuration of its own")
# Same Git ref as the other environment(s): a prod-only branch is a development line
# that only prod runs, which is exactly what R-ENV-03 forbids.
def git_ref(overlay: str):
kk = yaml.safe_load((ROOT / "clusters" / overlay / "kustomization.yaml").read_text()) or {}
for p in (kk.get("patches") or []):
body = p.get("patch") or ""
try:
doc = yaml.safe_load(body) or {}
except Exception:
continue
if doc.get("kind") == "GitRepository":
return (doc.get("spec") or {}).get("ref")
return None
others = [o.name for o in (ROOT / "clusters").iterdir()
if o.is_dir() and o.name not in ("base", prod_overlay)]
prod_ref = git_ref(prod_overlay)
check(prod_ref is not None,
f"clusters/{prod_overlay} does not pin a Git ref — prod would follow whatever the base "
"points at, with no record of what it runs")
for o in others:
check(git_ref(o) == prod_ref,
f"clusters/{o} tracks {git_ref(o)} but prod tracks {prod_ref} — a prod-only ref is a "
"development line only prod runs (R-ENV-03: prod is a target, not a sandbox)")
# No prod-only manifest anywhere in the rendered configuration.
tracked = [f for f in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
capture_output=True, text=True, check=True
).stdout.split("\0") if f]
RENDERED_DIRS = ("platform/", "platform-config", "apps/", "catalog/", "clusters/base/")
for f in tracked:
if f.startswith(RENDERED_DIRS) and re.search(rf"(^|[/_.-]){re.escape(prod_token)}([/_.-]|$)",
f, re.I):
check(False,
f"{f} is a prod-specific file inside the shared configuration — every environment "
"renders these paths, so prod-only content here is development against prod")
check(len(tracked) > 50, f"git ls-files returned only {len(tracked)} paths — not a real checkout")
# --------------------------- (6) prod work exists only with a declaration of readiness
# In-repo: the operator's recorded decision to open the prod phase.
log = ROOT / "DECISIONS-LOG.md"
check(log.is_file() and re.search(r"PHASE\s*7\b.*OPEN", log.read_text(errors="replace"), re.I),
"DECISIONS-LOG.md carries no record of the operator OPENING the prod phase, yet prod "
"inventory/overlay exist — R-ENV-03 allows prod build only once the roadmap declares it ready")
# In the spec (read-only, may not be mounted): the roadmap phase that opens prod.
spec = Path(os.environ.get("SPEC_REPO", ""))
phases = spec / "roadmap" / "phases.md"
if phases.is_file():
txt = phases.read_text(errors="replace")
check(re.search(r"##\s*Phase\s*7\b", txt) is not None and
re.search(r"prod", txt, re.I) is not None,
"the spec roadmap has no Phase 7 opening prod build — the declaration R-ENV-03 defers "
"to does not exist")
check(re.search(r"explicitly opened|declares prod hardware ready|operator declares", txt, re.I)
is not None,
"the spec roadmap's prod phase states no explicit entry declaration — the gate R-ENV-03 "
"names would be unenforceable")
else:
note(f"spec repo not mounted at {spec} — the roadmap half of (6) was NOT checked "
"(the in-repo declaration still was)")
# --------------------------- (4b) DRIVE the retarget offline: same chain, prod hosts
if shutil.which("ansible-playbook") is None:
note("ansible-playbook is not installed here (e.g. the in-cluster chainsaw image), so the "
"prod retarget was PARSED but not DRIVEN; the merge gate's container has it")
else:
sec_groups = [g for g in children.get(PROD_GROUP, []) if g != PROD_PRIMARY]
args = ["--list-hosts", "playbooks/site.yml",
"-e", f"k3s_hosts={PROD_GROUP}",
"-e", f"k3s_primary_hosts={PROD_PRIMARY}"]
if sec_groups:
args += ["-e", f"k3s_secondary_hosts={sec_groups[0]}"]
lh = subprocess.run(["ansible-playbook", *args], cwd=ANS, capture_output=True, text=True)
out = (lh.stderr or "") + (lh.stdout or "")
if lh.returncode != 0 and ENV_ONLY_FAILURE.search(out):
# See the same narrow, self-policing skip in dev-nodes-provisioned-as-code.
note("ansible-playbook lacks a LOCAL dependency kept outside the repo by design (the vault "
"passphrase and/or the gitignored Galaxy collections), so the prod retarget was not "
"driven; every parse assertion above still ran")
else:
check(lh.returncode == 0,
f"retargeting the SHARED chain at prod failed: {out.strip()[:400]} — docs/02 requires "
"the same Ansible bootstrap to run for prod")
seen: set[str] = set()
in_hosts = False
for line in lh.stdout.splitlines():
if re.match(r"\s*hosts \(\d+\):", line):
in_hosts = True
continue
if re.search(r"play #\d+", line):
in_hosts = False
continue
if in_hosts and line.strip():
seen.add(line.strip())
check(seen == set(prod_names),
f"the shared chain retargeted at prod resolves to {sorted(seen)}, not the three prod "
f"nodes {prod_names} — prod would be bootstrapped by something other than dev's path")
# --------------------------------------------------------------------------- verdict
check(CHECKS >= 25,
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
"would be meaningless")
if FAILURES:
print(f"CHECK FAIL (prod-baremetal-not-dev-surface): {len(FAILURES)} of {CHECKS} assertions failed",
file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
for n in NOTES:
print(f" note: {n}")
print(f"check OK: prod-baremetal-not-dev-surface — {CHECKS} assertions; prod group {PROD_GROUP!r} = "
f"{prod_names} (3 nodes, none created by IaC), overlay clusters/{prod_overlay} renders the base "
f"plus environment-scoped patches only, local Longhorn path "
f"{data_paths.get(prod_overlay, data_paths.get('base'))!r} with "
f"{replicas.get(prod_overlay, replicas.get('base'))} replica(s). R-ENV-03's AUTHORED clauses "
"only — the hardware facts (mini-PC, bare metal, local NVMe) are operator-attested and the "
"live cluster's shape is NOT asserted here (see the header's gap list).")
PY