jakman85andClaude Opus 5
377fa205ea
INCOMPLETE — preserved by the PM after the authoring agent hit the session
rate limit mid-task. Do not merge without finishing and gating it.
Closes the inverse of what this check already covered. It asserted that every
${...} in a substituted path is defined or escaped; it did NOT catch content
containing a literal doubled dollar that the author needs delivered verbatim
and that envsubst silently halves. That shipped a broken CronJob (fixed in
16b7fcd): Postgres dollar-quoting became a single dollar and every
Flux-reconciled run died with a syntax error, while passing review because
verification used kubectl apply, which runs no envsubst.
Introduces a dollar-collapse.allow allowlist so each legitimate use is a
reviewed decision rather than an accident.
WHAT REMAINS: the agent's own next step was proving it catches the CLASS and
not just this instance, via a scratch manifest under a substituted path. Its
scratch fixture (apps/wave-2/zz-scratch-class.yaml and the wave-2
kustomization edit) was deliberately temporary and has been reverted here, so
the fail-proof is NOT yet demonstrated. Also unanswered: brief point 3, on
whether the check can say anything about a manifest verified only through a
non-substituting path.
Co-Authored-By: Claude Opus 5 <[email protected]>
650 lines
37 KiB
Bash
Executable File
650 lines
37 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# =============================================================================
|
|
# L0 static check (offline, no cluster) — flux-postbuild-substitution
|
|
# Serves: R-ENV-02 ("Same GitOps config promotes dev/test -> prod (authored once)."
|
|
# — spec docs/02-environments-and-provisioning.md)
|
|
# R-GITOPS-01 ("All desired state in git, reconciled by Flux."
|
|
# — spec docs/05-gitops-and-delivery.md)
|
|
#
|
|
# WHY THIS EXISTS — Flux post-build substitution FAILS OPEN
|
|
# --------------------------------------------------------
|
|
# R-ENV-02 is implemented by exactly one mechanism: `clusters/base/cluster-vars.yaml`
|
|
# is injected into every `${TOKEN}` of the manifests each Flux Kustomization renders,
|
|
# via `postBuild.substituteFrom`. That mechanism is silently lossy by default: an
|
|
# UNDEFINED variable becomes the EMPTY STRING and the Kustomization still reports
|
|
# Ready. On 2026-09-01 that emptied `k8sServiceHost` in the Cilium HelmRelease and put
|
|
# cilium into Init:Error on 2 of 3 dev nodes (DECISIONS-LOG "substitution race that
|
|
# fails OPEN"). "Authored once, promoted" is therefore only true while every token a
|
|
# substituted path renders is either DEFINED or deliberately ESCAPED — and nothing in
|
|
# the repo asserted that. This check does.
|
|
#
|
|
# SECOND INCIDENT, 2026-09-13 — THE INVERSE HAZARD (commit 16b7fcd)
|
|
# ------------------------------------------------------------------
|
|
# The first incident was a variable the author wanted EXPANDED that rendered EMPTY. This one
|
|
# is its mirror: content the author wanted to reach the workload UNTOUCHED that substitution
|
|
# silently REWROTE. A CronJob reconciler in apps/wave-0/nextcloud-db.yaml asserted its own
|
|
# convergence in a dollar-quoted PL/pgSQL block —
|
|
#
|
|
# DO <dq> BEGIN ... END <dq>; where <dq> is two literal dollars: standard Postgres
|
|
#
|
|
# — inside a ConfigMap under apps-wave-0, which declares postBuild.substituteFrom. envsubst
|
|
# collapses a doubled dollar into ONE dollar, ALWAYS, so Postgres received a half-quoted
|
|
# `DO <one dollar>` and died with `syntax error at or near "$"`. Three scheduled runs left
|
|
# nine Error pods in nextcloud-db before anyone looked.
|
|
#
|
|
# It PASSED verification because it was hand-applied with `kubectl apply`, which runs NO
|
|
# envsubst. The defect existed ONLY on the Flux-rendered path. And the damage landed
|
|
# precisely on the self-check: the GRANT on line 1 still ran, so the privilege stayed
|
|
# correct and the only thing broken was the assertion meant to make a silent regression
|
|
# visible — the part whose whole job is to stop a blind spot being quiet.
|
|
#
|
|
# THE ASSERTION GAP THIS CLOSES: assertions (1)-(3) below ask whether every `${...}` is
|
|
# defined, escaped, or parseable. They said NOTHING about a doubled dollar that is not an
|
|
# escape, because the old classifier treated a `$` run of length >= 2 as "escaped, therefore
|
|
# fine" and stopped looking. Nothing failed the gate; the manifest rendered "successfully"
|
|
# into something broken. Assertion (7) is that hole, closed.
|
|
#
|
|
# ENVSUBST BEHAVIOUR ASSERTED HERE (empirically verified against the pinned toolchain,
|
|
# `flux envsubst`, flux 2.4.0 — this is measured behaviour, not assumption):
|
|
# ${DEFINED} -> substituted with the cluster-vars value.
|
|
# ${UNDEFINED} -> EMPTY STRING, silently. `--strict` / the cluster-wide
|
|
# StrictPostBuildSubstitutions feature gate turns this into a
|
|
# loud Kustomization failure ("variable not set (strict mode)").
|
|
# ${UNDEFINED:-d} -> the DEFAULT is applied AT RECONCILE TIME. Strict mode does
|
|
# NOT catch this (it exempts default-operator forms by design),
|
|
# so a shell guard written `[ -z "${X:-}" ]` inside a rendered
|
|
# manifest is pre-resolved to `[ -z "" ]` and its else-branch
|
|
# becomes unreachable in the container. Silent, and permanent.
|
|
# $${VAR}, $${V:-d}, $${ARR[0]} -> emitted as the LITERAL ${...} for a downstream
|
|
# shell/cloud-init consumer. Escaping is total: it neutralises
|
|
# even the indexed form.
|
|
# $VAR, $(cmd) -> left alone by envsubst; they need no escaping.
|
|
#
|
|
# DOLLAR-RUN ARITHMETIC — the exact rule, MEASURED, not inferred (see assertion (7))
|
|
# ---------------------------------------------------------------------------------
|
|
# envsubst here is `github.com/drone/envsubst`, vendored by fluxcd/pkg/kustomize and exposed
|
|
# by the `flux envsubst` CLI ("replicate the behavior of the Flux Kustomization post-build
|
|
# substitutions" — its own --help). Its scanner walks the input left to right; on a `$` it
|
|
# looks at the NEXT byte:
|
|
# * another `$` -> BOTH are consumed and ONE `$` is emitted. This is UNCONDITIONAL. What
|
|
# follows the pair is irrelevant — `{`, a letter, a digit, a semicolon,
|
|
# a newline, end of input. `$$` is not "the escape for `${`"; it is the
|
|
# escape for `$` itself, everywhere.
|
|
# * `{` -> the braced expression is parsed and EXPANDED.
|
|
# * anything else -> the `$` is emitted literally (hence `$VAR` and `$(cmd)` survive).
|
|
# So a run of N consecutive dollars emits floor(N/2) literal dollars, and when N is ODD the
|
|
# final dollar is still LIVE. Measured with flux 2.4.0 `flux envsubst`, DEFINED=VALUE — the
|
|
# left column is authored bytes, the right is what reaches the cluster:
|
|
#
|
|
# $ -> $ ${DEFINED} -> VALUE
|
|
# $$ -> $ $${DEFINED} -> ${DEFINED} <- the escape
|
|
# $$$ -> $$ $$${DEFINED} -> $VALUE <- ODD: brace LIVE
|
|
# $$$$ -> $$ $$$${DEFINED} -> $${DEFINED}
|
|
# $$$$$ -> $$$ $$$$${DEFINED} -> $$VALUE
|
|
# DO $$ .. $$; -> DO $ .. $; $$${NOPE:-d} -> $d
|
|
# awk '{print $$2}' -> awk '{print $2}' $${ARR[0]} -> ${ARR[0]} (safe)
|
|
# $$@ $$< $$^ -> $@ $< $^ $$${ARR[0]} -> HARD PARSE ERROR, whole
|
|
# Kustomization dies
|
|
#
|
|
# TWO CONSEQUENCES THIS CHECK IS BUILT ON:
|
|
# (a) THERE IS NO WAY TO SHIP A LITERAL `$$` THROUGH A SUBSTITUTED PATH. To make two
|
|
# dollars arrive you must author FOUR. So any authored `$$` that is meant to reach the
|
|
# workload verbatim is a BUG, full stop — that is the 2026-09-13 incident, and Postgres
|
|
# dollar-quoting is its commonest shape (Makefile `$$@`, awk `$$2` and docker-compose
|
|
# `$${}` are the same hazard arriving from the other direction: there the collapse is
|
|
# what the author wanted).
|
|
# (b) `$$$` + brace is NOT "more escaped" than `$$` + brace — it is NOT escaped at all. An
|
|
# odd run leaves one live dollar, so the brace expands, or, for an inner envsubst
|
|
# cannot parse, fails the ENTIRE Kustomization (proven above with $$${ARR[0]}, which
|
|
# aborted the whole stream while $${ARR[0]} on the line before it was fine).
|
|
# `--strict` reports NONE of this: every row of that table except the parse error was
|
|
# accepted SILENTLY by `flux envsubst --strict`, exit 0. Strict mode cannot see this class,
|
|
# so a source-level assertion is the only available guard.
|
|
# ${...}, ${a b}, ${ARR[0]} (UNESCAPED) -> HARD PARSE ERROR ("unable to parse
|
|
# variable name" / "missing closing brace") which fails the
|
|
# WHOLE Kustomization — every resource in the path, not just
|
|
# the offending one.
|
|
# The escape contract this enforces is the one already documented in
|
|
# `clusters/base/apps-vms.yaml`; this check is that comment made executable.
|
|
#
|
|
# GIVEN the repository as authored,
|
|
# WHEN every Flux Kustomization under clusters/ that declares postBuild.substituteFrom
|
|
# is resolved to the exact file set its `spec.path` renders (by following
|
|
# kustomization.yaml `resources:` transitively — a path, not a glob),
|
|
# THEN (1) every `${...}` in those files is DEFINED in cluster-vars, or ESCAPED `$${...}`;
|
|
# (2) no UNESCAPED expansion carries a default/length/substitution operator on an
|
|
# undefined name (the silent-empty class strict mode does not catch);
|
|
# (3) no UNESCAPED expansion is unparseable by envsubst (`${...}`, `${ARR[0]}`) —
|
|
# that fails the entire Kustomization, so it is fatal wherever it appears;
|
|
# (4) every Kustomization that renders a cluster-vars token actually declares
|
|
# `substituteFrom` a ConfigMap named cluster-vars (else the literal token
|
|
# ships into the cluster);
|
|
# (5) every knob an overlay patches (`/data/KEY`) exists in the BASE ConfigMap —
|
|
# the "add the knob with a base default BEFORE referencing it" rule that the
|
|
# incident's interim discipline depends on;
|
|
# (7) NO DOLLAR RUN IS SILENTLY REWRITTEN: the only run of length >= 2 that passes
|
|
# unremarked is exactly `$$` immediately followed by `{` — the documented, total,
|
|
# already-inventoried escape. EVERY other run of >= 2 dollars (a bare `$$`, an odd
|
|
# `$$$`, an over-counted `$$$$`) is rewritten by envsubst between git and the
|
|
# container, so it must be named and justified in `dollar-collapse.allow`, a file
|
|
# this check also holds to being exact and non-stale;
|
|
# AND (6) this check is LOAD-BEARING: it DECLARES `conformance.home.k3s/scope: whole-repo`,
|
|
# so ci/l0.sh's `--include-scope whole-repo` runs it on every gate run, and
|
|
# ci/gate.sh sets the merge block when the l0 stage fails.
|
|
# Assertion (1)-(3) are applied to COMMENTS as well as content: `kustomize build`
|
|
# preserves body comments into the stream envsubst reads (verified: a `$`-carrying
|
|
# comment survives into the built apps/wave-1 output), so a braced expression in a
|
|
# comment is a live hazard, not prose.
|
|
#
|
|
# WHAT THIS DOES *NOT* PROVE (honest gaps — this is a STATIC check):
|
|
# - It does NOT render the manifests. It scans the SOURCE files a path includes,
|
|
# which is a deliberate SUPERSET of the post-`kustomize build` stream (kustomize
|
|
# drops some comments). A finding in a comment may not reach envsubst TODAY; it is
|
|
# still reported, because whether a given comment survives the build is a kustomize
|
|
# implementation detail that changes with the pinned version, and the failure it
|
|
# causes is total (assertion 3). The end-to-end proof — `kustomize build <path> |
|
|
# flux envsubst --strict` for every substituted path — needs kustomize + flux,
|
|
# which the L0 CI toolchain does not install (.gitea/workflows/scripts/install-tools.sh
|
|
# l0), so it cannot run here without permanently SKIPPING. It was run by hand and
|
|
# recorded in DECISIONS-LOG; this check is the reproducible static half.
|
|
# - It does NOT prove the cluster's live state matches git, nor that the
|
|
# StrictPostBuildSubstitutions feature gate is actually enabled on
|
|
# kustomize-controller. Both are cluster facts (L2), untested here. Tagging
|
|
# R-GITOPS-01 claims ONLY the static slice: what git says a Kustomization renders
|
|
# is fully determined, with no variable silently emptied on the way in.
|
|
# - It CANNOT prove the thing that actually let 16b7fcd's bug reach a cluster: that a
|
|
# change was VERIFIED through a substituting path. "It was hand-applied with kubectl" is
|
|
# a fact about a shell session, not about any tracked file — git records the manifest,
|
|
# never the command someone ran to try it, so the property is not statically decidable
|
|
# here and no honest formulation of it exists. Nor is a proxy worth having: grepping
|
|
# commit prose or runbooks for `kubectl apply` would be trivially satisfiable by
|
|
# rewording, and would fire on the many legitimate break-glass runbooks whose whole
|
|
# content is that command. So this check does not attempt it, and does not pretend to.
|
|
# Assertion (7) answers that incident a different way: it moves the proof off the
|
|
# verification PATH, which is unknowable, and onto the SOURCE, which is not — the
|
|
# mangling is now caught by reading the manifest, whichever way a human chose to try it.
|
|
# (The one decidable half-measure, `kustomize build <path> | flux envsubst --strict` per
|
|
# substituted path in CI, is already ruled out above: the L0 toolchain installs neither
|
|
# tool, so the stage would SKIP permanently. The dev image has both, and it is how the
|
|
# arithmetic table above was measured.)
|
|
# - It does NOT validate cluster-vars VALUES (that GATEWAY_VIP is inside the LB pool,
|
|
# that an overlay's LAN block is self-consistent). Presence, not correctness.
|
|
# - A cluster-vars key nothing references is reported as an ADVISORY, never a failure:
|
|
# several are deliberately reference-only and say so in cluster-vars.yaml
|
|
# (PIA_REGION "retained, not substituted"; CLUSTER_POD_CIDR "Documents ..."), so
|
|
# failing on them would be a false block on documented intent.
|
|
# =============================================================================
|
|
set -euo pipefail
|
|
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
|
PYTHON="${PYTHON:-python3}"
|
|
|
|
# Honest SKIP (never green) if the thing under test is not present at all.
|
|
if [ ! -d "${ROOT}/clusters" ] || [ ! -f "${ROOT}/clusters/base/cluster-vars.yaml" ]; then
|
|
echo "check SKIPPED: clusters/ or clusters/base/cluster-vars.yaml absent — nothing to audit" >&2
|
|
exit 3
|
|
fi
|
|
|
|
cd "${ROOT}"
|
|
"${PYTHON}" - "${ROOT}" <<'PY'
|
|
"""Static audit of Flux postBuild substitution safety. Pure standard library.
|
|
|
|
No PyYAML: the file shapes parsed here are line-constrained and the parse contracts are
|
|
documented at each parser, matching lib/coverage.py + lib/status.py. The kustomization
|
|
`resources:` parser is IMPORTED from lib/status.py rather than forked, so there is one
|
|
definition of "what a Flux path actually renders" in the harness.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import bisect
|
|
import os
|
|
import re
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
ROOT = Path(sys.argv[1])
|
|
sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree
|
|
sys.path.insert(0, str(ROOT / "conformance" / "lib"))
|
|
import status as st # noqa: E402 — reuse the walk, don't fork it
|
|
|
|
FAILURES: list[str] = []
|
|
ADVISORIES: list[str] = []
|
|
CHECKS = 0
|
|
|
|
|
|
def check(cond, msg):
|
|
global CHECKS
|
|
CHECKS += 1
|
|
if not cond:
|
|
FAILURES.append(msg)
|
|
|
|
|
|
# ------------------------------------------------------------------ cluster-vars
|
|
# Parse contract: `data:` block of a single ConfigMap, one ` KEY: "value"` per line
|
|
# (every entry in clusters/base/cluster-vars.yaml is a quoted scalar; comments and
|
|
# continuation-comment lines are indented further and never match).
|
|
_DATA_KEY_RE = re.compile(r'^ ([A-Za-z_][A-Za-z0-9_]*):\s*(?:"|\')')
|
|
|
|
|
|
def cluster_vars_keys(path: Path) -> set[str]:
|
|
keys: set[str] = set()
|
|
in_data = False
|
|
for raw in path.read_text().splitlines():
|
|
if re.match(r"^data:\s*$", raw):
|
|
in_data = True
|
|
continue
|
|
if not in_data:
|
|
continue
|
|
if st._TOP_KEY_RE.match(raw): # a new top-level key ends the data block
|
|
break
|
|
m = _DATA_KEY_RE.match(raw)
|
|
if m:
|
|
keys.add(m.group(1))
|
|
return keys
|
|
|
|
|
|
VARS_FILE = ROOT / "clusters" / "base" / "cluster-vars.yaml"
|
|
KEYS = cluster_vars_keys(VARS_FILE)
|
|
check(len(KEYS) > 0, f"no data keys parsed from {VARS_FILE} — the parser or the file shape changed")
|
|
|
|
# ------------------------------------------- Flux Kustomizations with substituteFrom
|
|
# Parse contract mirrors lib/status.py's flux_paths(): docs split on a lone `---`, a doc
|
|
# is a Flux entrypoint only if it carries the kustomize.toolkit apiVersion AND
|
|
# `kind: Kustomization` AND a spec-level `path:`. Inline `patch: |-` bodies in the
|
|
# overlays carry Flux Kustomizations WITHOUT a path and are correctly skipped.
|
|
_SUBFROM_RE = re.compile(r"^(\s*)substituteFrom:\s*$", re.M)
|
|
_SUBITEM_KIND_RE = re.compile(r"^\s*-\s*kind:\s*(\S+)")
|
|
_SUBITEM_NAME_RE = re.compile(r"^\s*(?:-\s*)?name:\s*([^\s#]+)")
|
|
|
|
|
|
def substitute_sources(doc: str) -> list[tuple[str, str]]:
|
|
"""[(kind, name)] declared under postBuild.substituteFrom in this doc."""
|
|
m = _SUBFROM_RE.search(doc)
|
|
if not m:
|
|
return []
|
|
indent = len(m.group(1))
|
|
out: list[tuple[str, str]] = []
|
|
kind = ""
|
|
for raw in doc[m.end():].splitlines():
|
|
if raw.strip() == "" or raw.lstrip().startswith("#"):
|
|
continue
|
|
if len(raw) - len(raw.lstrip()) <= indent and not raw.lstrip().startswith("-"):
|
|
break # dedented out of the list
|
|
km = _SUBITEM_KIND_RE.match(raw)
|
|
if km:
|
|
kind = km.group(1)
|
|
continue
|
|
nm = _SUBITEM_NAME_RE.match(raw)
|
|
if nm and kind:
|
|
out.append((kind, nm.group(1)))
|
|
kind = ""
|
|
return out
|
|
|
|
|
|
kustomizations: list[dict] = []
|
|
for yfile in sorted((ROOT / "clusters").rglob("*.yaml")):
|
|
text = yfile.read_text()
|
|
if st.FLUX_API not in text:
|
|
continue
|
|
for doc in st._DOC_SPLIT_RE.split(text):
|
|
if st.FLUX_API not in doc or not re.search(r"^kind:\s*Kustomization\s*$", doc, re.M):
|
|
continue
|
|
pm = st._PATH_RE.search(doc)
|
|
if not pm:
|
|
continue
|
|
nm = st._NAME_RE.search(doc)
|
|
srcs = substitute_sources(doc)
|
|
if not srcs:
|
|
continue # no substitution => nothing to audit
|
|
kustomizations.append({
|
|
"file": str(yfile.relative_to(ROOT)),
|
|
"name": nm.group(1) if nm else "?",
|
|
"path": pm.group(1),
|
|
"sources": srcs,
|
|
})
|
|
|
|
check(len(kustomizations) > 0,
|
|
"no Flux Kustomization with postBuild.substituteFrom found under clusters/ — "
|
|
"the parser or the repo layout changed; this check would silently audit nothing")
|
|
|
|
# ------------------------------------------------------------------ token scanning
|
|
# One `$`-run followed by a braced expression. `[^{}]*` is exhaustive for this tree:
|
|
# a repo-wide sweep found no nested/unclosed brace expression in any manifest (only
|
|
# Gitea Actions `${{ ... }}` in .gitea/, which no Flux Kustomization renders).
|
|
TOKEN_RE = re.compile(r"(\$+)\{([^{}]*)\}")
|
|
NAME_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
|
|
# An expansion whose name is followed by a POSIX parameter operator. envsubst EVALUATES
|
|
# these, so an undefined name yields the default/empty AT RECONCILE TIME.
|
|
OP_RE = re.compile(r"^([A-Za-z_][A-Za-z0-9_]*)(:[-=?+].*|:[0-9].*|/.*|#.*|%.*|\^.*|,.*)$")
|
|
LEN_RE = re.compile(r"^#([A-Za-z_][A-Za-z0-9_]*)$")
|
|
|
|
|
|
def classify(dollars: int, inner: str) -> tuple[str, str] | None:
|
|
"""Return (severity-tag, remediation) for a hazardous expansion, else None.
|
|
|
|
PARITY, not ">= 2" (the 2026-09-13 correction). envsubst consumes dollars in PAIRS and
|
|
emits one per pair, so a run of N dollars in front of a brace leaves the brace ESCAPED
|
|
only when N is EVEN. An ODD run emits floor(N/2) literal dollars and its last dollar is
|
|
still LIVE, so `$$${X}` expands exactly as `${X}` does — and `$$${ARR[0]}` fails the
|
|
whole Kustomization where `$${ARR[0]}` is harmless. The old `dollars >= 2 -> return None`
|
|
called every such run "escaped" and stopped looking. Assertion (7) flags the run itself;
|
|
this function additionally classifies the still-live expansion an odd run carries.
|
|
"""
|
|
if dollars % 2 == 0:
|
|
return None # fully escaped: emitted as a literal
|
|
if NAME_RE.match(inner):
|
|
if inner in KEYS:
|
|
return None # defined knob — the happy path
|
|
return ("UNDEFINED",
|
|
f"strict mode fails the Kustomization; today it renders EMPTY. Either add "
|
|
f"'{inner}' to clusters/base/cluster-vars.yaml with a base default, or "
|
|
f"escape it as $${{{inner}}} if it is meant for a shell/cloud-init consumer")
|
|
m = OP_RE.match(inner) or LEN_RE.match(inner)
|
|
if m:
|
|
base = m.group(1)
|
|
if base in KEYS:
|
|
return None # defined knob with an operator
|
|
return ("SILENT-EMPTY",
|
|
f"envsubst resolves this AT RECONCILE TIME because '{base}' is not a "
|
|
f"cluster-var, so the rendered manifest carries the default, not the "
|
|
f"expression. StrictPostBuildSubstitutions does NOT catch this form. "
|
|
f"Escape it as $${{{inner}}} if the CONTAINER's shell is meant to expand it")
|
|
return ("FATAL-PARSE",
|
|
"envsubst cannot parse this and fails the ENTIRE Kustomization (every resource "
|
|
"in the path). Escape it as $${...} or rewrite the prose without braces")
|
|
|
|
|
|
def rendered_files(path_spec: str) -> tuple[list[str], bool]:
|
|
"""Repo-relative files the Flux `spec.path` renders, resolving `resources:` transitively."""
|
|
rel = os.path.normpath(path_spec.lstrip("./")) if path_spec not in (".", "./") else "."
|
|
if not (ROOT / rel).is_dir():
|
|
return [], False
|
|
applied, _ = st.reconciled_files(ROOT, rel)
|
|
return sorted(applied), True
|
|
|
|
|
|
findings: list[str] = []
|
|
referenced: dict[str, set[str]] = {}
|
|
inventory: list[str] = []
|
|
# rel path -> the Kustomization name(s) that render it, for assertion (7)'s second pass.
|
|
# A file can be rendered by more than one Kustomization; all of them substitute.
|
|
subst_files: dict[str, set[str]] = {}
|
|
|
|
for k in kustomizations:
|
|
files, exists = rendered_files(k["path"])
|
|
check(exists, f"{k['file']}: Kustomization '{k['name']}' renders path '{k['path']}' "
|
|
f"which is not a directory in this repo")
|
|
has_cluster_vars = ("ConfigMap", "cluster-vars") in k["sources"]
|
|
n_def = n_esc = n_bad = 0
|
|
for rel in files:
|
|
f = ROOT / rel
|
|
try:
|
|
text = f.read_text()
|
|
except (OSError, UnicodeDecodeError):
|
|
continue
|
|
subst_files.setdefault(rel, set()).add(k["name"])
|
|
for lineno, line in enumerate(text.splitlines(), 1):
|
|
for m in TOKEN_RE.finditer(line):
|
|
dollars, inner = len(m.group(1)), m.group(2)
|
|
verdict = classify(dollars, inner)
|
|
if verdict is None:
|
|
if dollars % 2 == 0: # PARITY, see classify(): an odd run is
|
|
n_esc += 1 # live, not escaped
|
|
else:
|
|
n_def += 1
|
|
referenced.setdefault(re.split(r"[:/#%^,]", inner)[0] or inner,
|
|
set()).add(k["name"])
|
|
# (4) a DEFINED knob is only substituted if this Kustomization
|
|
# actually pulls cluster-vars in; otherwise the literal ships.
|
|
check(has_cluster_vars,
|
|
f"{rel}:{lineno}: renders cluster-var ${{{inner}}} but "
|
|
f"Kustomization '{k['name']}' ({k['file']}) does not "
|
|
f"substituteFrom ConfigMap/cluster-vars — the literal token "
|
|
f"would ship into the cluster")
|
|
continue
|
|
n_bad += 1
|
|
tag, fix = verdict
|
|
findings.append(f" [{tag}] {rel}:{lineno} {m.group(0)}\n"
|
|
f" rendered by Flux Kustomization '{k['name']}' "
|
|
f"({k['file']}, path {k['path']})\n"
|
|
f" {fix}\n"
|
|
f" | {line.strip()[:120]}")
|
|
inventory.append(f" {k['name']:<16} path={k['path']:<22} files={len(files):<3} "
|
|
f"defined={n_def:<3} escaped={n_esc:<3} hazards={n_bad}")
|
|
|
|
check(not findings,
|
|
f"{len(findings)} unescaped, undefined or unparseable expansion(s) in substituted paths")
|
|
|
|
# ----------------------------------- (7) no dollar run is silently rewritten on the way out
|
|
# THE 2026-09-13 CLASS. envsubst consumes dollars in PAIRS and emits one per pair,
|
|
# unconditionally and everywhere (header: DOLLAR-RUN ARITHMETIC, measured against flux 2.4.0).
|
|
# So every run of >= 2 dollars in a substituted manifest is REWRITTEN between git and the
|
|
# container, and from the text alone the two intents are indistinguishable:
|
|
#
|
|
# (a) a deliberate escape the author WANTS collapsed — `$${VAR}` -> literal `${VAR}`;
|
|
# (b) content that must arrive VERBATIM — `DO $$ ... $$;` -> `DO $ ... $;`
|
|
# i.e. the incident. A BUG.
|
|
#
|
|
# THE RULE CHOSEN, AND WHY IT IS NEITHER NOISY NOR VACUOUS.
|
|
# Exactly ONE shape passes unremarked: a run of EXACTLY TWO dollars immediately followed by
|
|
# `{`. That is not a guess about intent, it is the escape contract this repo already
|
|
# documents in clusters/base/apps-vms.yaml, that assertions (1)-(3) already validate the
|
|
# inside of, and that 18 reviewed occurrences already rely on (unifi cloud-init, the gitea
|
|
# and nextcloud OIDC bootstrap scripts, two longhorn comments). Failing those would be a
|
|
# permanently-firing alert on documented, correct intent, and this repo has already been
|
|
# bitten by one of those.
|
|
#
|
|
# EVERY OTHER run of >= 2 dollars FAILS unless it is named in `dollar-collapse.allow`:
|
|
# * DOLLAR-COLLAPSE — a run not followed by `{`. Cannot survive verbatim: `$$` -> `$`.
|
|
# This is the incident's exact shape and a blanket bug by default.
|
|
# * DOLLAR-HALF-ESCAPED — an ODD run >= 3 before `{`. Looks escaped, is not: the last
|
|
# dollar is live. Occasionally legitimate (`$$${PRICE}` to render a
|
|
# literal `$` before a value), so it is allowlistable, not banned.
|
|
# * DOLLAR-OVER-ESCAPED — an EVEN run >= 4 before `{`. Emits N/2 dollars then a literal
|
|
# brace; nearly always a miscount, sometimes deliberate.
|
|
#
|
|
# WHY AN ALLOWLIST AND NOT A BLANKET BAN: a blanket ban was the first choice and was
|
|
# rejected on evidence, not taste — main carries two real non-braced `$$` occurrences today
|
|
# (prose inside embedded shell comments), so a blanket ban would have shipped RED. Nor is
|
|
# the allowlist a hole: an entry is a hand-written edit to a file inside the conformance
|
|
# test, which is loud in review; it must quote a >= 24-character VERBATIM context, so it can
|
|
# only ever cover the occurrence it was written for and never a future one that drifts into
|
|
# the same file; it must carry a >= 40-character reason; and every entry must still MATCH
|
|
# something, so the allowlist cannot rot into a blanket exemption after the code it excused
|
|
# is deleted. A wildcard, a bare `$$` context and a stale entry each FAIL.
|
|
ALLOW_FILE = (ROOT / "conformance" / "tests" / "l0-static"
|
|
/ "flux-postbuild-substitution" / "dollar-collapse.allow")
|
|
MIN_CONTEXT = 24
|
|
MIN_REASON = 40
|
|
DOLLAR_RUN_RE = re.compile(r"\$+")
|
|
|
|
allow_entries: list[dict] = []
|
|
check(ALLOW_FILE.is_file(),
|
|
f"{ALLOW_FILE.relative_to(ROOT)} is missing — assertion (7) needs its allowlist to "
|
|
f"exist even when empty, because an absent file and an empty file are the same green "
|
|
f"and the next author would not know the mechanism is there")
|
|
if ALLOW_FILE.is_file():
|
|
for lineno, raw in enumerate(ALLOW_FILE.read_text().splitlines(), 1):
|
|
if not raw.strip() or raw.lstrip().startswith("#"):
|
|
continue
|
|
parts = [x.strip() for x in raw.split("|")]
|
|
if len(parts) != 3:
|
|
check(False, f"dollar-collapse.allow:{lineno}: expected exactly 3 '|'-separated "
|
|
f"fields (path | verbatim-context | reason), got {len(parts)}")
|
|
continue
|
|
allow_entries.append({"line": lineno, "path": parts[0], "ctx": parts[1],
|
|
"reason": parts[2], "used": 0})
|
|
|
|
for e in allow_entries:
|
|
tag = f"dollar-collapse.allow:{e['line']}"
|
|
check((ROOT / e["path"]).is_file(),
|
|
f"{tag}: names '{e['path']}', which is not a file in this repo — a stale entry is "
|
|
f"a standing exemption for a path someone could recreate")
|
|
check(len(e["ctx"]) >= MIN_CONTEXT,
|
|
f"{tag}: context {e['ctx']!r} is {len(e['ctx'])} chars; at least {MIN_CONTEXT} are "
|
|
f"required so an entry cannot be a near-wildcard that swallows future occurrences")
|
|
check(DOLLAR_RUN_RE.search(e["ctx"]) is not None
|
|
and max((len(m.group(0)) for m in DOLLAR_RUN_RE.finditer(e["ctx"])), default=0) >= 2,
|
|
f"{tag}: context {e['ctx']!r} contains no run of two or more dollars, so it excuses "
|
|
f"nothing and is either a typo or a decoy")
|
|
check(len(e["reason"]) >= MIN_REASON,
|
|
f"{tag}: reason is {len(e['reason'])} chars; at least {MIN_REASON} are required. "
|
|
f"State what envsubst will actually emit and why that is acceptable — an entry "
|
|
f"nobody had to explain is an accident wearing an approval")
|
|
|
|
RUN_TAGS = {
|
|
"collapse": ("DOLLAR-COLLAPSE",
|
|
"envsubst collapses each PAIR of dollars into one, everywhere and "
|
|
"unconditionally, so this CANNOT reach the workload verbatim (a literal $$ "
|
|
"needs $$$$ in source). This is the 2026-09-13 nextcloud-db shape: a "
|
|
"dollar-quoted PL/pgSQL DO block arrived half-quoted and every Flux-"
|
|
"reconciled run died. Rewrite without dollars, double them again if the "
|
|
"collapse is what you want, or justify it in dollar-collapse.allow"),
|
|
"half": ("DOLLAR-HALF-ESCAPED",
|
|
"an ODD run of dollars before a brace is NOT escaped: floor(N/2) literal "
|
|
"dollars are emitted and the LAST dollar is still live, so this expansion "
|
|
"EVALUATES at reconcile time (and if its inner is unparseable it fails the "
|
|
"entire Kustomization). Use an EVEN run to escape, or justify it in "
|
|
"dollar-collapse.allow if a literal dollar before a substituted value is "
|
|
"really what you meant"),
|
|
"over": ("DOLLAR-OVER-ESCAPED",
|
|
"an EVEN run of four or more dollars emits N/2 literal dollars followed by a "
|
|
"LITERAL brace — the value is never substituted and extra dollars appear in "
|
|
"the rendered manifest. Almost always a miscount of the escape; justify it in "
|
|
"dollar-collapse.allow if it is not"),
|
|
}
|
|
|
|
run_findings: list[str] = []
|
|
n_runs_ok = 0
|
|
for rel in sorted(subst_files):
|
|
text = (ROOT / rel).read_text(errors="replace")
|
|
# Absolute offsets, not per-line: the gitea/nextcloud bootstrap scripts are single
|
|
# 4000-character folded scalars, so "the line" is not a reviewable unit of context here.
|
|
nl = [i for i, ch in enumerate(text) if ch == "\n"]
|
|
spans = [] # allowlisted [start, end) regions
|
|
for e in allow_entries:
|
|
if e["path"] != rel:
|
|
continue
|
|
start = text.find(e["ctx"])
|
|
while start >= 0:
|
|
spans.append((start, start + len(e["ctx"]), e))
|
|
start = text.find(e["ctx"], start + 1)
|
|
for m in DOLLAR_RUN_RE.finditer(text):
|
|
n = len(m.group(0))
|
|
if n < 2:
|
|
continue
|
|
braced = text[m.end():m.end() + 1] == "{"
|
|
if n == 2 and braced:
|
|
n_runs_ok += 1 # the documented escape; (1)-(3) own it
|
|
continue
|
|
covering = [e for (a, b, e) in spans if a <= m.start() and m.end() <= b]
|
|
if covering:
|
|
for e in covering:
|
|
e["used"] += 1
|
|
continue
|
|
kind = "collapse" if not braced else ("half" if n % 2 else "over")
|
|
tag, fix = RUN_TAGS[kind]
|
|
lineno = bisect.bisect_right(nl, m.start()) + 1
|
|
ctx = text[max(0, m.start() - 60):m.end() + 60].replace("\n", "\\n")
|
|
run_findings.append(
|
|
f" [{tag}] {rel}:{lineno} a run of {n} dollar(s)"
|
|
f"{' before a brace' if braced else ''}\n"
|
|
f" rendered by Flux Kustomization(s) {', '.join(sorted(subst_files[rel]))}"
|
|
f" — postBuild substitution REWRITES it\n"
|
|
f" {fix}\n"
|
|
f" | ...{ctx}...")
|
|
|
|
check(not run_findings,
|
|
f"{len(run_findings)} dollar run(s) in substituted paths that envsubst rewrites and "
|
|
f"nothing justifies (the 2026-09-13 nextcloud-db class)")
|
|
|
|
for e in allow_entries:
|
|
check(e["used"] > 0,
|
|
f"dollar-collapse.allow:{e['line']}: STALE — its context is not present in "
|
|
f"{e['path']} any more (or no longer contains a rewritten dollar run), so the entry "
|
|
f"now excuses nothing. Delete it. A stale allowlist is how an opt-in decays into a "
|
|
f"blanket exemption for whatever lands in that file next")
|
|
|
|
# --------------------------------------------- (5) overlay knobs exist in the base
|
|
# Parse contract: overlays patch cluster-vars with JSON6902 ops (`- op: replace` /
|
|
# `path: /data/KEY`) inside an inline `patch: |-` block. The repo rule (cluster-vars.yaml
|
|
# header) is that a knob is ADDED TO THE BASE with a default BEFORE any overlay or
|
|
# manifest references it — that ordering is what closes the define-then-use race.
|
|
_OP_PATH_RE = re.compile(r"^\s*(?:-\s*)?path:\s*/data/([A-Za-z_][A-Za-z0-9_]*)\s*$", re.M)
|
|
for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
|
|
for key in set(_OP_PATH_RE.findall(overlay.read_text())):
|
|
check(key in KEYS,
|
|
f"{overlay.relative_to(ROOT)}: patches /data/{key} but '{key}' has no BASE "
|
|
f"default in clusters/base/cluster-vars.yaml — a second overlay would render "
|
|
f"${{{key}}} empty (R-ENV-02: author once, supply per env)")
|
|
|
|
# ------------------------------------------------------- (6) the check is load-bearing
|
|
L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
|
|
GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
|
|
# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism.
|
|
# It used to assert `"flux-postbuild-substitution/check.sh" in L0`: ci/l0.sh carried a hand-written
|
|
# "UNCONDITIONAL" stage per whole-repo check, hardcoding this path. Those stages are gone
|
|
# (they did not scale, and the next whole-repo invariant only got one if someone REMEMBERED
|
|
# to write it). The same property is now asserted two ways: this test DECLARES
|
|
# `conformance.home.k3s/scope: "whole-repo"`, and ci/l0.sh unions every declaring test into
|
|
# the run via `--include-scope`. Neither half alone is enough, so both are checked here; the
|
|
# mechanism ITSELF (the union really selects, and a scope-declared SKIP exits 3 rather than
|
|
# passing) is proven end-to-end and falsifiably by
|
|
# conformance/tests/l0-static/whole-repo-invariants-wired.
|
|
SELF_YAML = ROOT / "conformance" / "tests" / "l0-static" / "flux-postbuild-substitution" / "chainsaw-test.yaml"
|
|
SELF = SELF_YAML.read_text() if SELF_YAML.is_file() else ""
|
|
check(re.search(r'^\s*conformance\.home\.k3s/scope:\s*["\']?whole-repo', SELF, re.M) is not None,
|
|
"flux-postbuild-substitution/chainsaw-test.yaml does not declare "
|
|
'conformance.home.k3s/scope: "whole-repo", so the merge gate selects this check only '
|
|
"when a change happens to touch one of the requirements it is labelled with. "
|
|
"The path map keys only clusters/* to R-ENV-02, so an edit to a SUBSTITUTED manifest under apps/ or platform/ attributes nothing and this check would be filtered out of exactly the change that empties a variable (the 2026-09-01 Cilium k8sServiceHost CNI outage)")
|
|
check("--include-scope" in L0,
|
|
"ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is "
|
|
"a purely requirement-scoped run again and the scope annotation is inert decoration — "
|
|
"this check would then run only by coincidence")
|
|
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
|
|
"ci/gate.sh does not set the merge block when the l0 stage fails")
|
|
|
|
# ------------------------------------------------------------- advisory: dead knobs
|
|
for key in sorted(KEYS - set(referenced)):
|
|
ADVISORIES.append(f"cluster-vars knob '{key}' is referenced by NO reconciled manifest "
|
|
f"(dead knob, or documentation-only — advisory, not a failure)")
|
|
|
|
# ------------------------------------------------------------------------- report
|
|
print("Flux postBuild substitution inventory (per Kustomization):")
|
|
for line in inventory:
|
|
print(line)
|
|
print(f"\ncluster-vars knobs defined: {len(KEYS)} referenced: {len(referenced)}")
|
|
print(f"dollar runs >=2 in substituted files: {n_runs_ok} documented $${{...}} escapes, "
|
|
f"{sum(e['used'] for e in allow_entries)} justified by dollar-collapse.allow "
|
|
f"({len(allow_entries)} entr{'y' if len(allow_entries) == 1 else 'ies'}), "
|
|
f"{len(run_findings)} unjustified")
|
|
|
|
if ADVISORIES:
|
|
print("\nADVISORIES (not failures):")
|
|
for a in ADVISORIES:
|
|
print(f" - {a}")
|
|
|
|
if findings:
|
|
print("\nHAZARDS — unescaped expansions that are not cluster-vars knobs:", file=sys.stderr)
|
|
for f in findings:
|
|
print(f, file=sys.stderr)
|
|
|
|
if run_findings:
|
|
print("\nHAZARDS — dollar runs envsubst rewrites between git and the container:",
|
|
file=sys.stderr)
|
|
for f in run_findings:
|
|
print(f, file=sys.stderr)
|
|
|
|
print()
|
|
if not FAILURES:
|
|
print(f"check OK: flux-postbuild-substitution — {CHECKS} assertions "
|
|
f"(R-ENV-02, R-GITOPS-01)")
|
|
raise SystemExit(0)
|
|
print(f"CHECK FAIL (flux-postbuild-substitution): {len(FAILURES)} failed, "
|
|
f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
|
|
for f in FAILURES:
|
|
print(f" FAIL {f}", file=sys.stderr)
|
|
raise SystemExit(1)
|
|
PY
|