jakman85andClaude Opus 5
830983b791
The implementation landed in the preceding WIP commit had never been executed. This finishes it: the arithmetic is re-measured, the guard and its allowlist are attacked, and the header's one factual error is corrected. WHAT ENVSUBST ACTUALLY IS. The header credited `github.com/drone/envsubst`. The pinned flux 2.4.0 binary in the dev image embeds `github.com/fluxcd/pkg/[email protected]` — Flux's own fork of drone/envsubst ("This package is a fork of drone/envsubst", its README) — read straight out of the binary with `grep -ao`. Same behaviour, different module; the header now says so and says how to check, because a claim about a dependency that nobody can re-derive is how the last wrong assumption survived. RE-MEASURED, NOT ASSERTED. Every row of the DOLLAR-RUN ARITHMETIC table was re-run through `flux envsubst` (flux 2.4.0, DEFINED=VALUE) and every row holds: `$$`->`$`, `$$$`->`$$`, `$${D}`->`${D}`, `$$${D}`->`$VALUE` (odd run: the brace is LIVE), `$$$${D}`->`$${D}`, `DO $$ x $$;`->`DO $ x $;`, and `$$${ARR[0]}` dies with "missing closing brace" while `$${ARR[0]}` is fine. `--strict` output was byte-identical to non-strict, exit 0 both ways (`cmp`) — strict mode is blind to this whole class, which is why a source assertion is the only guard available. The header now carries the 30-second reproduction, flagged to be re-measured on any FLUX_VERSION bump. END TO END ON THE REAL PATH. With the pre-16b7fcd manifest restored, `kubectl kustomize apps/wave-0` emits `DO $$` — valid SQL, and exactly what the `kubectl apply` verification saw — while the same stream through `flux envsubst --strict` emits `DO $` and `$;` and exits 0. That two-command contrast IS the incident. THE GUARD IS NOT TRIVIALLY SATISFIABLE — 16 adversarial cases, all as intended, run against a throwaway copy of the tree so nothing could be left behind: * an unmarked `$$` under a substituted path FAILS (DOLLAR-COLLAPSE); so do an odd `$$${...}` (DOLLAR-HALF-ESCAPED) and an even `$$$${...}` (OVER-ESCAPED); * a stale entry FAILS three ways — context no longer in the file, context that only covers an already-accepted `$${` escape and so excuses nothing, and a path that is not a file in the repo; * the allowlist cannot blanket-exempt a path: a 2-char context is rejected (<24), a `.*` wildcard is matched VERBATIM and therefore rejected as stale, a 24-char context carrying no dollar run is rejected, a 4-char reason is rejected (<40) — and, the central case, a genuine entry covers only the span its context matches, so a SECOND `$$` elsewhere in the same file still fails. Two genuine entries for two real occurrences do pass, so the mechanism is usable rather than merely unsatisfiable; * the CLASS, not the instance: the pre-fix `DO $$ ... $$;` reconstructed in a NEW ConfigMap under apps/wave-2 fails at both dollar runs, and restoring the real pre-16b7fcd apps/wave-0/nextcloud-db.yaml verbatim fails the check at lines 384 and 391. The guard rejects the exact commit that broke production. The scratch fixtures lived only in the throwaway copy; `git status` on this branch stayed clean throughout, which is the other half of the last attempt's unfinished business. LEGITIMATE USES EXIST AND STAY GREEN. 18 `$${...}` escapes (unifi cloud-init, the gitea and nextcloud OIDC bootstrap scripts, two longhorn comments) pass unremarked, and the 2 non-braced `$$` occurrences are the two allowlist entries. Those two are shell comments inside block scalars, and `kubectl kustomize` confirms the comment text really does survive into the stream envsubst reads — so they are genuine, reaching content whose collapse is cosmetic, not a hypothetical the allowlist was invented to excuse. STILL NOT PROVEN, AND STILL NOT PROXIED: that a change was verified through a substituting path. Which command a human ran is a fact about a shell session; git stores the manifest, never the invocation, so it is not statically decidable here and no honest formulation exists. Grepping commit prose for `kubectl apply` would be satisfiable by rewording and would fire on the break-glass runbooks whose whole content is that command. The answer is to move the proof off the verification path and onto the source, which is what assertion (7) does. No ci/l0.sh stage: the `conformance.home.k3s/scope: "whole-repo"` annotation is what wires it, and whole-repo-invariants-wired fails the gate on a hand-written stage. Co-Authored-By: Claude Opus 5 <[email protected]>
669 lines
39 KiB
Bash
Executable File
669 lines
39 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# =============================================================================
|
|
# L0 static check (offline, no cluster) — flux-postbuild-substitution
|
|
# Serves: R-ENV-02 ("Same GitOps config promotes dev/test -> prod (authored once)."
|
|
# — spec docs/02-environments-and-provisioning.md)
|
|
# R-GITOPS-01 ("All desired state in git, reconciled by Flux."
|
|
# — spec docs/05-gitops-and-delivery.md)
|
|
#
|
|
# WHY THIS EXISTS — Flux post-build substitution FAILS OPEN
|
|
# --------------------------------------------------------
|
|
# R-ENV-02 is implemented by exactly one mechanism: `clusters/base/cluster-vars.yaml`
|
|
# is injected into every `${TOKEN}` of the manifests each Flux Kustomization renders,
|
|
# via `postBuild.substituteFrom`. That mechanism is silently lossy by default: an
|
|
# UNDEFINED variable becomes the EMPTY STRING and the Kustomization still reports
|
|
# Ready. On 2026-09-01 that emptied `k8sServiceHost` in the Cilium HelmRelease and put
|
|
# cilium into Init:Error on 2 of 3 dev nodes (DECISIONS-LOG "substitution race that
|
|
# fails OPEN"). "Authored once, promoted" is therefore only true while every token a
|
|
# substituted path renders is either DEFINED or deliberately ESCAPED — and nothing in
|
|
# the repo asserted that. This check does.
|
|
#
|
|
# SECOND INCIDENT, 2026-09-13 — THE INVERSE HAZARD (commit 16b7fcd)
|
|
# ------------------------------------------------------------------
|
|
# The first incident was a variable the author wanted EXPANDED that rendered EMPTY. This one
|
|
# is its mirror: content the author wanted to reach the workload UNTOUCHED that substitution
|
|
# silently REWROTE. A CronJob reconciler in apps/wave-0/nextcloud-db.yaml asserted its own
|
|
# convergence in a dollar-quoted PL/pgSQL block —
|
|
#
|
|
# DO <dq> BEGIN ... END <dq>; where <dq> is two literal dollars: standard Postgres
|
|
#
|
|
# — inside a ConfigMap under apps-wave-0, which declares postBuild.substituteFrom. envsubst
|
|
# collapses a doubled dollar into ONE dollar, ALWAYS, so Postgres received a half-quoted
|
|
# `DO <one dollar>` and died with `syntax error at or near "$"`. Three scheduled runs left
|
|
# nine Error pods in nextcloud-db before anyone looked.
|
|
#
|
|
# It PASSED verification because it was hand-applied with `kubectl apply`, which runs NO
|
|
# envsubst. The defect existed ONLY on the Flux-rendered path. And the damage landed
|
|
# precisely on the self-check: the GRANT on line 1 still ran, so the privilege stayed
|
|
# correct and the only thing broken was the assertion meant to make a silent regression
|
|
# visible — the part whose whole job is to stop a blind spot being quiet.
|
|
#
|
|
# THE ASSERTION GAP THIS CLOSES: assertions (1)-(3) below ask whether every `${...}` is
|
|
# defined, escaped, or parseable. They said NOTHING about a doubled dollar that is not an
|
|
# escape, because the old classifier treated a `$` run of length >= 2 as "escaped, therefore
|
|
# fine" and stopped looking. Nothing failed the gate; the manifest rendered "successfully"
|
|
# into something broken. Assertion (7) is that hole, closed.
|
|
#
|
|
# ENVSUBST BEHAVIOUR ASSERTED HERE (empirically verified against the pinned toolchain,
|
|
# `flux envsubst`, flux 2.4.0 — this is measured behaviour, not assumption):
|
|
# ${DEFINED} -> substituted with the cluster-vars value.
|
|
# ${UNDEFINED} -> EMPTY STRING, silently. `--strict` / the cluster-wide
|
|
# StrictPostBuildSubstitutions feature gate turns this into a
|
|
# loud Kustomization failure ("variable not set (strict mode)").
|
|
# ${UNDEFINED:-d} -> the DEFAULT is applied AT RECONCILE TIME. Strict mode does
|
|
# NOT catch this (it exempts default-operator forms by design),
|
|
# so a shell guard written `[ -z "${X:-}" ]` inside a rendered
|
|
# manifest is pre-resolved to `[ -z "" ]` and its else-branch
|
|
# becomes unreachable in the container. Silent, and permanent.
|
|
# $${VAR}, $${V:-d}, $${ARR[0]} -> emitted as the LITERAL ${...} for a downstream
|
|
# shell/cloud-init consumer. Escaping is total: it neutralises
|
|
# even the indexed form.
|
|
# $VAR, $(cmd) -> left alone by envsubst; they need no escaping.
|
|
#
|
|
# DOLLAR-RUN ARITHMETIC — the exact rule, MEASURED, not inferred (see assertion (7))
|
|
# ---------------------------------------------------------------------------------
|
|
# envsubst here is `github.com/fluxcd/pkg/envsubst` v1.1.0 — Flux's own maintained FORK of
|
|
# `github.com/drone/envsubst` ("This package is a fork of drone/envsubst", its README) — and it
|
|
# is exposed by the `flux envsubst` CLI ("replicate the behavior of the Flux Kustomization
|
|
# post-build substitutions", its own --help). Named from the binary, not from memory: the
|
|
# pinned flux 2.4.0 in the dev image embeds the Go module path
|
|
# `/home/runner/go/pkg/mod/github.com/fluxcd/pkg/[email protected]/parse`
|
|
# (`grep -ao 'github.com/fluxcd/pkg/envsubst[^ ]*' "$(command -v flux)"`). Its scanner walks
|
|
# the input left to right; on a `$` it looks at the NEXT byte:
|
|
# * another `$` -> BOTH are consumed and ONE `$` is emitted. This is UNCONDITIONAL. What
|
|
# follows the pair is irrelevant — `{`, a letter, a digit, a semicolon,
|
|
# a newline, end of input. `$$` is not "the escape for `${`"; it is the
|
|
# escape for `$` itself, everywhere.
|
|
# * `{` -> the braced expression is parsed and EXPANDED.
|
|
# * anything else -> the `$` is emitted literally (hence `$VAR` and `$(cmd)` survive).
|
|
# So a run of N consecutive dollars emits floor(N/2) literal dollars, and when N is ODD the
|
|
# final dollar is still LIVE. Measured with flux 2.4.0 `flux envsubst`, DEFINED=VALUE — the
|
|
# left column is authored bytes, the right is what reaches the cluster:
|
|
#
|
|
# $ -> $ ${DEFINED} -> VALUE
|
|
# $$ -> $ $${DEFINED} -> ${DEFINED} <- the escape
|
|
# $$$ -> $$ $$${DEFINED} -> $VALUE <- ODD: brace LIVE
|
|
# $$$$ -> $$ $$$${DEFINED} -> $${DEFINED}
|
|
# $$$$$ -> $$$ $$$$${DEFINED} -> $$VALUE
|
|
# DO $$ .. $$; -> DO $ .. $; $$${NOPE:-d} -> $d
|
|
# awk '{print $$2}' -> awk '{print $2}' $${ARR[0]} -> ${ARR[0]} (safe)
|
|
# $$@ $$< $$^ -> $@ $< $^ $$${ARR[0]} -> HARD PARSE ERROR, whole
|
|
# Kustomization dies
|
|
#
|
|
# TWO CONSEQUENCES THIS CHECK IS BUILT ON:
|
|
# (a) THERE IS NO WAY TO SHIP A LITERAL `$$` THROUGH A SUBSTITUTED PATH. To make two
|
|
# dollars arrive you must author FOUR. So any authored `$$` that is meant to reach the
|
|
# workload verbatim is a BUG, full stop — that is the 2026-09-13 incident, and Postgres
|
|
# dollar-quoting is its commonest shape (Makefile `$$@`, awk `$$2` and docker-compose
|
|
# `$${}` are the same hazard arriving from the other direction: there the collapse is
|
|
# what the author wanted).
|
|
# (b) `$$$` + brace is NOT "more escaped" than `$$` + brace — it is NOT escaped at all. An
|
|
# odd run leaves one live dollar, so the brace expands, or, for an inner envsubst
|
|
# cannot parse, fails the ENTIRE Kustomization (proven above with $$${ARR[0]}, which
|
|
# aborted the whole stream while $${ARR[0]} on the line before it was fine).
|
|
# `--strict` reports NONE of this: every row of that table except the parse error was
|
|
# accepted SILENTLY by `flux envsubst --strict`, exit 0, and its output was BYTE-IDENTICAL to
|
|
# non-strict (`cmp`). Strict mode cannot see this class, so a source-level assertion is the
|
|
# only available guard.
|
|
#
|
|
# TO REPRODUCE THE TABLE (30 seconds, no cluster) — it is a measurement, so re-measure it
|
|
# after any FLUX_VERSION bump in infra/devcontainer/versions.env. Open the toolbox with
|
|
# `./infra/devcontainer/dev.sh` and run these INSIDE it (do NOT pipe into `dev.sh run` from
|
|
# the host: it starts the container with `-it`, so a piped stdin hangs):
|
|
# printf '[$$] [$$$] [$${D}] [$$${D}] [DO $$ x $$;]\n' | D=VALUE flux envsubst
|
|
# -> [$] [$$] [${D}] [$VALUE] [DO $ x $;]
|
|
# And end-to-end, on the actual delivery path, with the pre-16b7fcd manifest checked out
|
|
# (`git show 16b7fcd^:apps/wave-0/nextcloud-db.yaml > apps/wave-0/nextcloud-db.yaml`):
|
|
# kubectl kustomize apps/wave-0 | grep 'DO \$' -> "DO $$" (the kubectl path)
|
|
# kubectl kustomize apps/wave-0 | flux envsubst --strict | grep 'DO \$'
|
|
# -> "DO $" (the FLUX path), exit 0
|
|
# (`kubectl kustomize` IS the kustomize build here; the image ships no standalone binary.)
|
|
# That two-command contrast IS the 2026-09-13 incident, and is why review missed it.
|
|
# ${...}, ${a b}, ${ARR[0]} (UNESCAPED) -> HARD PARSE ERROR ("unable to parse
|
|
# variable name" / "missing closing brace") which fails the
|
|
# WHOLE Kustomization — every resource in the path, not just
|
|
# the offending one.
|
|
# The escape contract this enforces is the one already documented in
|
|
# `clusters/base/apps-vms.yaml`; this check is that comment made executable.
|
|
#
|
|
# GIVEN the repository as authored,
|
|
# WHEN every Flux Kustomization under clusters/ that declares postBuild.substituteFrom
|
|
# is resolved to the exact file set its `spec.path` renders (by following
|
|
# kustomization.yaml `resources:` transitively — a path, not a glob),
|
|
# THEN (1) every `${...}` in those files is DEFINED in cluster-vars, or ESCAPED `$${...}`;
|
|
# (2) no UNESCAPED expansion carries a default/length/substitution operator on an
|
|
# undefined name (the silent-empty class strict mode does not catch);
|
|
# (3) no UNESCAPED expansion is unparseable by envsubst (`${...}`, `${ARR[0]}`) —
|
|
# that fails the entire Kustomization, so it is fatal wherever it appears;
|
|
# (4) every Kustomization that renders a cluster-vars token actually declares
|
|
# `substituteFrom` a ConfigMap named cluster-vars (else the literal token
|
|
# ships into the cluster);
|
|
# (5) every knob an overlay patches (`/data/KEY`) exists in the BASE ConfigMap —
|
|
# the "add the knob with a base default BEFORE referencing it" rule that the
|
|
# incident's interim discipline depends on;
|
|
# (7) NO DOLLAR RUN IS SILENTLY REWRITTEN: the only run of length >= 2 that passes
|
|
# unremarked is exactly `$$` immediately followed by `{` — the documented, total,
|
|
# already-inventoried escape. EVERY other run of >= 2 dollars (a bare `$$`, an odd
|
|
# `$$$`, an over-counted `$$$$`) is rewritten by envsubst between git and the
|
|
# container, so it must be named and justified in `dollar-collapse.allow`, a file
|
|
# this check also holds to being exact and non-stale;
|
|
# AND (6) this check is LOAD-BEARING: it DECLARES `conformance.home.k3s/scope: whole-repo`,
|
|
# so ci/l0.sh's `--include-scope whole-repo` runs it on every gate run, and
|
|
# ci/gate.sh sets the merge block when the l0 stage fails.
|
|
# Assertion (1)-(3) are applied to COMMENTS as well as content: `kustomize build`
|
|
# preserves body comments into the stream envsubst reads (verified: a `$`-carrying
|
|
# comment survives into the built apps/wave-1 output), so a braced expression in a
|
|
# comment is a live hazard, not prose.
|
|
#
|
|
# WHAT THIS DOES *NOT* PROVE (honest gaps — this is a STATIC check):
|
|
# - It does NOT render the manifests. It scans the SOURCE files a path includes,
|
|
# which is a deliberate SUPERSET of the post-`kustomize build` stream (kustomize
|
|
# drops some comments). A finding in a comment may not reach envsubst TODAY; it is
|
|
# still reported, because whether a given comment survives the build is a kustomize
|
|
# implementation detail that changes with the pinned version, and the failure it
|
|
# causes is total (assertion 3). The end-to-end proof — `kustomize build <path> |
|
|
# flux envsubst --strict` for every substituted path — needs kustomize + flux,
|
|
# which the L0 CI toolchain does not install (.gitea/workflows/scripts/install-tools.sh
|
|
# l0), so it cannot run here without permanently SKIPPING. It was run by hand and
|
|
# recorded in DECISIONS-LOG; this check is the reproducible static half.
|
|
# - It does NOT prove the cluster's live state matches git, nor that the
|
|
# StrictPostBuildSubstitutions feature gate is actually enabled on
|
|
# kustomize-controller. Both are cluster facts (L2), untested here. Tagging
|
|
# R-GITOPS-01 claims ONLY the static slice: what git says a Kustomization renders
|
|
# is fully determined, with no variable silently emptied on the way in.
|
|
# - It CANNOT prove the thing that actually let 16b7fcd's bug reach a cluster: that a
|
|
# change was VERIFIED through a substituting path. "It was hand-applied with kubectl" is
|
|
# a fact about a shell session, not about any tracked file — git records the manifest,
|
|
# never the command someone ran to try it, so the property is not statically decidable
|
|
# here and no honest formulation of it exists. Nor is a proxy worth having: grepping
|
|
# commit prose or runbooks for `kubectl apply` would be trivially satisfiable by
|
|
# rewording, and would fire on the many legitimate break-glass runbooks whose whole
|
|
# content is that command. So this check does not attempt it, and does not pretend to.
|
|
# Assertion (7) answers that incident a different way: it moves the proof off the
|
|
# verification PATH, which is unknowable, and onto the SOURCE, which is not — the
|
|
# mangling is now caught by reading the manifest, whichever way a human chose to try it.
|
|
# (The one decidable half-measure, `kustomize build <path> | flux envsubst --strict` per
|
|
# substituted path in CI, is already ruled out above: the L0 toolchain installs neither
|
|
# tool, so the stage would SKIP permanently. The dev image has both, and it is how the
|
|
# arithmetic table above was measured.)
|
|
# - It does NOT validate cluster-vars VALUES (that GATEWAY_VIP is inside the LB pool,
|
|
# that an overlay's LAN block is self-consistent). Presence, not correctness.
|
|
# - A cluster-vars key nothing references is reported as an ADVISORY, never a failure:
|
|
# several are deliberately reference-only and say so in cluster-vars.yaml
|
|
# (PIA_REGION "retained, not substituted"; CLUSTER_POD_CIDR "Documents ..."), so
|
|
# failing on them would be a false block on documented intent.
|
|
# =============================================================================
|
|
set -euo pipefail
|
|
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
|
PYTHON="${PYTHON:-python3}"
|
|
|
|
# Honest SKIP (never green) if the thing under test is not present at all.
|
|
if [ ! -d "${ROOT}/clusters" ] || [ ! -f "${ROOT}/clusters/base/cluster-vars.yaml" ]; then
|
|
echo "check SKIPPED: clusters/ or clusters/base/cluster-vars.yaml absent — nothing to audit" >&2
|
|
exit 3
|
|
fi
|
|
|
|
cd "${ROOT}"
|
|
"${PYTHON}" - "${ROOT}" <<'PY'
|
|
"""Static audit of Flux postBuild substitution safety. Pure standard library.
|
|
|
|
No PyYAML: the file shapes parsed here are line-constrained and the parse contracts are
|
|
documented at each parser, matching lib/coverage.py + lib/status.py. The kustomization
|
|
`resources:` parser is IMPORTED from lib/status.py rather than forked, so there is one
|
|
definition of "what a Flux path actually renders" in the harness.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import bisect
|
|
import os
|
|
import re
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
ROOT = Path(sys.argv[1])
|
|
sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree
|
|
sys.path.insert(0, str(ROOT / "conformance" / "lib"))
|
|
import status as st # noqa: E402 — reuse the walk, don't fork it
|
|
|
|
FAILURES: list[str] = []
|
|
ADVISORIES: list[str] = []
|
|
CHECKS = 0
|
|
|
|
|
|
def check(cond, msg):
|
|
global CHECKS
|
|
CHECKS += 1
|
|
if not cond:
|
|
FAILURES.append(msg)
|
|
|
|
|
|
# ------------------------------------------------------------------ cluster-vars
|
|
# Parse contract: `data:` block of a single ConfigMap, one ` KEY: "value"` per line
|
|
# (every entry in clusters/base/cluster-vars.yaml is a quoted scalar; comments and
|
|
# continuation-comment lines are indented further and never match).
|
|
_DATA_KEY_RE = re.compile(r'^ ([A-Za-z_][A-Za-z0-9_]*):\s*(?:"|\')')
|
|
|
|
|
|
def cluster_vars_keys(path: Path) -> set[str]:
|
|
keys: set[str] = set()
|
|
in_data = False
|
|
for raw in path.read_text().splitlines():
|
|
if re.match(r"^data:\s*$", raw):
|
|
in_data = True
|
|
continue
|
|
if not in_data:
|
|
continue
|
|
if st._TOP_KEY_RE.match(raw): # a new top-level key ends the data block
|
|
break
|
|
m = _DATA_KEY_RE.match(raw)
|
|
if m:
|
|
keys.add(m.group(1))
|
|
return keys
|
|
|
|
|
|
VARS_FILE = ROOT / "clusters" / "base" / "cluster-vars.yaml"
|
|
KEYS = cluster_vars_keys(VARS_FILE)
|
|
check(len(KEYS) > 0, f"no data keys parsed from {VARS_FILE} — the parser or the file shape changed")
|
|
|
|
# ------------------------------------------- Flux Kustomizations with substituteFrom
|
|
# Parse contract mirrors lib/status.py's flux_paths(): docs split on a lone `---`, a doc
|
|
# is a Flux entrypoint only if it carries the kustomize.toolkit apiVersion AND
|
|
# `kind: Kustomization` AND a spec-level `path:`. Inline `patch: |-` bodies in the
|
|
# overlays carry Flux Kustomizations WITHOUT a path and are correctly skipped.
|
|
_SUBFROM_RE = re.compile(r"^(\s*)substituteFrom:\s*$", re.M)
|
|
_SUBITEM_KIND_RE = re.compile(r"^\s*-\s*kind:\s*(\S+)")
|
|
_SUBITEM_NAME_RE = re.compile(r"^\s*(?:-\s*)?name:\s*([^\s#]+)")
|
|
|
|
|
|
def substitute_sources(doc: str) -> list[tuple[str, str]]:
|
|
"""[(kind, name)] declared under postBuild.substituteFrom in this doc."""
|
|
m = _SUBFROM_RE.search(doc)
|
|
if not m:
|
|
return []
|
|
indent = len(m.group(1))
|
|
out: list[tuple[str, str]] = []
|
|
kind = ""
|
|
for raw in doc[m.end():].splitlines():
|
|
if raw.strip() == "" or raw.lstrip().startswith("#"):
|
|
continue
|
|
if len(raw) - len(raw.lstrip()) <= indent and not raw.lstrip().startswith("-"):
|
|
break # dedented out of the list
|
|
km = _SUBITEM_KIND_RE.match(raw)
|
|
if km:
|
|
kind = km.group(1)
|
|
continue
|
|
nm = _SUBITEM_NAME_RE.match(raw)
|
|
if nm and kind:
|
|
out.append((kind, nm.group(1)))
|
|
kind = ""
|
|
return out
|
|
|
|
|
|
kustomizations: list[dict] = []
|
|
for yfile in sorted((ROOT / "clusters").rglob("*.yaml")):
|
|
text = yfile.read_text()
|
|
if st.FLUX_API not in text:
|
|
continue
|
|
for doc in st._DOC_SPLIT_RE.split(text):
|
|
if st.FLUX_API not in doc or not re.search(r"^kind:\s*Kustomization\s*$", doc, re.M):
|
|
continue
|
|
pm = st._PATH_RE.search(doc)
|
|
if not pm:
|
|
continue
|
|
nm = st._NAME_RE.search(doc)
|
|
srcs = substitute_sources(doc)
|
|
if not srcs:
|
|
continue # no substitution => nothing to audit
|
|
kustomizations.append({
|
|
"file": str(yfile.relative_to(ROOT)),
|
|
"name": nm.group(1) if nm else "?",
|
|
"path": pm.group(1),
|
|
"sources": srcs,
|
|
})
|
|
|
|
check(len(kustomizations) > 0,
|
|
"no Flux Kustomization with postBuild.substituteFrom found under clusters/ — "
|
|
"the parser or the repo layout changed; this check would silently audit nothing")
|
|
|
|
# ------------------------------------------------------------------ token scanning
|
|
# One `$`-run followed by a braced expression. `[^{}]*` is exhaustive for this tree:
|
|
# a repo-wide sweep found no nested/unclosed brace expression in any manifest (only
|
|
# Gitea Actions `${{ ... }}` in .gitea/, which no Flux Kustomization renders).
|
|
TOKEN_RE = re.compile(r"(\$+)\{([^{}]*)\}")
|
|
NAME_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
|
|
# An expansion whose name is followed by a POSIX parameter operator. envsubst EVALUATES
|
|
# these, so an undefined name yields the default/empty AT RECONCILE TIME.
|
|
OP_RE = re.compile(r"^([A-Za-z_][A-Za-z0-9_]*)(:[-=?+].*|:[0-9].*|/.*|#.*|%.*|\^.*|,.*)$")
|
|
LEN_RE = re.compile(r"^#([A-Za-z_][A-Za-z0-9_]*)$")
|
|
|
|
|
|
def classify(dollars: int, inner: str) -> tuple[str, str] | None:
|
|
"""Return (severity-tag, remediation) for a hazardous expansion, else None.
|
|
|
|
PARITY, not ">= 2" (the 2026-09-13 correction). envsubst consumes dollars in PAIRS and
|
|
emits one per pair, so a run of N dollars in front of a brace leaves the brace ESCAPED
|
|
only when N is EVEN. An ODD run emits floor(N/2) literal dollars and its last dollar is
|
|
still LIVE, so `$$${X}` expands exactly as `${X}` does — and `$$${ARR[0]}` fails the
|
|
whole Kustomization where `$${ARR[0]}` is harmless. The old `dollars >= 2 -> return None`
|
|
called every such run "escaped" and stopped looking. Assertion (7) flags the run itself;
|
|
this function additionally classifies the still-live expansion an odd run carries.
|
|
"""
|
|
if dollars % 2 == 0:
|
|
return None # fully escaped: emitted as a literal
|
|
if NAME_RE.match(inner):
|
|
if inner in KEYS:
|
|
return None # defined knob — the happy path
|
|
return ("UNDEFINED",
|
|
f"strict mode fails the Kustomization; today it renders EMPTY. Either add "
|
|
f"'{inner}' to clusters/base/cluster-vars.yaml with a base default, or "
|
|
f"escape it as $${{{inner}}} if it is meant for a shell/cloud-init consumer")
|
|
m = OP_RE.match(inner) or LEN_RE.match(inner)
|
|
if m:
|
|
base = m.group(1)
|
|
if base in KEYS:
|
|
return None # defined knob with an operator
|
|
return ("SILENT-EMPTY",
|
|
f"envsubst resolves this AT RECONCILE TIME because '{base}' is not a "
|
|
f"cluster-var, so the rendered manifest carries the default, not the "
|
|
f"expression. StrictPostBuildSubstitutions does NOT catch this form. "
|
|
f"Escape it as $${{{inner}}} if the CONTAINER's shell is meant to expand it")
|
|
return ("FATAL-PARSE",
|
|
"envsubst cannot parse this and fails the ENTIRE Kustomization (every resource "
|
|
"in the path). Escape it as $${...} or rewrite the prose without braces")
|
|
|
|
|
|
def rendered_files(path_spec: str) -> tuple[list[str], bool]:
|
|
"""Repo-relative files the Flux `spec.path` renders, resolving `resources:` transitively."""
|
|
rel = os.path.normpath(path_spec.lstrip("./")) if path_spec not in (".", "./") else "."
|
|
if not (ROOT / rel).is_dir():
|
|
return [], False
|
|
applied, _ = st.reconciled_files(ROOT, rel)
|
|
return sorted(applied), True
|
|
|
|
|
|
findings: list[str] = []
|
|
referenced: dict[str, set[str]] = {}
|
|
inventory: list[str] = []
|
|
# rel path -> the Kustomization name(s) that render it, for assertion (7)'s second pass.
|
|
# A file can be rendered by more than one Kustomization; all of them substitute.
|
|
subst_files: dict[str, set[str]] = {}
|
|
|
|
for k in kustomizations:
|
|
files, exists = rendered_files(k["path"])
|
|
check(exists, f"{k['file']}: Kustomization '{k['name']}' renders path '{k['path']}' "
|
|
f"which is not a directory in this repo")
|
|
has_cluster_vars = ("ConfigMap", "cluster-vars") in k["sources"]
|
|
n_def = n_esc = n_bad = 0
|
|
for rel in files:
|
|
f = ROOT / rel
|
|
try:
|
|
text = f.read_text()
|
|
except (OSError, UnicodeDecodeError):
|
|
continue
|
|
subst_files.setdefault(rel, set()).add(k["name"])
|
|
for lineno, line in enumerate(text.splitlines(), 1):
|
|
for m in TOKEN_RE.finditer(line):
|
|
dollars, inner = len(m.group(1)), m.group(2)
|
|
verdict = classify(dollars, inner)
|
|
if verdict is None:
|
|
if dollars % 2 == 0: # PARITY, see classify(): an odd run is
|
|
n_esc += 1 # live, not escaped
|
|
else:
|
|
n_def += 1
|
|
referenced.setdefault(re.split(r"[:/#%^,]", inner)[0] or inner,
|
|
set()).add(k["name"])
|
|
# (4) a DEFINED knob is only substituted if this Kustomization
|
|
# actually pulls cluster-vars in; otherwise the literal ships.
|
|
check(has_cluster_vars,
|
|
f"{rel}:{lineno}: renders cluster-var ${{{inner}}} but "
|
|
f"Kustomization '{k['name']}' ({k['file']}) does not "
|
|
f"substituteFrom ConfigMap/cluster-vars — the literal token "
|
|
f"would ship into the cluster")
|
|
continue
|
|
n_bad += 1
|
|
tag, fix = verdict
|
|
findings.append(f" [{tag}] {rel}:{lineno} {m.group(0)}\n"
|
|
f" rendered by Flux Kustomization '{k['name']}' "
|
|
f"({k['file']}, path {k['path']})\n"
|
|
f" {fix}\n"
|
|
f" | {line.strip()[:120]}")
|
|
inventory.append(f" {k['name']:<16} path={k['path']:<22} files={len(files):<3} "
|
|
f"defined={n_def:<3} escaped={n_esc:<3} hazards={n_bad}")
|
|
|
|
check(not findings,
|
|
f"{len(findings)} unescaped, undefined or unparseable expansion(s) in substituted paths")
|
|
|
|
# ----------------------------------- (7) no dollar run is silently rewritten on the way out
|
|
# THE 2026-09-13 CLASS. envsubst consumes dollars in PAIRS and emits one per pair,
|
|
# unconditionally and everywhere (header: DOLLAR-RUN ARITHMETIC, measured against flux 2.4.0).
|
|
# So every run of >= 2 dollars in a substituted manifest is REWRITTEN between git and the
|
|
# container, and from the text alone the two intents are indistinguishable:
|
|
#
|
|
# (a) a deliberate escape the author WANTS collapsed — `$${VAR}` -> literal `${VAR}`;
|
|
# (b) content that must arrive VERBATIM — `DO $$ ... $$;` -> `DO $ ... $;`
|
|
# i.e. the incident. A BUG.
|
|
#
|
|
# THE RULE CHOSEN, AND WHY IT IS NEITHER NOISY NOR VACUOUS.
|
|
# Exactly ONE shape passes unremarked: a run of EXACTLY TWO dollars immediately followed by
|
|
# `{`. That is not a guess about intent, it is the escape contract this repo already
|
|
# documents in clusters/base/apps-vms.yaml, that assertions (1)-(3) already validate the
|
|
# inside of, and that 18 reviewed occurrences already rely on (unifi cloud-init, the gitea
|
|
# and nextcloud OIDC bootstrap scripts, two longhorn comments). Failing those would be a
|
|
# permanently-firing alert on documented, correct intent, and this repo has already been
|
|
# bitten by one of those.
|
|
#
|
|
# EVERY OTHER run of >= 2 dollars FAILS unless it is named in `dollar-collapse.allow`:
|
|
# * DOLLAR-COLLAPSE — a run not followed by `{`. Cannot survive verbatim: `$$` -> `$`.
|
|
# This is the incident's exact shape and a blanket bug by default.
|
|
# * DOLLAR-HALF-ESCAPED — an ODD run >= 3 before `{`. Looks escaped, is not: the last
|
|
# dollar is live. Occasionally legitimate (`$$${PRICE}` to render a
|
|
# literal `$` before a value), so it is allowlistable, not banned.
|
|
# * DOLLAR-OVER-ESCAPED — an EVEN run >= 4 before `{`. Emits N/2 dollars then a literal
|
|
# brace; nearly always a miscount, sometimes deliberate.
|
|
#
|
|
# WHY AN ALLOWLIST AND NOT A BLANKET BAN: a blanket ban was the first choice and was
|
|
# rejected on evidence, not taste — main carries two real non-braced `$$` occurrences today
|
|
# (prose inside embedded shell comments), so a blanket ban would have shipped RED. Nor is
|
|
# the allowlist a hole: an entry is a hand-written edit to a file inside the conformance
|
|
# test, which is loud in review; it must quote a >= 24-character VERBATIM context, so it can
|
|
# only ever cover the occurrence it was written for and never a future one that drifts into
|
|
# the same file; it must carry a >= 40-character reason; and every entry must still MATCH
|
|
# something, so the allowlist cannot rot into a blanket exemption after the code it excused
|
|
# is deleted. A wildcard, a bare `$$` context and a stale entry each FAIL.
|
|
ALLOW_FILE = (ROOT / "conformance" / "tests" / "l0-static"
|
|
/ "flux-postbuild-substitution" / "dollar-collapse.allow")
|
|
MIN_CONTEXT = 24
|
|
MIN_REASON = 40
|
|
DOLLAR_RUN_RE = re.compile(r"\$+")
|
|
|
|
allow_entries: list[dict] = []
|
|
check(ALLOW_FILE.is_file(),
|
|
f"{ALLOW_FILE.relative_to(ROOT)} is missing — assertion (7) needs its allowlist to "
|
|
f"exist even when empty, because an absent file and an empty file are the same green "
|
|
f"and the next author would not know the mechanism is there")
|
|
if ALLOW_FILE.is_file():
|
|
for lineno, raw in enumerate(ALLOW_FILE.read_text().splitlines(), 1):
|
|
if not raw.strip() or raw.lstrip().startswith("#"):
|
|
continue
|
|
parts = [x.strip() for x in raw.split("|")]
|
|
if len(parts) != 3:
|
|
check(False, f"dollar-collapse.allow:{lineno}: expected exactly 3 '|'-separated "
|
|
f"fields (path | verbatim-context | reason), got {len(parts)}")
|
|
continue
|
|
allow_entries.append({"line": lineno, "path": parts[0], "ctx": parts[1],
|
|
"reason": parts[2], "used": 0})
|
|
|
|
for e in allow_entries:
|
|
tag = f"dollar-collapse.allow:{e['line']}"
|
|
check((ROOT / e["path"]).is_file(),
|
|
f"{tag}: names '{e['path']}', which is not a file in this repo — a stale entry is "
|
|
f"a standing exemption for a path someone could recreate")
|
|
check(len(e["ctx"]) >= MIN_CONTEXT,
|
|
f"{tag}: context {e['ctx']!r} is {len(e['ctx'])} chars; at least {MIN_CONTEXT} are "
|
|
f"required so an entry cannot be a near-wildcard that swallows future occurrences")
|
|
check(DOLLAR_RUN_RE.search(e["ctx"]) is not None
|
|
and max((len(m.group(0)) for m in DOLLAR_RUN_RE.finditer(e["ctx"])), default=0) >= 2,
|
|
f"{tag}: context {e['ctx']!r} contains no run of two or more dollars, so it excuses "
|
|
f"nothing and is either a typo or a decoy")
|
|
check(len(e["reason"]) >= MIN_REASON,
|
|
f"{tag}: reason is {len(e['reason'])} chars; at least {MIN_REASON} are required. "
|
|
f"State what envsubst will actually emit and why that is acceptable — an entry "
|
|
f"nobody had to explain is an accident wearing an approval")
|
|
|
|
RUN_TAGS = {
|
|
"collapse": ("DOLLAR-COLLAPSE",
|
|
"envsubst collapses each PAIR of dollars into one, everywhere and "
|
|
"unconditionally, so this CANNOT reach the workload verbatim (a literal $$ "
|
|
"needs $$$$ in source). This is the 2026-09-13 nextcloud-db shape: a "
|
|
"dollar-quoted PL/pgSQL DO block arrived half-quoted and every Flux-"
|
|
"reconciled run died. Rewrite without dollars, double them again if the "
|
|
"collapse is what you want, or justify it in dollar-collapse.allow"),
|
|
"half": ("DOLLAR-HALF-ESCAPED",
|
|
"an ODD run of dollars before a brace is NOT escaped: floor(N/2) literal "
|
|
"dollars are emitted and the LAST dollar is still live, so this expansion "
|
|
"EVALUATES at reconcile time (and if its inner is unparseable it fails the "
|
|
"entire Kustomization). Use an EVEN run to escape, or justify it in "
|
|
"dollar-collapse.allow if a literal dollar before a substituted value is "
|
|
"really what you meant"),
|
|
"over": ("DOLLAR-OVER-ESCAPED",
|
|
"an EVEN run of four or more dollars emits N/2 literal dollars followed by a "
|
|
"LITERAL brace — the value is never substituted and extra dollars appear in "
|
|
"the rendered manifest. Almost always a miscount of the escape; justify it in "
|
|
"dollar-collapse.allow if it is not"),
|
|
}
|
|
|
|
run_findings: list[str] = []
|
|
n_runs_ok = 0
|
|
for rel in sorted(subst_files):
|
|
text = (ROOT / rel).read_text(errors="replace")
|
|
# Absolute offsets, not per-line: the gitea/nextcloud bootstrap scripts are single
|
|
# 4000-character folded scalars, so "the line" is not a reviewable unit of context here.
|
|
nl = [i for i, ch in enumerate(text) if ch == "\n"]
|
|
spans = [] # allowlisted [start, end) regions
|
|
for e in allow_entries:
|
|
if e["path"] != rel:
|
|
continue
|
|
start = text.find(e["ctx"])
|
|
while start >= 0:
|
|
spans.append((start, start + len(e["ctx"]), e))
|
|
start = text.find(e["ctx"], start + 1)
|
|
for m in DOLLAR_RUN_RE.finditer(text):
|
|
n = len(m.group(0))
|
|
if n < 2:
|
|
continue
|
|
braced = text[m.end():m.end() + 1] == "{"
|
|
if n == 2 and braced:
|
|
n_runs_ok += 1 # the documented escape; (1)-(3) own it
|
|
continue
|
|
covering = [e for (a, b, e) in spans if a <= m.start() and m.end() <= b]
|
|
if covering:
|
|
for e in covering:
|
|
e["used"] += 1
|
|
continue
|
|
kind = "collapse" if not braced else ("half" if n % 2 else "over")
|
|
tag, fix = RUN_TAGS[kind]
|
|
lineno = bisect.bisect_right(nl, m.start()) + 1
|
|
ctx = text[max(0, m.start() - 60):m.end() + 60].replace("\n", "\\n")
|
|
run_findings.append(
|
|
f" [{tag}] {rel}:{lineno} a run of {n} dollar(s)"
|
|
f"{' before a brace' if braced else ''}\n"
|
|
f" rendered by Flux Kustomization(s) {', '.join(sorted(subst_files[rel]))}"
|
|
f" — postBuild substitution REWRITES it\n"
|
|
f" {fix}\n"
|
|
f" | ...{ctx}...")
|
|
|
|
check(not run_findings,
|
|
f"{len(run_findings)} dollar run(s) in substituted paths that envsubst rewrites and "
|
|
f"nothing justifies (the 2026-09-13 nextcloud-db class)")
|
|
|
|
for e in allow_entries:
|
|
check(e["used"] > 0,
|
|
f"dollar-collapse.allow:{e['line']}: STALE — its context is not present in "
|
|
f"{e['path']} any more (or no longer contains a rewritten dollar run), so the entry "
|
|
f"now excuses nothing. Delete it. A stale allowlist is how an opt-in decays into a "
|
|
f"blanket exemption for whatever lands in that file next")
|
|
|
|
# --------------------------------------------- (5) overlay knobs exist in the base
|
|
# Parse contract: overlays patch cluster-vars with JSON6902 ops (`- op: replace` /
|
|
# `path: /data/KEY`) inside an inline `patch: |-` block. The repo rule (cluster-vars.yaml
|
|
# header) is that a knob is ADDED TO THE BASE with a default BEFORE any overlay or
|
|
# manifest references it — that ordering is what closes the define-then-use race.
|
|
_OP_PATH_RE = re.compile(r"^\s*(?:-\s*)?path:\s*/data/([A-Za-z_][A-Za-z0-9_]*)\s*$", re.M)
|
|
for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
|
|
for key in set(_OP_PATH_RE.findall(overlay.read_text())):
|
|
check(key in KEYS,
|
|
f"{overlay.relative_to(ROOT)}: patches /data/{key} but '{key}' has no BASE "
|
|
f"default in clusters/base/cluster-vars.yaml — a second overlay would render "
|
|
f"${{{key}}} empty (R-ENV-02: author once, supply per env)")
|
|
|
|
# ------------------------------------------------------- (6) the check is load-bearing
|
|
L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
|
|
GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
|
|
# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism.
|
|
# It used to assert `"flux-postbuild-substitution/check.sh" in L0`: ci/l0.sh carried a hand-written
|
|
# "UNCONDITIONAL" stage per whole-repo check, hardcoding this path. Those stages are gone
|
|
# (they did not scale, and the next whole-repo invariant only got one if someone REMEMBERED
|
|
# to write it). The same property is now asserted two ways: this test DECLARES
|
|
# `conformance.home.k3s/scope: "whole-repo"`, and ci/l0.sh unions every declaring test into
|
|
# the run via `--include-scope`. Neither half alone is enough, so both are checked here; the
|
|
# mechanism ITSELF (the union really selects, and a scope-declared SKIP exits 3 rather than
|
|
# passing) is proven end-to-end and falsifiably by
|
|
# conformance/tests/l0-static/whole-repo-invariants-wired.
|
|
SELF_YAML = ROOT / "conformance" / "tests" / "l0-static" / "flux-postbuild-substitution" / "chainsaw-test.yaml"
|
|
SELF = SELF_YAML.read_text() if SELF_YAML.is_file() else ""
|
|
check(re.search(r'^\s*conformance\.home\.k3s/scope:\s*["\']?whole-repo', SELF, re.M) is not None,
|
|
"flux-postbuild-substitution/chainsaw-test.yaml does not declare "
|
|
'conformance.home.k3s/scope: "whole-repo", so the merge gate selects this check only '
|
|
"when a change happens to touch one of the requirements it is labelled with. "
|
|
"The path map keys only clusters/* to R-ENV-02, so an edit to a SUBSTITUTED manifest under apps/ or platform/ attributes nothing and this check would be filtered out of exactly the change that empties a variable (the 2026-09-01 Cilium k8sServiceHost CNI outage)")
|
|
check("--include-scope" in L0,
|
|
"ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is "
|
|
"a purely requirement-scoped run again and the scope annotation is inert decoration — "
|
|
"this check would then run only by coincidence")
|
|
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
|
|
"ci/gate.sh does not set the merge block when the l0 stage fails")
|
|
|
|
# ------------------------------------------------------------- advisory: dead knobs
|
|
for key in sorted(KEYS - set(referenced)):
|
|
ADVISORIES.append(f"cluster-vars knob '{key}' is referenced by NO reconciled manifest "
|
|
f"(dead knob, or documentation-only — advisory, not a failure)")
|
|
|
|
# ------------------------------------------------------------------------- report
|
|
print("Flux postBuild substitution inventory (per Kustomization):")
|
|
for line in inventory:
|
|
print(line)
|
|
print(f"\ncluster-vars knobs defined: {len(KEYS)} referenced: {len(referenced)}")
|
|
print(f"dollar runs >=2 in substituted files: {n_runs_ok} documented $${{...}} escapes, "
|
|
f"{sum(e['used'] for e in allow_entries)} justified by dollar-collapse.allow "
|
|
f"({len(allow_entries)} entr{'y' if len(allow_entries) == 1 else 'ies'}), "
|
|
f"{len(run_findings)} unjustified")
|
|
|
|
if ADVISORIES:
|
|
print("\nADVISORIES (not failures):")
|
|
for a in ADVISORIES:
|
|
print(f" - {a}")
|
|
|
|
if findings:
|
|
print("\nHAZARDS — unescaped expansions that are not cluster-vars knobs:", file=sys.stderr)
|
|
for f in findings:
|
|
print(f, file=sys.stderr)
|
|
|
|
if run_findings:
|
|
print("\nHAZARDS — dollar runs envsubst rewrites between git and the container:",
|
|
file=sys.stderr)
|
|
for f in run_findings:
|
|
print(f, file=sys.stderr)
|
|
|
|
print()
|
|
if not FAILURES:
|
|
print(f"check OK: flux-postbuild-substitution — {CHECKS} assertions "
|
|
f"(R-ENV-02, R-GITOPS-01)")
|
|
raise SystemExit(0)
|
|
print(f"CHECK FAIL (flux-postbuild-substitution): {len(FAILURES)} failed, "
|
|
f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
|
|
for f in FAILURES:
|
|
print(f" FAIL {f}", file=sys.stderr)
|
|
raise SystemExit(1)
|
|
PY
|