jakman85andClaude Opus 5
377fa205ea
INCOMPLETE — preserved by the PM after the authoring agent hit the session
rate limit mid-task. Do not merge without finishing and gating it.
Closes the inverse of what this check already covered. It asserted that every
${...} in a substituted path is defined or escaped; it did NOT catch content
containing a literal doubled dollar that the author needs delivered verbatim
and that envsubst silently halves. That shipped a broken CronJob (fixed in
16b7fcd): Postgres dollar-quoting became a single dollar and every
Flux-reconciled run died with a syntax error, while passing review because
verification used kubectl apply, which runs no envsubst.
Introduces a dollar-collapse.allow allowlist so each legitimate use is a
reviewed decision rather than an accident.
WHAT REMAINS: the agent's own next step was proving it catches the CLASS and
not just this instance, via a scratch manifest under a substituted path. Its
scratch fixture (apps/wave-2/zz-scratch-class.yaml and the wave-2
kustomization edit) was deliberately temporary and has been reverted here, so
the fail-proof is NOT yet demonstrated. Also unanswered: brief point 3, on
whether the check can say anything about a manifest verified only through a
non-substituting path.
Co-Authored-By: Claude Opus 5 <[email protected]>
109 lines
7.1 KiB
YAML
109 lines
7.1 KiB
YAML
# =============================================================================
|
|
# L0 STATIC — Flux postBuild substitution is complete and fail-closed-safe
|
|
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
|
#
|
|
# R-ENV-02 ("authored once, promoted dev -> prod") is implemented by ONE mechanism:
|
|
# `postBuild.substituteFrom cluster-vars` injecting `${TOKEN}`s into the manifests each
|
|
# Flux Kustomization renders. That mechanism FAILS OPEN — an undefined variable becomes
|
|
# the empty string while the Kustomization still reports Ready — which emptied Cilium's
|
|
# k8sServiceHost and broke the CNI on 2/3 dev nodes (DECISIONS-LOG 2026-09-01).
|
|
#
|
|
# It is also LOSSY IN THE OTHER DIRECTION, and that is the second incident on this record.
|
|
# envsubst consumes dollars in PAIRS and emits one per pair, unconditionally and everywhere —
|
|
# so no substituted path can carry a literal `$$` at all. On 2026-09-13 a ConfigMap under
|
|
# apps-wave-0 shipped a dollar-quoted PL/pgSQL `DO $$ ... $$;` self-check; Postgres received
|
|
# a half-quoted `DO $` and every Flux-reconciled run of the nextcloud-db grant reconciler died
|
|
# with a syntax error, nine Error pods deep before anyone looked (commit 16b7fcd). It passed
|
|
# review because it was hand-applied with `kubectl apply`, which runs no envsubst: the defect
|
|
# existed ONLY on the rendered path, and it landed on the assertion whose whole job was to
|
|
# make a silent regression visible. Nothing here failed — the manifest rendered
|
|
# "successfully" into something broken, because the check treated any doubled dollar as a
|
|
# deliberate escape and stopped looking.
|
|
#
|
|
# GIVEN every Flux Kustomization under clusters/ that declares postBuild.substituteFrom,
|
|
# WHEN each `spec.path` is resolved to the exact file set it renders (following
|
|
# kustomization.yaml `resources:` transitively, not globbing the repo),
|
|
# THEN every `${...}` in those files is either DEFINED in clusters/base/cluster-vars.yaml
|
|
# or deliberately ESCAPED `$${...}`; no unescaped expansion carries a default
|
|
# operator on an undefined name (the silent-empty class strict mode does NOT catch);
|
|
# no unescaped expansion is unparseable by envsubst (which fails the WHOLE
|
|
# Kustomization); NO RUN OF TWO OR MORE DOLLARS IS SILENTLY REWRITTEN — the only shape
|
|
# that passes unremarked is exactly `$${`, the documented total escape, and every other
|
|
# run (a bare `$$` that cannot survive verbatim, an ODD `$$${...}` that only LOOKS
|
|
# escaped and in fact expands, an over-counted `$$$${...}`) is named and justified in
|
|
# `dollar-collapse.allow`, which is itself held to being exact and non-stale;
|
|
# every path rendering a knob actually substitutes cluster-vars;
|
|
# every overlay-patched knob has a base default; and the check is wired into the
|
|
# merge gate as a blocking, unconditionally-run stage.
|
|
#
|
|
# This is the prerequisite proof for enabling kustomize-controller's
|
|
# StrictPostBuildSubstitutions feature gate, and the permanent guard that keeps it safe.
|
|
#
|
|
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library
|
|
# only); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw
|
|
# Test wraps the SAME script.
|
|
#
|
|
# NOT proven, and deliberately NOT proxied: the wider lesson of 2026-09-13 is that the bug
|
|
# survived review because verification used a path (`kubectl apply`) that runs no
|
|
# substitution. That property is NOT statically decidable from this repo — which command
|
|
# someone ran is a fact about a shell session, and git stores the manifest, never the
|
|
# invocation. Grepping commit prose or runbooks for `kubectl apply` would be trivially
|
|
# satisfiable by rewording and would fire on the break-glass runbooks whose whole content is
|
|
# that command, so no such proxy is attempted. The dollar-run assertion is the honest answer
|
|
# instead: it moves the proof off the verification PATH and onto the SOURCE, where it IS
|
|
# decidable, so the mangling is caught by reading the manifest however a human chose to try
|
|
# it. An end-to-end `kustomize build | flux envsubst --strict` per substituted path would be
|
|
# the real thing, and it is how the arithmetic in check.sh's header was measured — but the L0
|
|
# CI toolchain installs neither tool, so as a gate stage it would SKIP permanently.
|
|
#
|
|
# NOT proven (named in check.sh's header): this is a STATIC scan of the SOURCE manifests
|
|
# a path includes — a deliberate superset of the post-`kustomize build` stream. It does
|
|
# not render, does not prove the feature gate is enabled on the live kustomize-controller,
|
|
# and does not validate knob VALUES. R-GITOPS-01 is tagged for the narrow static slice
|
|
# only: what git says a Kustomization renders is fully determined, with no variable
|
|
# silently emptied on the way in. The live-state half is L2, untested today.
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: flux-postbuild-substitution
|
|
labels:
|
|
conformance.home.k3s/R-ENV-02: "true"
|
|
conformance.home.k3s/R-GITOPS-01: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-ENV-02, R-GITOPS-01"
|
|
conformance.home.k3s/layer: "L0"
|
|
# WHOLE-REPO SCOPE — relocated verbatim from ci/l0.sh's hand-written "Stage 3c
|
|
# (UNCONDITIONAL)", which this annotation replaces:
|
|
#
|
|
# R-ENV-02's promote-once mechanism is `postBuild.substituteFrom cluster-vars`, and it
|
|
# FAILS OPEN: an undefined ${TOKEN} becomes the empty string and the Kustomization still
|
|
# reports Ready. That emptied Cilium's k8sServiceHost and broke the CNI on 2/3 dev nodes
|
|
# (DECISIONS-LOG 2026-09-01, "substitution race that fails OPEN").
|
|
#
|
|
# WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh's coarse path map
|
|
# keys only clusters/* to R-ENV-02 — an edit to a SUBSTITUTED manifest under apps/ or
|
|
# platform/ attributes no requirement, so the requirement-scoped selection would filter
|
|
# this check out of exactly the change that introduces the bug. The invariant spans every
|
|
# path any Flux Kustomization substitutes (apps/, platform/, platform-config/), so it runs
|
|
# on EVERY gate invocation.
|
|
conformance.home.k3s/scope: "whole-repo"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
Every variable a substituted Flux path renders is defined in cluster-vars or
|
|
escaped, so post-build substitution cannot silently empty a value and enabling
|
|
StrictPostBuildSubstitutions cannot fail a legitimate manifest; and no dollar run
|
|
envsubst rewrites reaches the cluster unreviewed, so content that must arrive
|
|
verbatim cannot be silently mangled on the Flux-rendered path (16b7fcd).
|
|
spec:
|
|
steps:
|
|
- name: audit-every-substituted-flux-path
|
|
description: >-
|
|
Resolve each postBuild.substituteFrom Kustomization to the files it renders and
|
|
assert every ${...} is defined or escaped and no dollar run is silently collapsed,
|
|
then assert the gate wires it blocking.
|
|
try:
|
|
- script:
|
|
content: bash ./check.sh
|