# ============================================================================= # L0 STATIC — Flux postBuild substitution is complete and fail-closed-safe # Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. # # R-ENV-02 ("authored once, promoted dev -> prod") is implemented by ONE mechanism: # `postBuild.substituteFrom cluster-vars` injecting `${TOKEN}`s into the manifests each # Flux Kustomization renders. That mechanism FAILS OPEN — an undefined variable becomes # the empty string while the Kustomization still reports Ready — which emptied Cilium's # k8sServiceHost and broke the CNI on 2/3 dev nodes (DECISIONS-LOG 2026-09-01). # # It is also LOSSY IN THE OTHER DIRECTION, and that is the second incident on this record. # envsubst consumes dollars in PAIRS and emits one per pair, unconditionally and everywhere — # so no substituted path can carry a literal `$$` at all. On 2026-09-13 a ConfigMap under # apps-wave-0 shipped a dollar-quoted PL/pgSQL `DO $$ ... $$;` self-check; Postgres received # a half-quoted `DO $` and every Flux-reconciled run of the nextcloud-db grant reconciler died # with a syntax error, nine Error pods deep before anyone looked (commit 16b7fcd). It passed # review because it was hand-applied with `kubectl apply`, which runs no envsubst: the defect # existed ONLY on the rendered path, and it landed on the assertion whose whole job was to # make a silent regression visible. Nothing here failed — the manifest rendered # "successfully" into something broken, because the check treated any doubled dollar as a # deliberate escape and stopped looking. # # GIVEN every Flux Kustomization under clusters/ that declares postBuild.substituteFrom, # WHEN each `spec.path` is resolved to the exact file set it renders (following # kustomization.yaml `resources:` transitively, not globbing the repo), # THEN every `${...}` in those files is either DEFINED in clusters/base/cluster-vars.yaml # or deliberately ESCAPED `$${...}`; no unescaped expansion carries a default # operator on an undefined name (the silent-empty class strict mode does NOT catch); # no unescaped expansion is unparseable by envsubst (which fails the WHOLE # Kustomization); NO RUN OF TWO OR MORE DOLLARS IS SILENTLY REWRITTEN — the only shape # that passes unremarked is exactly `$${`, the documented total escape, and every other # run (a bare `$$` that cannot survive verbatim, an ODD `$$${...}` that only LOOKS # escaped and in fact expands, an over-counted `$$$${...}`) is named and justified in # `dollar-collapse.allow`, which is itself held to being exact and non-stale; # every path rendering a knob actually substitutes cluster-vars; # every overlay-patched knob has a base default; and the check is wired into the # merge gate as a blocking, unconditionally-run stage. # # This is the prerequisite proof for enabling kustomize-controller's # StrictPostBuildSubstitutions feature gate, and the permanent guard that keeps it safe. # # STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library # only); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw # Test wraps the SAME script. # # NOT proven, and deliberately NOT proxied: the wider lesson of 2026-09-13 is that the bug # survived review because verification used a path (`kubectl apply`) that runs no # substitution. That property is NOT statically decidable from this repo — which command # someone ran is a fact about a shell session, and git stores the manifest, never the # invocation. Grepping commit prose or runbooks for `kubectl apply` would be trivially # satisfiable by rewording and would fire on the break-glass runbooks whose whole content is # that command, so no such proxy is attempted. The dollar-run assertion is the honest answer # instead: it moves the proof off the verification PATH and onto the SOURCE, where it IS # decidable, so the mangling is caught by reading the manifest however a human chose to try # it. An end-to-end `kustomize build | flux envsubst --strict` per substituted path would be # the real thing, and it is how the arithmetic in check.sh's header was measured — but the L0 # CI toolchain installs neither tool, so as a gate stage it would SKIP permanently. # # NOT proven (named in check.sh's header): this is a STATIC scan of the SOURCE manifests # a path includes — a deliberate superset of the post-`kustomize build` stream. It does # not render, does not prove the feature gate is enabled on the live kustomize-controller, # and does not validate knob VALUES. R-GITOPS-01 is tagged for the narrow static slice # only: what git says a Kustomization renders is fully determined, with no variable # silently emptied on the way in. The live-state half is L2, untested today. # ============================================================================= apiVersion: chainsaw.kyverno.io/v1alpha1 kind: Test metadata: name: flux-postbuild-substitution labels: conformance.home.k3s/R-ENV-02: "true" conformance.home.k3s/R-GITOPS-01: "true" annotations: conformance.home.k3s/requirements: "R-ENV-02, R-GITOPS-01" conformance.home.k3s/layer: "L0" # WHOLE-REPO SCOPE — relocated verbatim from ci/l0.sh's hand-written "Stage 3c # (UNCONDITIONAL)", which this annotation replaces: # # R-ENV-02's promote-once mechanism is `postBuild.substituteFrom cluster-vars`, and it # FAILS OPEN: an undefined ${TOKEN} becomes the empty string and the Kustomization still # reports Ready. That emptied Cilium's k8sServiceHost and broke the CNI on 2/3 dev nodes # (DECISIONS-LOG 2026-09-01, "substitution race that fails OPEN"). # # WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh's coarse path map # keys only clusters/* to R-ENV-02 — an edit to a SUBSTITUTED manifest under apps/ or # platform/ attributes no requirement, so the requirement-scoped selection would filter # this check out of exactly the change that introduces the bug. The invariant spans every # path any Flux Kustomization substitutes (apps/, platform/, platform-config/), so it runs # on EVERY gate invocation. conformance.home.k3s/scope: "whole-repo" conformance.home.k3s/safety-critical: "false" conformance.home.k3s/executable: "true" conformance.home.k3s/intent: >- Every variable a substituted Flux path renders is defined in cluster-vars or escaped, so post-build substitution cannot silently empty a value and enabling StrictPostBuildSubstitutions cannot fail a legitimate manifest; and no dollar run envsubst rewrites reaches the cluster unreviewed, so content that must arrive verbatim cannot be silently mangled on the Flux-rendered path (16b7fcd). spec: steps: - name: audit-every-substituted-flux-path description: >- Resolve each postBuild.substituteFrom Kustomization to the files it renders and assert every ${...} is defined or escaped and no dollar run is silently collapsed, then assert the gate wires it blocking. try: - script: content: bash ./check.sh