docs(pm): draft the R-ENV-03 amendment, and name the gate hole that hid it
The requirement's literal reading ("no prod-specific artifact in a shared path")
forbids the environment-selected PAIR that R-ENV-02 prescribes for a value that
must differ per environment while the manifests stay identical — and a Secret's
difference cannot live in cluster-vars, because it is ciphertext in a SOPS file.
Drafted wording that keeps the guard's teeth (an artifact only prod APPLIES
still fails) and removes the false positive, for the System Architect: the spec
repo is read-only here and CI-gated by R-TEST-06.
Also recorded, checked rather than assumed: the merge that turned R-ENV-03 red
was gated with a touched-requirement subset that never selected R-ENV-03, so
ci/l0.sh Stage 3b filtered the already-red check out of the run that printed
PASS. Replaying that delivery's twelve paths through
fix/touched-requirements-fail-closed's requirement-paths.map still yields no
R-ENV-03, so landing that branch would NOT have caught this. The two changes
that would (unconditional whole-repo L0 stages; a changed conformance test
attributing its own requirement labels) are ci/ work and are left to that
branch, not smuggled in here.
This commit is contained in:
1 parent
730d12ddc4
commit
e319777849
1 file changed
+135
@@ -1954,3 +1954,138 @@ not one.** Confirmed by `--list-hosts` (dev default → 3 dev hosts, where the n
|
|||||||
Nothing was run against either cluster for this change.
|
Nothing was run against either cluster for this change.
|
||||||
3. **Decide on `kubeEtcd` metrics**, so §2's exit condition can be measured with etcd's own fsync
|
3. **Decide on `kubeEtcd` metrics**, so §2's exit condition can be measured with etcd's own fsync
|
||||||
histograms rather than inferred from node PSI.
|
histograms rather than inferred from node PSI.
|
||||||
|
|
||||||
|
## 2026-09-07 — R-ENV-03's L0 check was RED on `main`, and the requirement's literal wording is why (branch `fix/r-env-03-overlay-selected-credentials`)
|
||||||
|
|
||||||
|
`main`'s `conformance/tests/l0-static/prod-baremetal-not-dev-surface` failed 4 of 96 assertions.
|
||||||
|
All four are the per-environment backup credentials merged in `e7c4a95`:
|
||||||
|
|
||||||
|
```
|
||||||
|
apps/wave-0/nextcloud-db-backup-credentials-prod.sops.yaml
|
||||||
|
apps/wave-0/semaphore-db-backup-credentials-prod.sops.yaml
|
||||||
|
platform/keycloak/keycloak-db-backup-credentials-prod.sops.yaml
|
||||||
|
platform/velero/velero-cloud-prod.sops.yaml
|
||||||
|
... is a prod-specific file inside the shared configuration
|
||||||
|
```
|
||||||
|
|
||||||
|
**The guard was a filename regex.** Any tracked file under `platform/ platform-config apps/
|
||||||
|
catalog/ clusters/base/` whose name contained a delimited `prod` token failed, with no allowlist
|
||||||
|
and no exception mechanism. Nothing about the four files' CONTENT was inspected.
|
||||||
|
|
||||||
|
**Renaming them to dodge the regex was rejected** (operator, explicitly): the guard exists for a
|
||||||
|
reason and a rename would leave the reason unserved while making the check lie. So the check was
|
||||||
|
refined to test the harm instead of the spelling, and the requirement wording is escalated below.
|
||||||
|
|
||||||
|
### Why these four files are compliant with R-ENV-02, and why the regex could not see it
|
||||||
|
|
||||||
|
Backup credentials became per-environment because dev backs up to MinIO and prod to Cloudflare
|
||||||
|
R2, and ONE shared Secret meant either environment's key rotation broke the other (worse for
|
||||||
|
CNPG: `archive_mode=on` turns a 403 on WAL archive into a monotonically filling pgdata PVC).
|
||||||
|
|
||||||
|
The mechanism is a single cluster-vars knob, `BACKUP_CREDENTIALS_SUFFIX` (base `""`, dev `""`,
|
||||||
|
prod `"-prod"`). **Both Secret objects are applied to BOTH clusters** as base resources; the
|
||||||
|
overlay changes only which one the shared consumer REFERENCES
|
||||||
|
(`<name>${BACKUP_CREDENTIALS_SUFFIX}`). Every environment renders the identical manifests and the
|
||||||
|
identical object set. That is exactly the "same manifests, overlay-scoped values" contract of
|
||||||
|
R-ENV-02 — nothing is forked, and there is no prod-only content for anyone to develop against.
|
||||||
|
The `-prod` in a filename is not a prod-only manifest; it is one half of an environment-selected
|
||||||
|
pair, present everywhere.
|
||||||
|
|
||||||
|
### What the check now asserts (committed on the branch; not a path allowlist)
|
||||||
|
|
||||||
|
A file whose name carries the environment token is excused ONLY when the tree proves all of:
|
||||||
|
|
||||||
|
- **(A)** it declares nothing but Secrets whose names end in the environment suffix — an overlay
|
||||||
|
can select which SECRET a shared manifest references and nothing else, so any other
|
||||||
|
prod-suffixed object in a shared path is prod-only content and still fails;
|
||||||
|
- **(1)** its own directory's `kustomization.yaml` lists it in `resources` (so EVERY environment
|
||||||
|
applies it), and EVERY environment's value of the selector resolves to a Secret that some
|
||||||
|
resource of that same kustomization declares, in the same namespace — no environment is left
|
||||||
|
selecting an object that does not exist;
|
||||||
|
- **(2)** nothing names the suffixed Secret — or the suffixed FILE — literally: every reference is
|
||||||
|
`<base-name>${VAR}`, and only the applying kustomization may name the file;
|
||||||
|
- **(3)** that `VAR` is defined for `clusters/base` AND for every `clusters/<env>` overlay
|
||||||
|
(StrictPostBuildSubstitutions fails on an UNDEFINED variable, so an overlay that forgot it
|
||||||
|
would stop that cluster reconciling), and prod's value is the one that selects THIS file.
|
||||||
|
|
||||||
|
References are read from PARSED YAML VALUES, never raw text — the mechanism is documented by name
|
||||||
|
in comments beside the resources, and a name in a comment references nothing.
|
||||||
|
|
||||||
|
Mutation-tested: eleven mutations, each restored afterwards, each of which must FAIL and does — a
|
||||||
|
prod-only ConfigMap in a shared path, a prod-only Secret with no counterpart, a consumer
|
||||||
|
hardcoding prod's Secret name, a kustomization naming prod's file from another directory, the
|
||||||
|
knob missing from dev, the knob missing from base, prod's value drifted to a suffix nothing
|
||||||
|
declares, the prod half dropped from `resources`, the base half deleted (with and without its
|
||||||
|
`resources` entry), the halves split across namespaces, and one of two consumer references moved
|
||||||
|
to a second knob. The legitimate four pass, with a `note:` per file saying why they were excused.
|
||||||
|
|
||||||
|
### PROPOSED R-ENV-03 AMENDMENT — for the System Architect (spec repo is READ-ONLY here)
|
||||||
|
|
||||||
|
**What the requirement says today** (`docs/00-vision-and-scope.md`, detail in `docs/02`): prod is
|
||||||
|
a 3x mini-PC bare-metal k3s cluster with local storage and **"MUST NOT be an active development
|
||||||
|
surface until the roadmap declares it ready"**; `docs/02` adds **"Never develop against prod.
|
||||||
|
Prod hardware is a target, not a sandbox (R-ENV-03)."**
|
||||||
|
|
||||||
|
**The problem with the literal reading.** Taken literally against the repository, "no prod-only
|
||||||
|
content in the shared configuration" reads as "no artifact in a shared path may be specific to
|
||||||
|
prod". That forbids the environment-selected PAIR, which is the very pattern R-ENV-02 prescribes
|
||||||
|
for a value that must differ per environment while the manifests stay identical. The two
|
||||||
|
requirements contradict each other on this case: R-ENV-02 says put the difference in an
|
||||||
|
overlay-scoped value, and a Secret's difference cannot live in cluster-vars (it is ciphertext, per
|
||||||
|
SOPS file, R-SEC-01), so it must live in a per-environment FILE that every environment applies.
|
||||||
|
The requirement should forbid what is actually harmful — configuration only prod runs — not the
|
||||||
|
appearance of an environment name in a path.
|
||||||
|
|
||||||
|
**Proposed wording** (add as a clarifying clause to R-ENV-03 in `docs/02`, and mirror the phrase
|
||||||
|
in `docs/00`):
|
||||||
|
|
||||||
|
> Prod MUST NOT be an active development surface. Concretely: no artifact in the shared
|
||||||
|
> configuration may be reachable by prod alone. An artifact whose NAME carries an environment
|
||||||
|
> token is not by itself a violation — provided every environment APPLIES it, no shared manifest
|
||||||
|
> references it by its environment-qualified name, and the selection between environment
|
||||||
|
> counterparts is made solely by an overlay-scoped cluster-vars value that is DEFINED for every
|
||||||
|
> environment. An artifact that only prod applies, or that a shared manifest names literally, IS
|
||||||
|
> a violation, whatever it is called.
|
||||||
|
|
||||||
|
That wording keeps the guard's teeth (an artifact only prod applies still fails), removes the
|
||||||
|
false positive, and states the property the check now enforces so the test and the spec agree.
|
||||||
|
|
||||||
|
**Related spec ask, same clause.** R-ENV-02 should say out loud that a per-environment SECRET is
|
||||||
|
expressed as counterpart SOPS files applied everywhere plus a cluster-vars selector, because that
|
||||||
|
is the one value class that cannot be expressed as a cluster-vars value itself. Without it, the
|
||||||
|
next person hits the same contradiction.
|
||||||
|
|
||||||
|
### How this reached `main` green — the gate hole, and whether the fix in flight closes it
|
||||||
|
|
||||||
|
`ci/l0.sh` Stage 3b scopes the L0 conformance run to the TOUCHED requirement subset
|
||||||
|
(`load_touched_requirements` -> `run.sh --requirements ...`). The delivery `e7c4a95` touched
|
||||||
|
`apps/wave-0/`, `catalog/render.py`, `platform/keycloak/`, `platform/velero/` and declared no
|
||||||
|
R-ID on a non-comment line, so the path map attributed R-GITOPS-02 / R-IDP-01..06 / R-STATE-03 —
|
||||||
|
**and never R-ENV-03**. `prod-baremetal-not-dev-surface` was therefore FILTERED OUT of the run
|
||||||
|
that gated the merge. The gate printed PASS having never executed the check that was already red.
|
||||||
|
|
||||||
|
**Landing `fix/touched-requirements-fail-closed` would NOT have caught this.** Checked, not
|
||||||
|
assumed: that branch's `ci/requirement-paths.map` classifies every path (so nothing is silently
|
||||||
|
unmapped any more), but it attributes R-ENV-03 only to `infra/ansible/*` and `infra/tofu/*`.
|
||||||
|
Replaying `e7c4a95`'s twelve paths through that map yields
|
||||||
|
`R-GITOPS-01 R-GITOPS-02 R-IDP-01..06 R-PLAT-02 R-STATE-03` — still no R-ENV-03. Its `ci/l0.sh`
|
||||||
|
change also leaves Stage 3b scoped (only the substitution stage stays unconditional). The
|
||||||
|
fail-closed branch fixes the "unmapped path contributes nothing" hole; it does not fix this one.
|
||||||
|
|
||||||
|
**Two things that would have caught it, for the operator/architect to choose between:**
|
||||||
|
|
||||||
|
1. **Whole-repo L0 invariants should be unconditional.** `flux-postbuild-substitution`,
|
||||||
|
`keycloak-client-secret-reconciler` and `spec-immutability` already run as their own
|
||||||
|
unconditional stages for exactly this reason. `prod-baremetal-not-dev-surface`,
|
||||||
|
`no-plaintext-secrets` and `least-privilege-rbac` are the same shape — they assert over the
|
||||||
|
WHOLE tracked tree, so scoping them to a changed path is meaningless. Cheapest correct fix.
|
||||||
|
2. **A changed test should attribute the requirements it declares.** `conformance/*` maps to
|
||||||
|
R-TEST-01..04 only, so editing `prod-baremetal-not-dev-surface/check.sh` does not select
|
||||||
|
R-ENV-03 — this very branch's gate run reports `touched requirements: R-ENV-02 R-TEST-01
|
||||||
|
R-TEST-02 R-TEST-03 R-TEST-04` and ran 2 of 12 L0 checks, NOT including the one it changed.
|
||||||
|
A changed `conformance/tests/<test>/` should contribute that test's own
|
||||||
|
`conformance.home.k3s/requirements` label.
|
||||||
|
|
||||||
|
Both belong to `ci/`, so they are left to that branch rather than smuggled in here. The full L0
|
||||||
|
layer was run unscoped on this branch to compensate: **12 offline checks, 11 PASS, 1 SKIP**
|
||||||
|
(`placeholder-guard-coupling`, pre-existing), 0 FAIL.
|
||||||
Reference in new issue
Block a user