Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity

fix(conformance): placeholder-guard-coupling actually RUNS — it was skipping forever

Browse Source 
`conformance/tests/l0-static/placeholder-guard-coupling` (R-ENV-02, R-STATE-03) exited 3
SKIPPED on every gate run since it landed, because its check keyed its whole existence on
`bootstrap/standup.sh`. Measured, not assumed: that file has NEVER been on `main` — there is
no deletion commit for it anywhere; it landed only on unmerged prod-standup branches
(`work/prod-standup-campaign`, `prod/*`, `feat/prod-hardening`). It was planned and never
merged, not removed. SKIPPED is not green, and a permanently-skipped incident-derived test is
the quiet version of the failure it was written to prevent.

The skip was also hiding a RED. Run against the real `standup.sh` from
`work/prod-standup-campaign`, the old check FAILED 2 of 34 assertions — false positives from
treating the empty string as placeholder-shaped for knobs cluster-vars authors empty on
purpose (`BACKUP_CREDENTIALS_SUFFIX`, `VELERO_BACKUP_PREFIX`, both `""` on dev and both
documented there as load-bearing). It would have false-blocked the merge landing its own input.

NOT deleted — the hazard is not gone. Split into two halves, one of which needs no guard:

  HALF A (always runs): no environment overlay ships a placeholder-shaped EFFECTIVE
  cluster-vars value; an empty effective value is STATED by the overlay rather than inherited
  in silence; a LOUD base sentinel is explicitly overridden by every overlay. This is the
  2026-09-05 outcome asserted at the source instead of at the guard — if prod's
  S3_BACKUP_ENDPOINT is never a placeholder, velero cannot be pointed at a hostname that does
  not exist. Input surface is `clusters/`, all of it on `main`.

  HALF B (self-activating): the real predicate extracted and executed against those same
  values, the consumer trace, and the mid-string anchoring property.

The check NEVER exits 3 again, and the guard is DISCOVERED by content rather than hardcoded:
any tracked `*.sh` defining `<name>_is_placeholder()` is found, so a guard landing at
`bootstrap/standup.sh` — or moved into a sourceable library, the remedy the check already
recommends — activates HALF B with no edit to the test. Two files defining it is a FAIL. When
no guard exists the gap is printed loudly every run instead of swallowed as a skip.

LOUD vs EMPTY is the correction that makes it honest in both modes: a loud placeholder is
always a defect in an overlay, while `""` is held to the explicitness rule instead — and still
counts as a placeholder for a knob a guard covers, because a `""` case-arm is the guard
author's declaration that empty is not a usable target.

Proof, all reverted: 84 assertions and exit 0 on `main`; prod S3_BACKUP_ENDPOINT set to the
literal `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` FAILS (A1); deleting dev's
explicit VELERO_BACKUP_PREFIX op so it inherits `""` FAILS (A2); a LOUD base BULK_NFS_SERVER
with prod's override deleted FAILS 2/86 (A1+A3); and with the real `standup.sh` present,
reverting `s3_is_placeholder` to the `TBD-*` PREFIX glob FAILS 12/110 (B3 anchoring). With the
pristine `standup.sh` the check PASSES at 110 assertions where the old one failed 2 of 34.

L0 layer: 14 checks, 14 PASS, 0 SKIP, 0 FAIL (was 13 PASS / 1 SKIP). Coverage unchanged —
R-ENV-02 and R-STATE-03 both still list this test, 49/51 with the same 2 gaps.

Also:
  * `ci/touched-requirements.sh` — `bootstrap/*` -> R-STATE-03 R-GITOPS-01 is KEPT, its stale
    comment corrected. It claimed standup.sh "carries the placeholder guards", describing a
    file that is not here; R-STATE-03 there is ANTICIPATORY. Deleting the entry would leave the
    change that lands a placeholder guard selected by nothing.
  * `ci/tests/touched-requirements.test.sh` — new path-map section (12 assertions, was 8)
    locking `bootstrap/*` and `clusters/*`, so removing either entry is a test failure rather
    than a silent loss of selection. All three ci/tests pass (12 + 31 + 15).
  * `docs/BULK-NFS-STORAGE.md` §7 — its "nothing in the gate currently refuses a
    placeholder-shaped overlay value" is no longer true; corrected.
  * Scope re-audited: still deliberately requirement-scoped, not whole-repo. Every path that
    can break it attributes a requirement it declares.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-13 00:10:37 -05:00
1 parent 55b90849d1
commit 87c3d83dff
6 files changed
+716 -299
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

DECISIONS-LOG.md
+128
View File
@@ -2869,3 +2869,131 @@ PVC and both change the operator's own login story, so **neither was done**. Fla
in these pods), and three ~1 GB files named `...S18E05....zipx` / `...S06E01....exe` /
`...S29E01....exe`. Media files with `.exe`/`.zipx` extensions are worth an operator's eye; nothing
here read or moved them.
## 2026-09-13 — `placeholder-guard-coupling` was SKIPPING FOREVER, and the skip was hiding a RED (branch `fix/placeholder-guard-coupling-unskipped`)
An L0 audit flagged `conformance/tests/l0-static/placeholder-guard-coupling` (R-ENV-02, R-STATE-03)
as permanently SKIPPED. Confirmed, and the specifics change what the correct fix is, so they were
measured rather than assumed.
### Did `bootstrap/standup.sh` ever exist? NO — not on `main`, and it was never deleted
- `git log --all --diff-filter=D -- bootstrap/standup.sh` → **empty**. There is no deletion commit
for it anywhere in the repo.
- `git ls-tree main bootstrap/` → `README.md`, `flux/`, `gitea/`, `sops/`. No `standup.sh`.
- The file is real and substantial (988 lines) but lives only on **unmerged prod-standup branches**:
`work/prod-standup-campaign`, `prod/*`, `feat/prod-hardening`. First added in `4160fee`, last
touched in `8f9b2e4` ("revives a guard I had killed").
- So the check was authored against a file that **was planned and never landed**, not one that was
removed. `docs/BULK-NFS-STORAGE.md` §7 had already recorded this, including the consequence:
*"nothing in the gate currently refuses a placeholder-shaped overlay value."*
- The 2026-09-05 incident itself (`32d6766`, the MinIO → Cloudflare R2 move) is likewise **not in
this log on `main`** — it is on the same unmerged branches. The only record on `main` was the
check's own header, `ci/touched-requirements.sh`'s comment, and `dr-restore`'s test header. It is
written down here now so it survives whether or not those branches ever land.
### The skip was hiding a RED, which is worse
Dropped the real `standup.sh` from `work/prod-standup-campaign` into the tree and ran the OLD check:
**FAIL, 2 of 34 assertions.** Both were false positives from treating the **empty string** as
placeholder-shaped for knobs cluster-vars authors empty *on purpose* — `BACKUP_CREDENTIALS_SUFFIX`
and `VELERO_BACKUP_PREFIX`, both `""` on dev, and both documented in `cluster-vars.yaml` as
load-bearing ("THE BASE DEFAULT IS THE EMPTY STRING, AND THAT IS LOAD-BEARING … an EMPTY VALUE IS
SAFE UNDER StrictPostBuildSubstitutions: strict mode fails on a variable that is not DEFINED, not on
one defined as `""`"). So the check would have **false-blocked the merge that landed its own input**.
### NOT deleted — the hazard is not gone. Split into a half that needs no guard, and a half that self-activates
The invariant has two independent halves and only one needs the guard:
- **HALF A — the authored values.** No environment overlay may ship a placeholder-shaped *effective*
cluster-vars value. This is the 2026-09-05 **outcome** asserted at the source instead of at the
guard: if prod's `S3_BACKUP_ENDPOINT` is never a placeholder, velero cannot be pointed at a
hostname that does not exist, guard or no guard. Input surface is `clusters/`, all on `main`.
Runs unconditionally, 70 authored overlay values screened today.
- **HALF B — the coupling.** The real predicate extracted and executed against those same values,
the consumer trace, and the mid-string anchoring property. Needs a guard.
**The check never exits 3 again.** And the guard is now **discovered, not hardcoded**: any tracked
`*.sh` defining `<name>_is_placeholder()` is found, so a guard landing at `bootstrap/standup.sh` —
or moved to a sourceable library, which is the remedy the check already recommends — activates HALF B
**with no edit to the test**. Two files defining it is a FAIL (that is the 2026-09-05 divergence with
a second file's name on it). When no guard exists, the gap is printed loudly on stdout every run
instead of being swallowed as a skip.
**LOUD vs EMPTY** is the correction that makes it honest in both modes. A loud placeholder (`TBD`,
`TODO`, `REPLACE-ME`, `<angle-brackets>`, …) is always a defect in an overlay. The empty string is
not, so it is held to a *different real rule*: an empty effective value must be **stated explicitly**
by that overlay, never inherited in silence — which is cluster-vars' own documented convention
("Both overlays state this knob explicitly … so neither inherits it silently"). Emptiness still
counts as a placeholder for a knob a guard covers, because a `""` case-arm is the guard author's
declaration that empty is not a usable target.
### Proof it runs, and proof it is not vacuous
`check.sh` exits 0 at **84 assertions** on `main` (`(A1)` 70 authored overlay values, `(A2)` 2 empty
values checked for explicit statement, `(A3)` 0 live base sentinels, detector canaries, load-bearing).
Four deliberate breakages, each reverted:
| break | result |
| --- | --- |
| prod `S3_BACKUP_ENDPOINT` → `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` (the literal 2026-09-05 value) | **FAIL** 1/84 — (A1) |
| delete dev's explicit `VELERO_BACKUP_PREFIX` op so it inherits `""` | **FAIL** 1/84 — (A2) |
| base `BULK_NFS_SERVER` → `TBD-PROD-BULK-NFS-SERVER` + delete prod's override | **FAIL** 2/86 — (A1)+(A3) |
| drop in the real `standup.sh`, then revert `s3_is_placeholder` to the `TBD-*` **prefix** glob | **FAIL** 12/110 — (B3) anchoring |
With the pristine `standup.sh` present, discovery finds it and the check **PASSES at 110
assertions** — where the old one failed 2 of 34. A second file defining the predicate is detected and
named. Coverage is unchanged: R-ENV-02 and R-STATE-03 both still list this test (49/51, 2 gaps,
identical before and after), because nothing was deleted.
**L0 layer: 14 checks, 14 PASS, 0 SKIP, 0 FAIL** (was 13 PASS / 1 SKIP).
### The other L0 skip-capable checks — audited, and none has this defect
Every L0 check whose `check.sh` can `exit 3` on a missing input path, with the path verified against
the tree:
| check | skip condition | on `main` |
| --- | --- | --- |
| `placeholder-guard-coupling` | *was* `bootstrap/standup.sh` | **was the only real one — fixed; it now FAILS rather than skips if `clusters/base/cluster-vars.yaml` goes missing, because that means the knobs moved and the check must be re-pointed in the same change** |
| `bulk-tier-distinct-from-replicated` | `catalog/` or `apps/` absent | both present — never fires |
| `flux-postbuild-substitution` | `clusters/` or `clusters/base/cluster-vars.yaml` absent | present — never fires |
| `keycloak-client-secret-reconciler` | any of 4 `platform/keycloak/*.yaml` absent | all 4 present — never fires |
| `spec-immutability-enforced` | the sibling spec repo unavailable | **legitimate and left alone** — an environmental dependency (`../home-network-k3s`, R-TEST-06), not a repo-state defect; present here, so it PASSES |
| `whole-repo-invariants-wired` | — | its `exit 3`s are *synthetic fixtures* it creates to prove the runner treats a skipped whole-repo test as UNWIRED, not its own skip |
So the disease was confined to one test. The four defensive guards above are dead code in practice
and were left alone rather than expanding scope.
### `ci/touched-requirements.sh` — the entry STAYS, the comment was wrong
`bootstrap/*` → `R-STATE-03 R-GITOPS-01` is **kept**. Its comment claimed `bootstrap/standup.sh`
"carries the placeholder guards", describing a file that is not on `main`; corrected to say
R-STATE-03 there is **anticipatory**. Deleting the entry would mean the change that eventually lands
a placeholder guard is selected by nothing — the 2026-09-05 class left to reintroduce itself — and
with content-based guard discovery that entry is exactly what pulls this test into that change's gate
run. `ci/tests/touched-requirements.test.sh` gained a **path-map section** (12 assertions, up from 8)
locking `bootstrap/*` → R-STATE-03 + R-GITOPS-01 and `clusters/*` → R-ENV-02 + R-GITOPS-01, so the
entry's removal is now a test failure rather than a silent loss of selection. Also corrected: the
check's own header claimed "an edit to `bootstrap/*` attributes NO requirement in the path map",
stale since 2026-09-06.
### Scope annotation: re-audited and still deliberately requirement-scoped, NOT whole-repo
HALF A reads `clusters/` (→ R-ENV-02); HALF B reads wherever the guard lands, i.e. `bootstrap/`
(→ R-STATE-03). Every path that can break the check attributes a requirement it declares, so
requirement-scoped selection cannot hide a breaking change and `whole-repo` would add runtime, not
coverage. One residual, recorded rather than papered over: `(A5)` reads `ci/l0.sh` and `ci/gate.sh`,
and `ci/*` attributes neither of this test's requirements — but that wiring is separately and
unconditionally asserted by the whole-repo-scoped `whole-repo-invariants-wired`, so `(A5)` here is
belt-and-braces rather than sole proof.
### Debt / follow-ups
- **HALF B is genuinely unproven on `main`,** and that is now *reported on stdout every run* rather
than hidden. It cannot be fixed from `main`: it needs `bootstrap/standup.sh` (or its predicate,
extracted into a sourceable library) to land. When it does, no edit to this test is required.
- **`prom-rules` still reports "skipped (missing tool) — not green"** in the gate; pre-existing,
`promtool` is being baked into the toolchain on another branch.
- **A placeholder hardcoded in a manifest** rather than in a cluster-vars knob is out of HALF A's
scope: it reads the ConfigMap and the overlay patches, not every YAML in the tree.