2e259a808fd995fa6196de2da07fd4a1ad5ac0a7
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
da8a0eac04 |
test(l0): R-STATE-04 gets a real check — bulk storage provably is not the replicated tier
The gate BLOCKED this branch, correctly, and the block was the interesting part.
Attaching the media library selects R-STATE-04 as a touched requirement ("bulk media storage has
a capacity strategy distinct from the replicated tier"), and R-STATE-04 had NO scaffolded test, so
the coverage REQUIRE stage failed both l0 and l1. There were two ways out. One was to keep
avoiding the token — the existing discipline in catalog/plex.yaml and catalog/qbittorrent.yaml,
which say in as many words that naming the requirement would block the merge. That made sense while
the tier was a stub. It does not make sense in the change that DELIVERS the tier: it would mean
shipping the requirement and claiming nothing. So: the check.
WHAT IT ASSERTS, and why each one is a real failure mode rather than a shape
The `bulk` StorageClass sat as `server: REPLACE-ME` for six roadmap phases, and the only thing
keeping a multi-terabyte library off the 3-replica `longhorn` class was four hand-written comments.
That is the class of failure being closed: a bulk volume on the replicated tier does not fail
loudly — Longhorn accepts the PVC and starts replicating 3x of data three mini-PCs cannot hold.
(1) DISTINCTNESS — no bulk PV/PVC on any class the repo declares with the Longhorn provisioner
(derived from platform/longhorn/, so adding a Longhorn class cannot create a blind spot),
and the class is explicitly EMPTY.
(2) CAPACITY STRATEGY as arithmetic — a bulk volume at least an order of magnitude above the
same service's replicated volume. The requirement is about two magnitudes of data; a "bulk"
volume sized like a config PVC means the tier was named, not applied. The 10x floor is
CALIBRATED against the real corpus, not guessed: the media library is ~460x its config
volume and the Nextcloud archive ~13x its 20Gi, so a higher floor would fire on a legitimate
volume and teach the next reader to weaken the check. My first draft used 100x and did
exactly that — the corpus caught it.
(3) PRE-BOUND FROM BOTH ENDS (claimRef + volumeName, agreeing). Guards the most deceptive
failure available in this tier: a dynamic bulk class hands out a FRESH EMPTY directory, the
pod mounts, the PVC Binds, everything is green, and the 8 TiB was never attached.
(4) Retain, always. (5) mountOptions carry `hard` and exactly one nfsvers=4.x, and never `soft`
— a soft mount hands a server hiccup to the app as a short read, which for a media library
is silent corruption; and v4 is what keeps the firewall surface at one port.
(6) ONE EXPORT, AT MOST ONE MOUNT PER WORKLOAD. This is the assertion I most wanted. Splitting
one export into two subPath mounts looks TIGHTER and breaks link() with EXDEV; Sonarr and
Radarr then degrade to copy-then-delete, rewriting every import over NFS and destroying the
seeding copy, surfacing weeks later as "imports are slow and my ratio is gone". No reviewer
sees that in a diff.
(7) READ-ONLY COUPLING — nfs.readOnly, the volumeMount flag and ReadOnlyMany must agree, so
dropping one still leaves the others; and every bulk mode is a `...Many`.
(8) The server is a BARE cluster-vars token, that knob has a base default, and EVERY overlay
supplies it — the seam that lets one authored definition serve two clusters that reach this
host by different addresses.
(9) LOAD-BEARING and NON-VACUOUS. Wired unconditionally into ci/l0.sh as Stage 3e for the same
reason as 3c/3d: Stage 3b scopes to touched requirements, and the path map keys NEITHER
apps/* NOR platform/longhorn/* to this requirement — so the change most likely to break
this could attribute nothing and be gated by nothing. An EMPTY bulk corpus FAILS rather
than passing trivially, and a floor on the assertion count keeps a pass from degenerating.
It reads the COMMITTED apps/ goldens rather than calling the renderer. Asking the renderer would
only prove it agrees with itself; the goldens are what Flux reconciles, and render==golden is
already its own gate stage.
PERTURBED, because a check that has never failed is not evidence. Each of these was applied to the
real tree and each fired with the intended message: bulk PVC moved onto `longhorn`; Retain -> Delete;
`hard` -> `soft`; the token replaced by the literal 192.168.1.9; sonarr's single mount split into
two subPath mounts (assertion 6); and the prod overlay's knob deleted.
Also hardens render.py: `size` is structurally required of a bulk entry (a PV with no capacity is
rejected by the API server) but the JSON Schema only enforces `export`, so omitting it raised a bare
KeyError three frames down. It now raises a sentence naming the service, the volume and the remedy.
Found by the same perturbation pass.
Coverage: 48/51 -> 49/51 (96.1%), 2 gaps, 0 safety-critical.
BASE_REF=main make -C ci gate -> MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85
|
89cf59ffb7 | docs(conformance): the GIVEN/WHEN/THEN header states the exception it now allows | ||
|
jakman85
|
730d12ddc4 |
fix(conformance): R-ENV-03 shared-path rule tests the harm, not the filename
The blanket regex around RENDERED_DIRS failed any tracked file whose name
carries a delimited `prod` token. That caught the four per-environment backup
credential halves (BACKUP_CREDENTIALS_SUFFIX) even though BOTH halves are
applied to BOTH clusters as base resources and the overlay selects only which
one the shared consumer REFERENCES — i.e. exactly the R-ENV-02 contract the
regex exists to protect. Renaming the files to dodge the regex was rejected as
gaming the guard, so the rule now encodes the property instead.
The fix is not an allowlist of those four paths. A prod-token file in a shared
path is excused only when the tree proves all of:
(A) it declares nothing but Secrets whose names end in the environment suffix
(1) its own directory's kustomization applies it, and EVERY environment value
of the selector resolves to a Secret that some resource of that same
kustomization declares, in the same namespace
(2) nothing names the suffixed Secret — or the suffixed file — literally;
every reference is `<base-name>${VAR}`
(3) that VAR is defined for the base and for every overlay, and prod's value
is the one that selects this file
References are read from parsed YAML values, never raw text: the mechanism is
spelled out by name in comments beside the resources, and a comment references
nothing.
No other assertion is weakened; the check still runs 96 assertions, and the
pass output now notes each excused file and why.
|
||
|
jakman85andClaude Opus 5
|
dc2d1ce17e |
fix(conformance): the R-NET-04 scan descended into nested agent worktrees
TRACKED_YAML rglob'd from the repo root excluding only .git and _artifacts, so it also walked .claude/worktrees/agent-*/ — the isolated worktrees created for agent runs. The check counted 5 CiliumEgressGatewayPolicy objects where exactly 1 is authored, failing R-NET-04 for a reason unrelated to egress steering. Now skips dot-DIRECTORIES (covering .git, .claude, .cache) while still matching dot-FILES such as .sops.yaml. .claude/ is gitignored, so nothing beneath it is authored surface. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
6a8953e288 |
feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing
Minimal prod delivery, authored directly on main (= prod's live revision
|
||
|
jakman85andClaude Opus 5
|
8a2be824f6 |
feat(ci): substitution-safety gate + escape 6 comment hazards; strict mode is now SAFE
Adds conformance/tests/l0-static/flux-postbuild-substitution/ — an offline, stdlib-only
L0 check tagged R-ENV-02 + R-GITOPS-01 (deliberately NOT R-TEST-01, which would
overclaim). It reuses lib/status.py's `resources:` walk rather than forking it, so "what a
path renders" has exactly one definition. It asserts: every ${...} on a substituted path
is defined-or-escaped (comments included); no default-operator form on an undefined name;
no unparseable expansion; the rendering path actually substitutes cluster-vars; every
overlay-patched /data/KEY has a base default; and that the gate itself is load-bearing.
WIRED AS AN UNCONDITIONAL STAGE (3c) in ci/l0.sh, and that matters: stage 3b scopes to
touched requirements, and ci/touched-requirements.sh maps only clusters/* to R-ENV-02 —
so an edit to a substituted manifest under apps/ or platform/ would have filtered this
check out of exactly the change that introduces the bug. l0 fail -> BLOCK=1 in gate.sh.
Mutation-tested all five assertion classes (undefined token, indexed expansion, :- form,
missing substituteFrom, un-wired gate); each fails as intended, baseline green.
STRICT MODE IS PROVEN SAFE TO ENABLE — measured, not inspected. For all 11 substituted
paths, `kubectl kustomize --load-restrictor LoadRestrictionsNone <path> | flux envsubst
--strict` exits 0 against cluster-vars base defaults. Negative control confirms the
incident: with K3S_API_HOST unset, non-strict renders k8sServiceHost empty while strict
exits 1 with `variable not set (strict mode)`. The feature gate itself is NOT flipped
here — that is a live-cluster change and the operator's call.
Escapes the six comment hazards that stood in the way (apps/vms/unifi x4,
platform/longhorn x2). All were prose ABOUT the escaping convention; since the repo's own
contract is that $${...} emits a literal ${...}, escaping them is both precise and
self-consistent. kustomize strips these particular comments today, but comment survival is
an implementation detail — a body comment carrying $ WAS observed surviving into built
apps/wave-1 — and a survivor would kill the whole Kustomization.
DOCTRINE CORRECTED (measured against flux 2.4.0): clusters/base/apps-vms.yaml claimed an
indexed expansion is fatal "escaped or not". Escaping is TOTAL — $${ARR[0]} emits the
literal text safely; only the UNESCAPED indexed form is fatal. Corrected in place.
Also empties DOCUMENTED_GAPS in conformance/run.sh with rationale: R-ENV-02 and
R-GITOPS-01 now have a real verifying test, which run.sh's own comment named as the
trigger for removing them.
Advisory output (not failures): 6 dead cluster-vars knobs — BASE_DOMAIN, CLUSTER_ENV,
CLUSTER_POD_CIDR, PIA_REGION, PLATFORM_NODE_SELECTOR_{KEY,VALUE}. The last four are
documented reference-only; BASE_DOMAIN and CLUSTER_ENV look like genuine rot (hostnames
are hardcoded elsewhere) and are worth a follow-up.
Gate: MERGE GATE: PASS. 102 assertions, 0 hazards.
Serves: R-ENV-02, R-GITOPS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
6e77150ae1 |
fix(ci): kill the false-green base resolution; cover R-SEC-01/02 + R-TEST-06 (L0)
Unblocks the enforced Gitea gate, which correctly BLOCKED this branch on ['R-SEC-01','R-SEC-02','R-TEST-06'] being touched but uncovered. Tests were authored; the coverage rule was NOT narrowed. === THE FALSE GREEN (root cause) === `make -C ci gate` locally reported PASS having checked NOTHING. resolve_base() validated nothing, and EVERY way of failing to find a base degraded silently to an empty touched set + exit 0: an `HEAD~1` last resort, a two-dot diff fallback masking a failing three-dot diff, and no check that HEAD is even AHEAD of the base. Run while sitting on `main` (main == origin/main), `origin/main...HEAD` is empty -> "no changed files vs base" -> the empty set means run.sh is invoked with NO --requirements -> coverage.py's --require gate never runs at all. Green by vacuum. Now: candidates are BASE_REF > GITHUB_BASE_REF > origin/main > main, each verified to resolve. HEAD~1 and the two-dot fallback are gone. Unresolvable base / no merge base / failed diff / HEAD-not-ahead -> loud banner, EMPTY stdout, exit 3, and a machine-readable reason (on-base | no-base | diff-failed). gate.sh turns exit 3 into an UNDECIDED decision: `on-base` -> "NOT APPLICABLE — no branch under gate" (exit 0 so push-to-main runs don't break, but it never reads as PASS); `no-base`/`diff-failed` -> BLOCKED. It also now prints the diff base it used. A stale local main warns (widens, never falsely greens). Also fixed `make -C ci test`, which was SWALLOWING failing suites. Regression: ci/tests/touched-requirements-base.test.sh — 31 assertions over 8 throwaway-repo scenarios including the exact false-green case, an already-merged branch, no base ref, a single-commit repo, bogus BASE_REF, a shallow GITHUB_BASE_REF, and origin/main-preferred-over-stale-main. === NEW L0 STATIC LAYER (conformance/tests/l0-static/) === Where docs/14 actually puts these invariants. Each check.sh really executes offline via `run.sh --layer l0` and is wired into ci/l0.sh, so it runs on every PR. No scaffolds. R-SEC-01 no-plaintext-secrets (165 assertions / 445 tracked files): .sops.yaml holds only PUBLIC age recipients; every committed Secret is SOPS-wrapped with a mac and ENC[...] values; ansible vault.yml files are $ANSIBLE_VAULT; no secret-valued ansible key carries a literal outside a vault; no PEM key / age secret key / kubeconfig client-key-data is tracked; Flux declares sops decryption; the disallow-plaintext-secrets policy Enforces and has a proving `result: fail` row. Exemptions are SELF-POLICING, not path allowlists — a .example must be referenced by no kustomization and hold only placeholders, and the Kyverno fixture is exempt ONLY while a test asserts the policy rejects it. Proven against 7 injected violations, including deleting the negative row and weakening Enforce->Audit. R-SEC-02 least-privilege-rbac (41 assertions): no wildcard apiGroup/resource/verb and no escalate|bind|impersonate in any in-repo Role/ClusterRole; no ServiceAccount bound to cluster-admin or a wildcard role; the conformance runner's own test identity is scoped and holds no rights over RBAC objects; the default-deny/host-namespace/egress-class policies Enforce and each carry a negative row. Proven against 6 injected violations. R-TEST-06 spec-immutability-enforced (16 assertions): drives the REAL ci/spec-immutability.sh against HERMETIC throwaway spec repos and asserts its exit code in every state — clean-at-pin 0, edits under docs/ or decisions/ 1, untracked addition 1, unadopted drift 1, explicit pin bump 0, ABSENT spec repo 3 (honest skip, never green), unpinned SPEC_PIN 1 — plus that gate.sh sets BLOCK=1 on its failure, so the mechanism is load-bearing. The real spec repo is never written to. Each check documents what it does NOT cover (runtime decryption, git history, runtime least-privilege, branch protection) rather than overclaiming. status.py now reads check.sh too, so a placeholder L0 body still outranks an `executable: true` annotation. Coverage 37/51 -> 40/51 (11 gaps); executable tests 32 -> 35; scaffold-only stays 5 — no sixth scaffold was added. Serves: R-SEC-01, R-SEC-02, R-TEST-01..06. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |