Commit Graph
6 Commits
Author SHA1 Message Date
jakman85andClaude Opus 5 6f61c9622d feat(ha): the CNI dials the control-plane VIP — kube-vip becomes a CNI prerequisite
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
OPERATOR DECISION 2026-09-06: "use the VIP as intended — a virtual address to reach
the cluster and provide fault tolerant HA. Without a change it will not work in HA."

THE DEFECT. Cilium's kube-proxy replacement dialled a SINGLE NODE. Re-confirmed live
before changing anything: all three prod cilium-agent pods carried
KUBERNETES_SERVICE_HOST=192.168.2.10 — k3s-prod-1's own address — while the kube-vip
VIP 192.168.2.9 was Running on all three nodes with a held lease. Lose prod-1 and the
CNI's control-plane path is dead on an "HA" cluster. clusters/prod's own comment said
the value must be the VIP and NOT .10, then called it "Not yet decided" — I treated
that stale comment as an open question when the evidence already settled it.

SPLIT, NOT REPOINT, because k3s_api_host had two consumers with opposite needs:
  k3s_api_host            unchanged — the FOUNDER's IP: the cold-bootstrap join
                          target and a tls-san. infra/tofu/outputs.tf GENERATES this
                          name, so renaming would ripple into tofu and the runbooks.
  k3s_apiserver_endpoint  NEW = {{ kube_vip_address }} — the HA address; what Cilium
                          consumes. Derived, not restated, so an environment overrides
                          ONE line and the VIP, the SAN and the CNI endpoint move
                          together. Prod needed no new override.

THE SECOND HAZARD, WHICH MY BRIEF DID NOT NAME and the agent found: pointing Cilium at
the VIP breaks a COLD bootstrap for the same reason the join would — site.yml ran
node-baseline -> k3s -> cilium, so Cilium would install against an address nothing
announced yet. That is the 2026-09-01 deadlock shape. Fixed by SEQUENCE, not value:
site.yml now imports kube-vip.yml BETWEEN k3s and cilium. It cannot go earlier
(kube-vip is a k3s auto-deploy manifest, so it needs a cluster) and it works there
because the DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 — needing
neither CNI nor kube-proxy. roles/kube_vip gained a read-only `wait_for` on the VIP so
a cold run stops THERE with a named cause instead of surfacing as unreachable agents.

Also fixed a pre-existing bug that only mattered once this play joined site.yml: the
role's read-only interface probe ran without `check_mode: false`, so under --check it
was skipped and the assert fired with a misleading "set kube_vip_interface explicitly".

VERIFIED BY ME, not just reported:
  renders    dev K3S_API_HOST=192.168.1.49   prod=192.168.2.9  (each environment's VIP)
  cilium     template consumes k3s_apiserver_endpoint, not k3s_api_host
  cold join  still `--server https://{{ k3s_api_host }}:6443` — the founder
  order      site.yml: node-baseline -> k3s -> kube-vip -> cilium -> verify -> flux
  gate       full CI gate PASS
Agent-side, additionally: in-memory render of cilium-values.yaml.j2 per host showing
k8sServiceHost=the VIP with the join still on the founder; k3s_tls_sans renders
['.10','.9','127.0.0.1','.11','.12'] so NO cert reissue and NO k3s restart is involved
— .9 already answers under full CA verification; and --check --diff of kube-vip.yml
against prod reported 6 ok / 0 changed, i.e. the role edits change nothing about the
kube-vip prod already runs.

ha-control-plane-as-code 92 -> 112 assertions, turning two escalations into machine
checks: the Cilium template must use the endpoint var; the endpoint must derive from
kube_vip_address and k3s_api_host must not; per environment K3S_API_HOST must equal
that environment's VIP and be none of its node IPs (the [parity] contract, now
machine-checked rather than a comment); site.yml must order k3s -> kube-vip -> cilium.
Every new assertion mutation-tested and bites. One honest negative recorded: reverting
only the BASE default is not caught, because both overlays patch the key — the check
reads effective per-environment values, so the base default is unguarded.

APPLY ORDER, and this is the operator's step — nothing was applied:
  PROD FIRST, and it is the low-risk direction (prod is the broken one). Push and let
  Flux reconcile platform/cilium: the value change rolls the cilium-agent DaemonSet
  ONE NODE AT A TIME (maxUnavailable: 1 — the knob that bounded the 2026-09-01
  incident), it will not proceed past an unready pod, existing datapath survives an
  agent restart, and .9 is already live before the change lands.
  DEV SEPARATELY, operator present. Same mechanics but live dev agents dial
  192.168.1.50, and dev is mid-DNS-migration on dev-live. Do not fold it into that.
  ANSIBLE: nothing to run for either live cluster; the Ansible half matters only for a
  cold build. Remove the now-redundant standalone kube-vip.yml step from any runbook.

NEW NAMED GAP, unwritten: the L2 test cannot assert the running agents' live
KUBERNETES_SERVICE_HOST, which is the drift the L0 check cannot see. Recorded in that
test's gap list.

Refs R-PLAT-01, R-NET-01, R-ENV-02.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-06 20:24:51 -05:00
jakman85andClaude Opus 5 cb1916f092 feat(conformance): cover R-PLAT-01, R-NET-04, R-SEC-03 — 88.2% -> 94.1%, and fix an unrecordable-proof defect
Coverage 45/51 -> 48/51, gaps 6 -> 3 (R-C2-01, R-C2-02, R-STATE-04), 0
safety-critical. --strict-coverage exit 0; run.sh --layer l0 11/11 PASS.

R-PLAT-01 — split L0 + L2, because "3 nodes with production-grade HA control
plane" is half authored and half observable.
  l0-static/ha-control-plane-as-code (92 assertions): environments DERIVED from
  the R-ENV-02 seam (never a hardcoded group name); k3s installed as `server` with
  no --datastore-endpoint/--disable-etcd anywhere; exactly ONE --cluster-init
  founder per environment; odd, quorum-capable server count with tolerance >= 1;
  tls-sans carry the VIP and every server IP; kube-vip leader-elected with
  cp_enable/svc_enable parsed from the template's OWN env pairs; each VIP outside
  its environment's LB pool and not a node IP.
  l2-behavioral/ha-control-plane: control-plane AND etcd role on every node,
  every node a REGISTERED etcd member, exactly 3 when the cluster says
  CLUSTER_ENV=prod, and an in-cluster Job curling /readyz through the VIP AND every
  server's own address — which proves the VIP fronts the API and that every server
  runs an apiserver, not one founder plus two joiners.

  An earlier draft probed those endpoints FROM THE RUNNER: that passes from prod's
  LAN and fails against dev (nodes behind the pfSense NAT), making the test a
  property of where it was launched. Hence the in-cluster Job.

R-NET-04 — what it adds over R-NET-05/07 is the PATH SET and the
totality/disjointness of the selecting rules, not another per-service check.
  l0-static/egress-paths-selected-by-rules (68 assertions): exactly two paths, the
  same closed set in schema + Kyverno policy + renderer; EXACTLY ONE
  CiliumEgressGatewayPolicy with a single selector matching only egress-class=vpn;
  nothing selects `internet` (absence of a rule IS the path); RFC1918 containment
  computed with `ipaddress` plus every address each overlay assigns itself; the
  tunnel default route and blackhole confined to table 51820 behind an ip rule that
  must carry a from/fwmark/iif selector; and the nft chain `policy accept` with
  every drop qualified — the internet-fails-OPEN direction that R-NET-06's L3 test
  does not assert.

R-SEC-03 — refuses the tautology "a test exists, therefore it is auditable".
  l0-static/auditability-proofs-producible (78 assertions) drives the real
  machinery: each of docs/07's five proofs must have an EXECUTABLE test at the layer
  the spec's wording demands (fault injection and DR drill => L3), taken from
  status.py's own verdict; run.sh --requirements must resolve and pass the gate's
  --require assertion; each proof declares a sentinel record_run.py can attribute,
  computed with record_run.parse_sentinels itself; and TWO FORGERIES are mounted
  against the real tools — a green JUnit whose log holds only a SKIP token, and a
  ledger row claiming `proven` at the root commit — which must come back not-proven
  and stale-proof. The five proofs are parsed from docs/07 when the spec is mounted,
  not hand-listed.

*** A REAL DEFECT FOUND AND FIXED, not asserted around. *** l2-behavioral's
default-deny, trusted-tls and gateway-ingress-reachable declared NO pass sentinel,
so record_run.py could never attribute a pass to them — the isolation and TLS audit
proofs were STRUCTURALLY UNRECORDABLE (landing as unknown-no-sentinel-declared,
and status.py requires all of a requirement's tests to be proven). Each now ends
with an additive terminal sentinel step reached only when every prior step passed.
Verified: all three carry it.

TWO HOLLOW DRAFT ASSERTIONS CAUGHT BY THE AGENT ITSELF, both the failure mode this
repo keeps hitting: (a) `svc_enable=false` matched a COMMENT in the kube-vip role
defaults, so flipping the template to true still passed — now parses the DaemonSet
template's env pairs; (b) the nft scoping check searched for the VALUE of a Jinja
variable while the template carries its NAME, matching nothing and passing
vacuously — templates are now rendered against role defaults first. 30 perturbations
across the three checks, all caught, plus 16 synthetic L2 perturbations via a
fake-kubectl shim.

R-C2-01/02 judged IMPLEMENTATION gaps, so no test was written: Homepage's
widgets.yaml carries only datetime and search, kubernetes.yaml is empty,
automountServiceAccountToken is false, and its NetworkPolicy allows DNS egress
only — no health source is even reachable. It has already drifted (crafty authors a
route with no tile). A test today would go red on both; it must ship WITH the
config fix.

VERIFIED: full CI gate PASS. Neither cluster mutated.

Refs R-PLAT-01, R-NET-04, R-SEC-03, R-TEST-01..05.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-06 19:35:18 -05:00
jakman85andClaude Opus 5 a984442cae harden: pin node timezone to UTC, and TEST the guard/value coupling that bit us
Two debt items, both about making a latent problem impossible rather than fixed.

=== 1. Node timezone pinned to Etc/UTC (node_baseline, `system` tag) ===

The prod nodes disagreed, measured from k3s's own etcd snapshot annotations:
prod-1/2 at -05:00 (Central), prod-3 at -04:00 (Eastern). Functionally harmless —
Kubernetes and etcd are UTC internally, and in UTC those snapshots are ~4 minutes
apart, which is correct staggering — but it makes cross-node log correlation
actively misleading, and this cluster has had two unplanned reboots in two days.

`Etc/UTC` chosen on repo convention, not just principle: catalog/plex.yaml records
"no other service in the repo sets a timezone ... so use the safe default" with
TZ: Etc/UTC, and apps/wave-0/plex.yaml matches (platform/crafty is America/Chicago
DELIBERATELY — a user-facing game schedule, not a log clock). Decisive extra fact:
THE THREE DEV NODES ARE ALREADY Etc/UTC, so this makes prod match dev rather than
inventing a convention. UTC also removes the class — no DST discontinuity, node
time == platform time.

New knob `node_baseline_timezone`, overridable per environment via group_vars (the
same seam as node_baseline_backports_kernel) but deliberately NOT restated there:
no environment needs to differ, and a second copy is a second thing to disagree.
Guarded by an assert in the role's existing BLOCKED style before the module runs.

`--check --diff --tags system -e k3s_hosts=k3s_prod` INDEPENDENTLY CONFIRMS the
offset evidence, and is the whole verification — the play was never run for real:
  k3s-prod-1: -"America/Chicago"  +"Etc/UTC"  changed
  k3s-prod-2: -"America/Chicago"  +"Etc/UTC"  changed
  k3s-prod-3: -"America/New_York" +"Etc/UTC"  changed
The same --check against DEV reports the task `ok` on all three — idempotence
demonstrated against live hosts, in check mode only. `--list-hosts` confirms
`-e k3s_hosts=k3s_prod` selects the three prod nodes (the diagnostic plays take
their OWN var; the wrong flag silently targets dev).

*** AUTHORED, NOT APPLIED. *** Applying it is a live-cluster action and was
deliberately left for the operator to schedule: prod's Longhorn had only just
finished resyncing after a node reboot, and changing a node's clock config during
storage recovery is not an unattended move.

`ansible-inventory --list` was NOT used anywhere — it prints DECRYPTED vault values
(the passphrase is mounted into the dev container) and has leaked the prod
become-password to a transcript once already.

=== 2. New L0 check `placeholder-guard-coupling` (R-ENV-02, R-STATE-03) ===

On 2026-09-05 the PM changed a placeholder in clusters/base/cluster-vars.yaml from
`TBD-PROD-S3-BACKUP-ENDPOINT` to `https://TBD-R2-ACCOUNT-ID...`. The guards in
bootstrap/standup.sh matched the glob `TBD-*` — a PREFIX. All three silently died,
and `resume` would have enabled velero on prod against a nonexistent hostname:
installs clean, reports healthy, no backups exist. Found by accident. A guard in
one file keyed on the SHAPE of a value in another file, with nothing tying them.

The check exercises the REAL predicate — it does not reimplement it. standup.sh is
not sourceable (top-level argv parsing hard-exits, cd, export, EXIT trap), so the
check extracts the predicate's own source lines and sources only those. Faithfulness
is pinned by: discovery of every `*_is_placeholder` (not a hardcoded name),
exactly-one-definition, "extracted block is a verbatim substring", a self-containment
gate, a canary (must refuse "", must accept a real URL), and a scan for a divergent
second copy. If the predicate grows a dependency the check FAILS LOUDLY and names
the remedy rather than degrading to testing a copy — the failure mode it exists for.

Both sides derive from real files: guarded knobs from a taint trace in standup.sh
(`x="$(cvar KEY)"` -> alias hop -> `*_is_placeholder "$x"`); values from base
cluster-vars plus each overlay's JSON6902 /data/KEY patches. The independent
detector is deliberately NOT the guard: seed idioms UNION the guard's own words with
glob anchors STRIPPED — so a prefix-anchored guard cannot pass, and deleting an arm
cannot make a value stop being placeholder-shaped.

Real tree: 32 assertions, 18 values through the extracted predicate, PASS.

NON-VACUITY VERIFIED BY ME, twice, on throwaway copies (the real repo and the
read-only spec repo untouched). My first perturbation tripped the self-containment
rail instead of the anchoring property, so I redid it shape-preserving — substring
globs -> prefix globs, nothing else — and it failed on exactly the right property,
naming mid-string and tail acceptances with the diagnosis "This is the 2026-09-05
failure exactly". Control: unperturbed copy exits 0.

=== 3. ci/touched-requirements.sh: bootstrap/ was UNMAPPED (found by the agent) ===

The path->requirement map had no `bootstrap/*` entry, so a change touching
standup.sh ALONGSIDE any already-mapped path scoped the L0 run to that other path's
requirements and silently excluded the new check — the one that exists to catch
exactly the 2026-09-05 class. A safety net with a hole where the incident was.
Mapped to R-STATE-03 R-GITOPS-01; verified the map now resolves bootstrap paths.

VERIFIED: full CI gate PASS. `run.sh --layer l0` 8/8 PASS including the new check.
Coverage unchanged at 45/51 (the new check tags existing requirements).

Refs R-ENV-02, R-STATE-03, R-TEST-01, R-PLAT-01.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-06 08:47:08 -05:00
jakman85andClaude Opus 5 fe45d0f5c5 feat(conformance): cover R-TLS-02, R-ENV-01, R-ENV-03 — 42/51 -> 45/51, and find a real gap
DECISIONS-LOG records Phase 7 being opened knowingly while the uncovered set
included R-ENV-01/02/03 — the very requirements prod provisioning serves. Two of
those three are now covered, plus R-TLS-02.

  tls-lifecycle-automated          R-TLS-02  L0 (in the merge gate)
  cert-lifecycle-automated         R-TLS-02  L2 (authored, NOT executed)
  dev-nodes-provisioned-as-code    R-ENV-01  L0
  prod-baremetal-not-dev-surface   R-ENV-03  L0

Coverage 42/51 (82.4%) -> 45/51 (88.2%), 6 gaps, 0 safety-critical. Remaining:
R-C2-01, R-C2-02, R-NET-04, R-PLAT-01, R-SEC-03, R-STATE-04. Verified through the
real runner: --layer l0 7/7 PASS, the three new ones 3/3 PASS, --strict-coverage
exit 0 with 0 label/annotation mismatches.

*** R-TLS-02 "REVOKED" IS NOT SATISFIED — A REAL GAP, NOT A TEST DEFECT. ***
The spec requires certificates be "issued, renewed, AND REVOKED without manual
steps", and docs/08 reads revocation as "removing/rotating a certificate is a
declarative change". That needs enableCertificateOwnerRef: true;
platform/cert-manager/helmrelease.yaml does not set it (chart default false).
Verified read-only on BOTH live clusters: gateway/wildcard-home-tls has ZERO
ownerReferences, so deleting the Certificate would leave the private key serving
indefinitely. The L2 test asserts the requirement AS WRITTEN and will fail until
this is decided. It was deliberately NOT softened — weakening a test to match the
implementation is how a suite stops meaning anything.

NOT FLIPPED UNATTENDED, and the reason is specific rather than caution: the
wildcard Certificate lives under ./platform, which runs prune: true. With owner
refs ON, any transient prune of that Certificate — exactly what happened to the
CDI StorageProfiles hours ago — would delete the SERVING TLS Secret for every
host, and reissuance is bounded by Let's Encrypt's duplicate-certificate limit
(5/week per identical name set), which could leave a cluster with no trusted cert
for a week. Security gap vs a plausible week-long TLS outage is an operator call.

NON-VACUITY DEMONSTRATED, which is the part that makes these worth having:
7/7 perturbations fail for R-TLS-02, 8/8 for R-ENV-01, 13/13 for R-ENV-03, and
each restored copy passes. Perturbations were applied to a throwaway copy; neither
this repo nor the read-only spec repo was modified.

TWO CHECKS INITIALLY PASSED WHEN PERTURBED and were rewritten — the exact
degrade-to-no-op failure this repo keeps hitting. The first draft asked "is this
file's DIRECTORY referenced by a kustomization", which a directory reference always
satisfies, and keyed Secrets by name alone, so external-dns's copy of
`cloudflare-api-token` satisfied cert-manager's reference. Now a recursive
kustomize resource-graph walk from each Flux path, and Secrets keyed by
(namespace, name). Both bite.

SECURITY NOTE, recorded because this project has already leaked a password once:
`ansible-inventory --list` PRINTS DECRYPTED VAULT VALUES — it emitted the prod
become-password to stdout during exploration, since the vault passphrase file is
mounted into the dev container. Both new checks use only --syntax-check and
--list-hosts, and say so in a comment. Confirmed no plaintext value reached disk:
vault.yml is still AES256-encrypted and the tree has no assigned value.

WHAT IS NOT PROVEN, per requirement: R-TLS-02 "revoked" (above), and issuance +
observed renewal await the unexecuted L2. R-ENV-01 cannot prove the running VMs
came from this module (needs OpenTofu state) — it proves the four lifecycle stages
exist as code and cover every dev node, and that the chain RESOLVES under real
ansible-playbook. R-ENV-03 cannot prove the HARDWARE (the K8s API exposes no
"am I virtual" fact; mini-PC/bare-metal is operator-attested) — it proves prod is
AUTHORED as 3 directly-reached nodes with no IaC creating them, sharing dev's
playbooks and roles, with no prod-only manifest, branch or role anywhere.

VERIFIED BY ME: coverage delta and the 3/3 L0 pass through the real runner; the
zero-ownerReferences finding on both clusters; that the new checks avoid
ansible-inventory; that no credential material reached disk. Full CI gate PASS.

Refs R-TLS-02, R-ENV-01, R-ENV-03, R-TEST-01..05.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-05 14:17:03 -05:00
jakman85andClaude Opus 5 8a2be824f6 feat(ci): substitution-safety gate + escape 6 comment hazards; strict mode is now SAFE
Adds conformance/tests/l0-static/flux-postbuild-substitution/ — an offline, stdlib-only
L0 check tagged R-ENV-02 + R-GITOPS-01 (deliberately NOT R-TEST-01, which would
overclaim). It reuses lib/status.py's `resources:` walk rather than forking it, so "what a
path renders" has exactly one definition. It asserts: every ${...} on a substituted path
is defined-or-escaped (comments included); no default-operator form on an undefined name;
no unparseable expansion; the rendering path actually substitutes cluster-vars; every
overlay-patched /data/KEY has a base default; and that the gate itself is load-bearing.

WIRED AS AN UNCONDITIONAL STAGE (3c) in ci/l0.sh, and that matters: stage 3b scopes to
touched requirements, and ci/touched-requirements.sh maps only clusters/* to R-ENV-02 —
so an edit to a substituted manifest under apps/ or platform/ would have filtered this
check out of exactly the change that introduces the bug. l0 fail -> BLOCK=1 in gate.sh.

Mutation-tested all five assertion classes (undefined token, indexed expansion, :- form,
missing substituteFrom, un-wired gate); each fails as intended, baseline green.

STRICT MODE IS PROVEN SAFE TO ENABLE — measured, not inspected. For all 11 substituted
paths, `kubectl kustomize --load-restrictor LoadRestrictionsNone <path> | flux envsubst
--strict` exits 0 against cluster-vars base defaults. Negative control confirms the
incident: with K3S_API_HOST unset, non-strict renders k8sServiceHost empty while strict
exits 1 with `variable not set (strict mode)`. The feature gate itself is NOT flipped
here — that is a live-cluster change and the operator's call.

Escapes the six comment hazards that stood in the way (apps/vms/unifi x4,
platform/longhorn x2). All were prose ABOUT the escaping convention; since the repo's own
contract is that $${...} emits a literal ${...}, escaping them is both precise and
self-consistent. kustomize strips these particular comments today, but comment survival is
an implementation detail — a body comment carrying $ WAS observed surviving into built
apps/wave-1 — and a survivor would kill the whole Kustomization.

DOCTRINE CORRECTED (measured against flux 2.4.0): clusters/base/apps-vms.yaml claimed an
indexed expansion is fatal "escaped or not". Escaping is TOTAL — $${ARR[0]} emits the
literal text safely; only the UNESCAPED indexed form is fatal. Corrected in place.

Also empties DOCUMENTED_GAPS in conformance/run.sh with rationale: R-ENV-02 and
R-GITOPS-01 now have a real verifying test, which run.sh's own comment named as the
trigger for removing them.

Advisory output (not failures): 6 dead cluster-vars knobs — BASE_DOMAIN, CLUSTER_ENV,
CLUSTER_POD_CIDR, PIA_REGION, PLATFORM_NODE_SELECTOR_{KEY,VALUE}. The last four are
documented reference-only; BASE_DOMAIN and CLUSTER_ENV look like genuine rot (hostnames
are hardcoded elsewhere) and are worth a follow-up.

Gate: MERGE GATE: PASS. 102 assertions, 0 hazards.

Serves: R-ENV-02, R-GITOPS-01.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-01 01:31:07 -05:00
jakman85andClaude Opus 5 6e77150ae1 fix(ci): kill the false-green base resolution; cover R-SEC-01/02 + R-TEST-06 (L0)
Unblocks the enforced Gitea gate, which correctly BLOCKED this branch on
['R-SEC-01','R-SEC-02','R-TEST-06'] being touched but uncovered. Tests were authored;
the coverage rule was NOT narrowed.

=== THE FALSE GREEN (root cause) ===
`make -C ci gate` locally reported PASS having checked NOTHING. resolve_base() validated
nothing, and EVERY way of failing to find a base degraded silently to an empty touched
set + exit 0: an `HEAD~1` last resort, a two-dot diff fallback masking a failing
three-dot diff, and no check that HEAD is even AHEAD of the base. Run while sitting on
`main` (main == origin/main), `origin/main...HEAD` is empty -> "no changed files vs
base" -> the empty set means run.sh is invoked with NO --requirements -> coverage.py's
--require gate never runs at all. Green by vacuum.

Now: candidates are BASE_REF > GITHUB_BASE_REF > origin/main > main, each verified to
resolve. HEAD~1 and the two-dot fallback are gone. Unresolvable base / no merge base /
failed diff / HEAD-not-ahead -> loud banner, EMPTY stdout, exit 3, and a machine-readable
reason (on-base | no-base | diff-failed). gate.sh turns exit 3 into an UNDECIDED
decision: `on-base` -> "NOT APPLICABLE — no branch under gate" (exit 0 so push-to-main
runs don't break, but it never reads as PASS); `no-base`/`diff-failed` -> BLOCKED. It
also now prints the diff base it used. A stale local main warns (widens, never falsely
greens). Also fixed `make -C ci test`, which was SWALLOWING failing suites.

Regression: ci/tests/touched-requirements-base.test.sh — 31 assertions over 8
throwaway-repo scenarios including the exact false-green case, an already-merged branch,
no base ref, a single-commit repo, bogus BASE_REF, a shallow GITHUB_BASE_REF, and
origin/main-preferred-over-stale-main.

=== NEW L0 STATIC LAYER (conformance/tests/l0-static/) ===
Where docs/14 actually puts these invariants. Each check.sh really executes offline via
`run.sh --layer l0` and is wired into ci/l0.sh, so it runs on every PR. No scaffolds.

R-SEC-01 no-plaintext-secrets (165 assertions / 445 tracked files): .sops.yaml holds only
PUBLIC age recipients; every committed Secret is SOPS-wrapped with a mac and ENC[...]
values; ansible vault.yml files are $ANSIBLE_VAULT; no secret-valued ansible key carries
a literal outside a vault; no PEM key / age secret key / kubeconfig client-key-data is
tracked; Flux declares sops decryption; the disallow-plaintext-secrets policy Enforces
and has a proving `result: fail` row. Exemptions are SELF-POLICING, not path allowlists —
a .example must be referenced by no kustomization and hold only placeholders, and the
Kyverno fixture is exempt ONLY while a test asserts the policy rejects it. Proven against
7 injected violations, including deleting the negative row and weakening Enforce->Audit.

R-SEC-02 least-privilege-rbac (41 assertions): no wildcard apiGroup/resource/verb and no
escalate|bind|impersonate in any in-repo Role/ClusterRole; no ServiceAccount bound to
cluster-admin or a wildcard role; the conformance runner's own test identity is scoped
and holds no rights over RBAC objects; the default-deny/host-namespace/egress-class
policies Enforce and each carry a negative row. Proven against 6 injected violations.

R-TEST-06 spec-immutability-enforced (16 assertions): drives the REAL
ci/spec-immutability.sh against HERMETIC throwaway spec repos and asserts its exit code
in every state — clean-at-pin 0, edits under docs/ or decisions/ 1, untracked addition 1,
unadopted drift 1, explicit pin bump 0, ABSENT spec repo 3 (honest skip, never green),
unpinned SPEC_PIN 1 — plus that gate.sh sets BLOCK=1 on its failure, so the mechanism is
load-bearing. The real spec repo is never written to.

Each check documents what it does NOT cover (runtime decryption, git history, runtime
least-privilege, branch protection) rather than overclaiming. status.py now reads
check.sh too, so a placeholder L0 body still outranks an `executable: true` annotation.

Coverage 37/51 -> 40/51 (11 gaps); executable tests 32 -> 35; scaffold-only stays 5 —
no sixth scaffold was added.

Serves: R-SEC-01, R-SEC-02, R-TEST-01..06.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-31 23:26:49 -05:00