diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index d203550..63b347c 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -1250,3 +1250,249 @@ fresh agent should read this entry + `infra/PHASE1-STATUS.md` + `docs/plans/` an 4. **Remaining Phase-6 onboarding** — see DESIGN-waves.md / `catalog/` (services rendered-but-not-onboarded). 5. **Vendor platform/kubevirt** remote bases proactively (same flake class as #8). 6. **Human action**: finish Headlamp login = complete TOTP enrollment in the browser. + +--- + +## 2026-08-31 — **PHASE 7 OPENED (prep only)**: first prod node on the bench + +The operator added the first **production** node and opened roadmap Phase 7. This entry records the +opening, the state of the hardware, the operator's three decisions, and the blockers that stand +between "a node exists" and "prod reconciles from `clusters/prod`". + +### Governance — Phase 7 entry gate, recorded not waived + +`roadmap/phases.md` gates Phase 7 on **two** conditions: *"Phases 1–6 proven in dev/test; operator +declares prod hardware ready."* The operator has declared hardware ready. **The first condition is +not met** — conformance is 37/51, and the uncovered set includes **R-ENV-01/02/03**, the exact +requirements prod provisioning serves. We proceed with *prep only* on that basis, with no prod +install until the §7 repo blockers close and the gate is green. This is an accepted, explicit risk, +not an oversight. + +### Node 1 as-found (`192.168.2.10`, verified by SSH 2026-08-31) + +Not blank — imaged and reachable. **HP EliteDesk 800 G5 Desktop Mini** (matches the spec's `3× mini-PC +bare metal`), i7-9700T 8c/8t, **31 GiB RAM**, Debian 13 trixie / kernel 6.12, hostname already +`k3s-prod-1`, `k3s` user reachable with the `home_network` key, VT-x + `/dev/kvm` present +(KubeVirt-capable), k3s not installed. Firmware HP R21 Ver. 02.26.00. + +**One 477 GB SATA SSD** (`C400-MTFDDAC512M` — a ~2011-era Micron/Crucial consumer drive) carrying +everything: `/` 30G, `/var` 12G, swap, and `/srv` 433G. No second device. Dev, by contrast, runs two +dedicated data disks per node (Longhorn + a dedicated SSD for etcd, added after HDD contention +starved etcd WAL fsync). + +### Operator decisions (2026-08-31) + +1. **Phase 7 opens, prep work only** — repo work and node baselining now; no k3s install until the + parameterization blockers land and the gate is green. +2. **Wait for all three nodes before `cluster-init`** — prod comes up as spec-conformant 3-node HA + embedded etcd on day one; no single-server-etcd window, no later member-add. +3. **Check drive health before choosing the disk layout** — a 15-year-old consumer SSD is not an + obvious home for prod state. + +### Findings — prod would build WRONG today + +1. **R-ENV-02 is violated in the base (the headline blocker).** The overlay model promises dev and + prod differ *only* by `cluster-vars` values. In fact dev LAN addresses are hardcoded into base + manifests: `k8sServiceHost: 192.168.1.50` (`platform/cilium/helmrelease.yaml:50` + the + `values.yaml` mirror), the Cilium LB pool `192.168.1.53-.60` + (`platform-config/cilium/l2-ip-pool.yaml`), crafty `.55`, the UniFi VM `.54`, and three S3 backup + endpoints at `192.168.1.9:9000`. **Prod is on `192.168.2.x` — none of these follow.** Fix is to + tokenize each into a `cluster-vars` knob and set it in both overlays; the `postBuild.substituteFrom` + seam already exists on every relevant Kustomization. Note `platform/cilium/values.yaml` is only a + *reference mirror* — the authoritative pair is the HelmRelease (Flux) and the Ansible j2 template + (already parameterized), so the Ansible side needs no change. +2. **`clusters/prod` never sets `GATEWAY_VIP`** → it would inherit the base default `10.73.0.240`, + an address on the **retired libvirt network**. `cluster-vars`' own comment flags this as owed + "when hardware is built". +3. **Sudo on node 1 requires a password** — every Ansible playbook fails at its first `become`. + **Operator ruling: prod does NOT get NOPASSWD sudo** (my initial recommendation to mirror dev was + rejected, correctly — dev's passwordless sudo is a convenience that should not propagate to + prod). Instead prod uses Ansible `become` with an **ansible-vault-encrypted** + `ansible_become_password` in `inventory/group_vars/k3s_prod/vault.yml`, mirroring the existing + `hypervisor` and `proxmox_hosts` groups; the decrypting passphrase stays outside the repo at + `~/.config/home-k3s/ansible-vault-pass`. Plays still run unattended, no plaintext secret in git, + and prod keeps a real sudo boundary. +4. **`node_baseline` device defaults are virtio** (`/dev/vdb`, `/dev/vdc`) and do not exist on bare + metal; the role makes a filesystem on a *block device*, so a single-disk node needs either a + second drive, `/dev/sda5`, or a repartition. Prod `group_vars` must override. +5. **Single upstream resolver on node 1** (`192.168.2.1`) — the same SPOF class that broke + containerd pulls and CoreDNS forwarding on dev. The fix is **stranded unmerged** on + `fix/node-dns-resilient-upstream`. + +### RESOLVED BY OPERATOR — status comes from the repository, not the task board + +I raised an escalation here: `AGENTS.md` §11 designates `agents/task-board.md` as the one surface +agents may edit, yet `ci/spec-immutability.sh` fails the merge gate on **any** spec-repo dirt with no +path exception, and committing the board there moves spec `HEAD` off `ci/SPEC_PIN` — which also +fails. Editing the board as instructed blocks every merge. That is why the board still reads +all-`todo` after six delivered phases. + +**Operator ruling (2026-08-31): the task board is not the status surface. The repository is.** +Status of each requirement/spec is to be *derived from the state of the repo* — what is authored, +what is reconciled, what has a green requirement-tagged test — not from a hand-maintained board. + +Consequences, adopted: +- The spec board is left alone. It is a historical seed, not a live surface. No agent edits it, and + no `SPEC_PIN` churn is spent keeping it current. +- `ci/spec-immutability.sh` keeps its full strictness — **no path exception was added**. Weakening + an R-TEST-06 gate to make a bookkeeping file writable would have been a defect, not a fix. +- The **repo-derived spec status report** becomes the artifact of record, alongside the existing + requirement-coverage report and this log. Coverage already answers "does a tagged test exist"; + the status view must also answer "is this requirement actually *implemented and reconciled* in + the repo as it stands". + +### Next steps + +1. **Operator:** configure NOPASSWD sudo on `k3s-prod-1`, then capture `smartctl -a /dev/sda` + (commands in `infra/PROD-PROVISIONING.md` §5). The disk decision waits on that output. +2. **Land the R-ENV-02 tokenization** — the critical path for prod reconciling at all. Dev overlay + must set every new knob to today's exact value so live dev sees zero drift (`make -C ci render-golden`). +3. **Complete `clusters/prod`** — `GATEWAY_VIP`, the L2 pool, and the prod backup target. +4. **Land the two stranded fix branches** (`fix/node-dns-resilient-upstream`, + `fix/act-runner-dind-readiness`) — the DNS one now matters for prod too. +5. Resolve the task-board escalation above. +6. Nodes 2 and 3 through `infra/PROD-PROVISIONING.md` §1–§5 as they arrive. + +--- + +## 2026-08-31 — CI is the enforcement point; the LOCAL gate produced a FALSE GREEN + +**Operator directive: "Conformance and gates are enforced at the gitea runner and CI/CD level. +Use it."** Adopted. The authority on whether a change is mergeable is the Gitea Actions run, not a +local `make -C ci gate`. Local runs are a fast pre-check only. + +This was proven immediately and expensively. Branch `docs/phase-7-open-prod-node-1`: + +- **Local** `make -C ci gate` → `no changed files vs base; empty touched-requirement set` → + **MERGE GATE: PASS**. +- **CI**, same commit (`a1fb31a`, task 200) → 17 touched requirements correctly computed → + `REQUIRE: FAILED — required requirement(s) uncovered: ['R-SEC-01','R-SEC-02','R-TEST-06']` → + **MERGE GATE: BLOCKED**. + +The local run resolved an empty diff base and therefore checked nothing, reporting green. **A local +false green is the most dangerous failure mode this project has** — it hid real coverage debt from +the PM and would have hidden it from any future agent. `ci/touched-requirements.sh` base-ref +resolution is being fixed so local agrees with CI, or announces loudly that it cannot determine a +base. An honest noisy skip beats a quiet false pass (`ci/README.md` exit-code contract). + +The gate itself was RIGHT: we authored a vaulted-secret flow (R-SEC-01) and status tooling that +reasons about spec immutability (R-TEST-06) while those requirements had **no covering test at +all**. Tests are being authored rather than the rule being narrowed. + +**How to read CI from a workstation** (the API paths are non-obvious, so recording them): +```bash +CRED=$(printf 'protocol=https\nhost=gitea.home.keithjackson.dev\n\n' | git credential fill) +# list recent runs (id | workflow | branch | status | sha): +curl -sk -u "$U:$P" "$B/api/v1/repos/Home-Network/home-network-k3s-impl/actions/tasks?limit=10" +# fetch a task's log — NOTE the job-id/task-id OFFSET: task N is at jobs/(N+5)/logs on this instance +curl -sk -u "$U:$P" "$B/api/v1/repos/Home-Network/home-network-k3s-impl/actions/jobs/205/logs" +``` +`.../actions/runs//jobs` is 404 on this Gitea version; use `actions/tasks` + `actions/jobs//logs`. + +### OPERATOR ATTESTATIONS (2026-08-31) — capability vs. automated proof + +The operator states, and this log records as fact: + +1. **DR is PROVEN** — Velero backups to the Proxmox server, restore demonstrated. R-STATE-03's + capability is delivered and in operational use. +2. **Cluster resilience is PROVEN at the dev-cluster level** — node loss / rebuild demonstrated. + It is slow because Longhorn storage is on HDD (etcd is the exception, on its dedicated SSD), + which is a performance property, not a correctness gap. + +**Distinguish capability from automated proof.** The new repo-derived status report flags +`R-STATE-03` as a safety-critical requirement whose L3 test body asserts nothing. That finding +stands and is NOT contradicted by the attestation above — it is **test debt, not capability debt**. +The system does the thing; the suite does not yet assert that it does. Per the project's own +definition of done, only a green requirement-tagged test is acceptance — so the remaining work is +to make `dr-restore/chainsaw-test.yaml` and the node-loss test actually assert the behaviour the +operator has already demonstrated, and let the runner enforce it. Recording the attestation here +does not close the requirement, and the gate must not be softened to make it appear closed. + +--- + +## 2026-09-01 — Phase-7 prep session complete. **PM HANDOFF.** + +Everything below is on branch `docs/phase-7-open-prod-node-1` / **PR #4**, 13 commits, the +enforced Gitea gate **green**. Nothing has been installed on prod hardware. `main` is untouched +until the operator merges. + +### Delivered +1. **Phase 7 opened (prep only)** with the entry-gate shortfall recorded, not waived. +2. **T-7.1 checklist** (`infra/PROD-PROVISIONING.md`) written against the real hardware. +3. **R-ENV-02 tokenization** — 6 `cluster-vars` knobs; prod would otherwise have come up + announcing DEV addresses. Dev proven byte-identical. +4. **Prod Ansible**: inventory, vaulted `become` (prod gets NO passwordless sudo), and + **target-group parameterisation** so prod reuses the roles unchanged. +5. **`node_health` role** — read-only, portable dev+prod, run against both. +6. **Repo-derived spec status report** (`run.sh --status`). +7. **CI false-green killed** + L0 static tests for R-SEC-01/02 and R-TEST-06. +8. **dind readiness fix**; **DNS resilience re-authored** for the current stack. +9. **`dr-restore` now asserts** — safety-critical scaffolds 0. + +### Numbers +Coverage 37/51 → **40/51**. Executable tests 32 → **36**. Scaffold-only 5 → **4**. +Safety-critical without an executable L3 1 → **0**. **PROVEN: 0 — still not derivable**, by +design; proof-of-pass is not recoverable from repo state, so a green-run ledger is the only +thing that ever closes a requirement. + +### Corrections made during the session (recorded so they are not re-litigated) +- **NOPASSWD sudo on prod: REJECTED** by the operator. Prod uses vaulted `become`. My original + checklist copied dev's convenience onto prod; that was wrong. +- **The "`$i` envsubst bug" was FALSE.** An assessment claimed Flux envsubst ate bare `$i`, + making the dind cert-wait bound a no-op. Running the real `flux envsubst` disproved it: bare + `$VAR` passes through; only `${VAR}` expands — matching this repo's own contract in + `clusters/base/apps-vms.yaml`, itself written after empirical probing. **No escaping was + applied**; `$${i}` would have been an active regression (correct only on the envsubst path — + on a plain `kustomize build`, `sh` reads `$$` as its PID). Agent reports are not evidence; + the tool is. +- **The stranded DNS branch was NOT landed.** Its netplan mechanism no longer exists on Debian + 13; landing it would have broken `node-baseline.yml` on every node. + +### Near-misses worth remembering +- A YAML **comment** nearly broke every merge in the repo: Flux envsubst runs over comments too, + so a braced name with punctuation is a fatal parse error that fails the whole `apps-wave-1` + Kustomization. +- `roles/cilium`, `cluster_verify`, `flux_bootstrap` all hardcoded `.cache/dev.kubeconfig` — the + file `dev.sh` exports as `$KUBECONFIG`. The moment prod became targetable, a prod run would + have silently overwritten it and left the operator's `kubectl` answering from **prod**. +- Prod's `GATEWAY_VIP` outside the LB pool would have brought the Gateway up on `.21` while + split-horizon DNS answered `.20` — blackholing in-cluster SSO while reporting healthy. + +### BLOCKED ON THE OPERATOR — nothing else moves without these +1. **Create `inventory/group_vars/k3s_prod/vault.yml`** (`ansible_become_password`). The + passphrase file already exists. This gates SMART, the packages-only baseline, and every + future prod play. +2. **Drive health decision** — run `smartctl -a /dev/sda` once `smartmontools` lands via + `node-baseline.yml -e k3s_hosts=k3s_prod --tags packages`. Prod state on an unmeasured + ~2011 consumer SSD is the open risk. +3. **Prod's second DNS upstream** — `192.168.2.1` is both gateway and only IPv4 resolver. +4. **Prod backup target** — `S3_BACKUP_ENDPOINT` is still `TBD-PROD-*`; dev's versitygw is on + the far side of the pfSense boundary. +5. **`UNIFI_L4_VIP` / `CRAFTY_L4_VIP`** — allocatable from `.20-.30`, not invented here. +6. **Merge PR #4.** + +### Known, documented, NOT fixed +- `roles/flux_bootstrap` shells to `bootstrap/flux/bootstrap.sh` with a hardcoded + `--path=clusters/dev` — retargeting prod needs a `bootstrap/` change. +- Prod has no VPN egress group (needs its own vaulted PIA credentials). +- **`dr-restore` is AUTHORED, NEVER EXECUTED** (`conformance.home.k3s/never-executed`). It is + destructive; converting it to proven needs an operator maintenance-window run. The log must + show `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK`, **not** `DR_RESTORE_SKIPPED_OK` — green but + proving nothing. +- `dind` → native sidecar `initContainer` with a `docker info` startupProbe (NOT a bare + tcpSocket probe) is the right shape on k3s 1.36; deferred so a merge-gate failure is + unambiguous. +- **Non-ECC RAM — RISK ACCEPTED by the operator (2026-09-01), not an open item.** `node_health` + correctly reports ECC status as unknowable (no EDAC controller bound). Operator rationale: + this is a 3-node k3s homelab, not enterprise k8s; corruption on one node is outweighed by the + other two and results in node ejection. That holds for the dominant failure mode — etcd raft + (per-entry CRC + boltdb page checksums, quorum of 2 survives) and Longhorn (3 replicas, hard + anti-affinity, rebuild from healthy peers) both detect a *divergent* node and eject/rebuild it. + The residual it does not cover is corruption introduced *upstream* of replication: bytes + corrupted before the checksum is computed replicate faithfully to all three replicas, so + quorum sees no disagreement. Low probability, and the mitigation (ECC hardware) is out of + scope for this platform. **Do not re-litigate.** The one thing that operationalises the + operator's own argument is Longhorn's `snapshotDataIntegrity`, which is currently left at the + chart default rather than set explicitly — see the follow-up in `platform/longhorn/`. +- L2/L3 conformance still SKIPs in CI (`docs/plans/l2-l3-conformance-ci-wiring.md`); it needs + the trigger-credential decision — a scoped in-cluster SA, never a broad kubeconfig in CI. diff --git a/apps/vms/unifi/30-virtualmachine.yaml b/apps/vms/unifi/30-virtualmachine.yaml index 988aa7a..dc9a963 100644 --- a/apps/vms/unifi/30-virtualmachine.yaml +++ b/apps/vms/unifi/30-virtualmachine.yaml @@ -89,7 +89,13 @@ spec: # userDataSecretRef only if needed). qemu-guest-agent gives the KubeVirt AgentConnected # signal (proves cloud-init ran); install-uos resolves the latest UOS linux-x64 build from # Ubiquiti's PUBLIC firmware API (no auth) and installs it non-interactively (UI on :11443). - # NOTE: apps-vms has NO postBuild.substituteFrom, so shell ${...} below is NOT envsubst'd. + # NOTE: apps-vms NOW carries postBuild.substituteFrom cluster-vars (needed for + # ${UNIFI_L4_VIP} in 50-service-l4.yaml), so Flux envsubst DOES run over this userData. + # Every shell brace-expansion below is therefore escaped as $${...}, which envsubst emits + # as a literal ${...} — verified with `flux envsubst`. Bare $VAR and $(...) are left alone + # by envsubst and need no escaping. NEVER write an unescaped ${...} here, and never write + # ${ARR[0]} even escaped-adjacent: envsubst hard-fails on `[` ("missing closing brace") + # and would break the whole apps-vms apply. userData: |- #cloud-config hostname: unifi @@ -103,11 +109,11 @@ spec: content: | #!/usr/bin/env bash set -euo pipefail - if [ -z "${UOS_URL:-}" ]; then + if [ -z "$${UOS_URL:-}" ]; then UOS_URL="$(curl -fsSL https://fw-update.ui.com/api/firmware-latest | jq -r '._embedded.firmware|map(select(.product=="unifi-os-server" and .platform=="linux-x64"))|sort_by(.version_major,.version_minor,.version_patch)|last._links.data.href')" fi - [ -n "${UOS_URL}" ] && [ "${UOS_URL}" != "null" ] || { echo "no UOS URL"; exit 1; } - t="$(mktemp)"; echo "downloading ${UOS_URL}"; curl -fL --retry 5 --retry-delay 10 -o "$t" "$UOS_URL"; chmod +x "$t"; echo "installing"; yes | "$t"; echo "done" + [ -n "$${UOS_URL}" ] && [ "$${UOS_URL}" != "null" ] || { echo "no UOS URL"; exit 1; } + t="$(mktemp)"; echo "downloading $${UOS_URL}"; curl -fL --retry 5 --retry-delay 10 -o "$t" "$UOS_URL"; chmod +x "$t"; echo "installing"; yes | "$t"; echo "done" runcmd: - [ systemctl, enable, --now, qemu-guest-agent ] - [ systemctl, enable, --now, podman.socket ] diff --git a/apps/vms/unifi/50-service-l4.yaml b/apps/vms/unifi/50-service-l4.yaml index 66d2c56..4ba925a 100644 --- a/apps/vms/unifi/50-service-l4.yaml +++ b/apps/vms/unifi/50-service-l4.yaml @@ -8,8 +8,9 @@ # CANNOT express: raw TCP on non-HTTP ports (8080 inform, 6789), UDP at all (3478 STUN, 10001 # discovery), or the device-facing protocols that never speak HTTP. UniFi device ADOPTION rides # exactly those L4 ports. So the controller is exposed as a type=LoadBalancer Service whose VIP -# is pulled from the CiliumLoadBalancerIPPool `default-pool` (192.168.1.53-.60) and ARP-announced -# on the flat LAN by the existing CiliumL2AnnouncementPolicy `default-l2` — the SAME L2/LB-IPAM +# is pulled from the CiliumLoadBalancerIPPool `default-pool` (dev 192.168.1.53-.60 — the +# CILIUM_LB_POOL_START/STOP cluster-vars knobs) and ARP-announced on the flat LAN by the existing +# CiliumL2AnnouncementPolicy `default-l2` — the SAME L2/LB-IPAM # machinery the Gateway ingress VIP already uses (platform-config/cilium/*). No new Cilium config # is required; these Services just consume the pool. # @@ -23,8 +24,11 @@ # into the guest. externalTrafficPolicy: Local preserves the LAN client source IP and keeps the # VIP pinned to the node running the VM (also what L2 announcement does). # -# FLAG: confirm 192.168.1.54 is free, inside default-pool, and reserved in tofu (the Gateway -# already holds one pool IP, typically .53). Adjust if it collides. +# VIP: ${UNIFI_L4_VIP} (cluster-vars, R-ENV-02; dev 192.168.1.54, confirmed free, inside +# default-pool and reserved in tofu — the Gateway already holds the pool's first IP, dev .53). +# Substituted by the `apps-vms` Flux Kustomization, which carries postBuild.substituteFrom for +# exactly this token; every shell ${...} in 30-virtualmachine.yaml's cloud-init is escaped as $${...} +# accordingly. A new environment MUST set UNIFI_L4_VIP to a free address in ITS pool. # # REUSABLE PATTERN (the deferred `minecraft` :25565/L4 problem is the same shape): expose an # L4-only workload as a Cilium LoadBalancer on a pinned, L2-announced VIP with a per-app sharing @@ -39,8 +43,8 @@ metadata: app.kubernetes.io/name: unifi annotations: lbipam.cilium.io/sharing-key: "unifi" - lbipam.cilium.io/ips: "192.168.1.54" - # external-dns (source: service) publishes A unifi.home.keithjackson.dev -> 192.168.1.54. + lbipam.cilium.io/ips: "${UNIFI_L4_VIP}" + # external-dns (source: service) publishes A unifi.home.keithjackson.dev -> the pinned VIP. # This is the browser + device target; UI is https://unifi.home.keithjackson.dev:11443. external-dns.alpha.kubernetes.io/hostname: "unifi.home.keithjackson.dev" spec: @@ -65,7 +69,7 @@ metadata: app.kubernetes.io/name: unifi annotations: lbipam.cilium.io/sharing-key: "unifi" # same key + same IP => shares the VIP with -tcp - lbipam.cilium.io/ips: "192.168.1.54" + lbipam.cilium.io/ips: "${UNIFI_L4_VIP}" spec: type: LoadBalancer externalTrafficPolicy: Local diff --git a/apps/wave-0/nextcloud-db.yaml b/apps/wave-0/nextcloud-db.yaml index c61f487..accab0f 100644 --- a/apps/wave-0/nextcloud-db.yaml +++ b/apps/wave-0/nextcloud-db.yaml @@ -74,11 +74,13 @@ spec: # (defaults to the Cluster name), so objects land under s3://cnpg/nextcloud-db/. Path-style HTTP. # Credentials: the scoped key from the in-namespace SOPS Secret nextcloud-db-backup-credentials # (same-ns only), kept in sync with platform/backup/minio-credentials.sops.yaml. + # endpointURL is ${S3_BACKUP_ENDPOINT} (cluster-vars, R-ENV-02; dev http://192.168.1.9:9000), + # substituted by the `apps-wave-0` Flux Kustomization. Prod supplies its own off-cluster target. backup: retentionPolicy: "30d" barmanObjectStore: destinationPath: s3://cnpg - endpointURL: http://192.168.1.9:9000 + endpointURL: ${S3_BACKUP_ENDPOINT} s3Credentials: accessKeyId: name: nextcloud-db-backup-credentials diff --git a/apps/wave-0/semaphore-db.yaml b/apps/wave-0/semaphore-db.yaml index cdd5884..7c8c3a5 100644 --- a/apps/wave-0/semaphore-db.yaml +++ b/apps/wave-0/semaphore-db.yaml @@ -75,11 +75,13 @@ spec: # (defaults to the Cluster name), so objects land under s3://cnpg/semaphore-db/. Path-style HTTP. # Credentials: the scoped key from the in-namespace SOPS Secret semaphore-db-backup-credentials # (same-ns only), kept in sync with platform/backup/minio-credentials.sops.yaml. + # endpointURL is ${S3_BACKUP_ENDPOINT} (cluster-vars, R-ENV-02; dev http://192.168.1.9:9000), + # substituted by the `apps-wave-0` Flux Kustomization. Prod supplies its own off-cluster target. backup: retentionPolicy: "30d" barmanObjectStore: destinationPath: s3://cnpg - endpointURL: http://192.168.1.9:9000 + endpointURL: ${S3_BACKUP_ENDPOINT} s3Credentials: accessKeyId: name: semaphore-db-backup-credentials diff --git a/apps/wave-1/gitea-actions.yaml b/apps/wave-1/gitea-actions.yaml index 4447d82..198edb5 100644 --- a/apps/wave-1/gitea-actions.yaml +++ b/apps/wave-1/gitea-actions.yaml @@ -103,21 +103,65 @@ spec: # ---- act_runner: registers with Gitea, dispatches jobs to the DinD daemon ---- - name: runner image: gitea/act_runner:0.2.11 - # Wait for the DinD sidecar to publish its TLS client certs into the shared - # emptyDir before starting the daemon — act_runner reads /certs/client/ca.pem at - # startup and hard-exits if it is missing (a startup race against the sidecar). - # Then exec the image's normal entrypoint (/sbin/tini -- /opt/act/run.sh), which - # registers-from-env on first boot and runs `act_runner daemon`. + # TWO-STAGE BOUNDED WAIT on the DinD sidecar, then exec the image's normal + # entrypoint (/sbin/tini -- /opt/act/run.sh), which registers-from-env on first + # boot and runs `act_runner daemon`. + # + # STAGE 1 — the certs must EXIST: act_runner reads /certs/client/ca.pem at startup + # and hard-exits if it is missing (a startup race against the sidecar). + # STAGE 2 — the daemon must actually ACCEPT a connection: the cert FILE appearing is + # necessary but NOT sufficient. dockerd writes the certs and only THEN begins + # listening on :2376, so a runner that proceeds on the file alone still hard-exits + # with "cannot ping the docker daemon at tcp://localhost:2376" — observed + # crashlooping 11 times in ~8 minutes on a fresh pod after a node bounce. + # Both loops are bounded (60 x 1s) and exit 1 on timeout, so a genuinely dead sidecar + # surfaces as a CrashLoopBackOff with a clear message instead of hanging forever. + # + # ENVSUBST SAFETY — the BARE $i / $j below are CORRECT. Do not "tidy" them into the + # braced form or the double-dollar form; both are wrong here, for different reasons. + # clusters/base/apps-wave-1.yaml carries postBuild.substituteFrom cluster-vars, so + # Flux envsubst runs over this ENTIRE manifest — YAML comments very much included. + # Its contract (drone/envsubst; the same one spelled out in clusters/base/apps-vms.yaml, + # and re-verified against this exact file with `flux envsubst`, flux v2.4.0) is: + # - A bare $name is NOT a reference to envsubst: it passes through untouched, and + # that holds even when the name IS set in cluster-vars. So $i and $j reach the + # cluster verbatim and both 60s bounds below are real, enforced bounds. + # - $((i + 1)) is likewise untouched: "(" cannot begin a variable name. + # - A BRACED dollar-name IS a reference and gets replaced — with the empty string + # when the name is unset. One of those here would make a bound read [ "" -ge 60 ], + # which errors, short-circuits the &&, and silently turns the wait UNBOUNDED. + # - A DOUBLE-dollar braced name is envsubst's escape and survives as the braced + # form — still wrong for us, because on any path where envsubst does NOT run + # (a plain `kustomize build` or `kubectl apply -k` of this directory) the shell + # reads that leading double-dollar as its own PID, so a bound would compare a + # string like 1234{i} against 60. The bare form is the only spelling that is + # correct on BOTH paths. + # - A braced name containing punctuation is a HARD ERROR ("unable to parse variable + # name") that fails this whole Kustomization — and this Kustomization gates every + # merge in the repo. So never write one anywhere in this file, INCLUDING in a + # comment. That is why this note names the three forms in words rather than + # showing them: an earlier draft of it spelled them out literally and broke + # `flux envsubst` on the file. (Same trap the indexed-expansion warning in + # clusters/base/apps-vms.yaml describes.) command: - /bin/sh - -c - | + # Stage 1: the sidecar's TLS client certs must appear in the shared emptyDir. i=0 until [ -f /certs/client/ca.pem ]; do i=$((i + 1)) [ "$i" -ge 60 ] && { echo "dind certs never appeared" >&2; exit 1; } echo "waiting for dind TLS certs ($i)"; sleep 1 done + # Stage 2: dockerd must actually accept a connection on :2376. busybox nc in this + # Alpine-based image (gitea/act_runner:0.2.11) supports -z (scan) and -w (timeout). + j=0 + until nc -z -w1 127.0.0.1 2376; do + j=$((j + 1)) + [ "$j" -ge 60 ] && { echo "dind daemon never accepted a connection on 2376" >&2; exit 1; } + echo "waiting for dind daemon on :2376 ($j)"; sleep 1 + done exec /sbin/tini -- /opt/act/run.sh env: - name: GITEA_INSTANCE_URL @@ -155,6 +199,31 @@ spec: limits: memory: 512Mi # ---- Docker-in-Docker: the job executor daemon (privileged) ---- + # + # FOLLOW-UP (deliberately NOT bundled into the readiness fix above — own change): + # Convert this container into a NATIVE SIDECAR and delete both shell wait-loops. + # k3s is now v1.36, where sidecar containers are GA, so kubelet can enforce the + # ordering that the loops above hand-roll — which also removes the envsubst hazard + # from this file entirely (no shell, no $i/$j, nothing to escape or mis-escape). + # Shape to build: + # 1. Move this whole container into `initContainers:` as the FIRST entry. + # 2. Add `restartPolicy: Always` to it. That single field is what makes an + # initContainer a sidecar: it starts before the regular containers, keeps + # running alongside them, and is terminated after them. + # 3. Give it a `startupProbe` that succeeds only when dockerd accepts a TLS + # connection — e.g. `exec` a `docker -H tcp://127.0.0.1:2376 --tlsverify + # --tlscacert /certs/client/ca.pem --tlscert /certs/client/cert.pem + # --tlskey /certs/client/key.pem info`. Do NOT use a bare `tcpSocket` probe: + # it would re-open the same "port accepting but TLS/certs not ready" gap that + # Stage 1 above exists to close. `periodSeconds: 1` + `failureThreshold: 60` + # preserves today's 60s cap. + # 4. kubelet then holds the `runner` container until that startupProbe passes, so + # `runner` can drop its `command:` override completely and run the image's + # default entrypoint. + # Why it is a separate change: this Deployment is the ENFORCED MERGE GATE for this + # entire repository. If the conversion is wrong, nothing can merge — so it must land + # alone, where the failure is unambiguous and not entangled with anything else, and + # only after a real workflow run has been observed green on the converted pod. - name: dind image: docker:27-dind securityContext: diff --git a/ci/Makefile b/ci/Makefile index 632926f..3d426da 100644 --- a/ci/Makefile +++ b/ci/Makefile @@ -44,7 +44,11 @@ touched: ## Print the touched-requirement subset for the current branch vs main @"$(CI_DIR)touched-requirements.sh" test: ## Run ci/ script unit tests (no cluster; e.g. touched-requirements content scan) - @for t in "$(CI_DIR)"tests/*.test.sh; do echo "== $$t"; bash "$$t"; done + @rc=0; for t in "$(CI_DIR)"tests/*.test.sh; do \ + echo "== $$t"; bash "$$t" || rc=1; \ + done; \ + if [ $$rc -ne 0 ]; then echo "ci/tests: FAILED" >&2; else echo "ci/tests: all suites passed"; fi; \ + exit $$rc versions: ## Print pinned tool versions (R-TEST-02) @cat "$(CI_DIR)versions.env" diff --git a/ci/README.md b/ci/README.md index eac3253..bec4949 100644 --- a/ci/README.md +++ b/ci/README.md @@ -20,17 +20,54 @@ Individual stages: ```bash make -C ci lint # yaml/lint (no cluster; live now) -make -C ci l0 # ServiceDefinition schema + Kyverno (no cluster; live now) +make -C ci l0 # schema + Kyverno + L0 static conformance (no cluster; live now) make -C ci l1 # L1 rendering checks (no cluster; live now) make -C ci conformance # L2/L3 in-cluster Chainsaw Job (SKIPS unless CLUSTER_AVAILABLE=true) +make -C ci test # unit tests for these scripts themselves (ci/tests/*.test.sh) ``` +The `l0` stage has three sub-stages: ServiceDefinition schema validation, the Kyverno policy gate +(`kyverno test`), and **`conformance/run.sh --layer l0`** — the whole-repo L0 invariants a +per-resource policy cannot express (secret hygiene across every tracked file, least-privilege RBAC, +spec-immutability enforcement). See `conformance/tests/README.md`. + ## Exit-code contract Every stage script uses: **`0` = pass · `1` = fail (blocks merge) · `3` = SKIPPED** (cluster-gated with no cluster, or an unwired seam / missing tool). A `3` is **never** counted as green — that is how the gate stays honest in Phase 0 (no fake green). +### The touched-requirement subset can also be *undeterminable* (exit 3) + +`touched-requirements.sh` is the input to the R-TEST-03 decision. It resolves a diff base in this +order, **verifying each candidate exists** before using it: + +``` +BASE_REF (explicit) > GITHUB_BASE_REF (Actions PR runs) > origin/main > main +``` + +If no candidate resolves, if there is no merge base, if the diff command fails, or if **HEAD is not +ahead of the base** (you are sitting on `main`, or the branch is already merged), it prints a loud +banner, writes **nothing** to stdout, and exits **3** — plus a machine-readable reason code +(`on-base` / `no-base` / `diff-failed`) to `$TOUCHED_REASON_FILE` when that variable names a path. + +**Why this is not cosmetic.** Every one of those conditions used to degrade *silently* to an empty +touched set and exit 0. An empty set means `conformance/run.sh` is invoked with no `--requirements`, +which means its coverage `--require` gate never runs — so on 2026-08-31 the local gate printed +`MERGE GATE: PASS` having checked nothing, while CI (correct base, 17 touched requirements) BLOCKED +on real coverage debt. There is no silent narrowing left: the `HEAD~1` last resort and the two-dot +diff fallback are both gone. Locked down by `ci/tests/touched-requirements-base.test.sh`. + +`gate.sh` turns exit 3 into an **UNDECIDED** merge decision, never a pass: + +| reason | headline | exit | +|---|---|---| +| `on-base` | `MERGE GATE: NOT APPLICABLE — no branch under gate` | 0 (a push-to-`main` CI run legitimately lands here; it just must not read as "my branch is green") | +| `no-base`, `diff-failed` | `MERGE GATE: BLOCKED` | 1 (the environment cannot support a merge decision; refusing to decide must not look like deciding yes) | + +A stage that needs the subset (`l0.sh`, `l1.sh`) still runs its FULL layer as a superset sanity net +in that case — nothing goes untested — and says plainly that the *scoped* check did not happen. + ## Reproducibility (R-TEST-02) - **Version-pinned:** all tool versions live in [`versions.env`](versions.env) — the single source of @@ -43,16 +80,19 @@ the gate stays honest in Phase 0 (no fake green). ``` ci/ - Makefile targets: lint · l0 · l1 · conformance · gate · touched · tools · versions + Makefile targets: lint · l0 · l1 · conformance · gate · touched · test · tools · versions versions.env PINNED tool versions (R-TEST-02) — single source of truth lib.sh shared helpers: logging, tool checks, cluster detection, artifact paths yamllint.yaml lint config for the yaml/lint stage lint.sh Stage: yaml/lint - l0.sh Stage: ServiceDefinition schema validation + Kyverno L0 policy gate + l0.sh Stage: schema validation + Kyverno L0 gate + L0 static conformance l1.sh Stage: L1 rendering checks (Chainsaw L1 via the conformance runner) conformance.sh Stage: L2/L3 in-cluster Chainsaw Job (trigger + collect) - touched-requirements.sh compute the R-* subset a change touches (drives the R-TEST-03 filter) + touched-requirements.sh compute the R-* subset a change touches (drives the R-TEST-03 filter); + exits 3 LOUDLY rather than guessing when the diff base is unreliable gate.sh run all stages + the merge decision (the single command) + tests/*.test.sh unit tests for these scripts (`make -C ci test`), run against + throwaway git repos via the REPO_ROOT override — no network, no cluster README.md this file ``` @@ -67,7 +107,7 @@ below are this workstream's **assumptions** — flagged for the PM to verify at |---|---|---|---|---| | ServiceDefinition schema validator | WS-GITOPS · **T-0.3** (`schemas/`) | `SCHEMA_VALIDATE_CMD` | `schemas/validate.sh` | `validate.sh catalog/*.yaml` → exit 0/1; `--version` supported | | Kyverno L0 policies + tests | WS-TEST · **T-0.5** (`platform/kyverno/`) | `KYVERNO_POLICY_DIR` | `platform/kyverno` | contains `kyverno-test.yaml` files discoverable by `kyverno test ` | -| Conformance runner (Chainsaw) | WS-TEST · **T-0.4** (`conformance/`) | `CONFORMANCE_CMD` | `conformance/run.sh` | `run.sh --layer [--requirements R-A,R-B] [--trigger in-cluster-job] --junit --coverage ` | +| Conformance runner (Chainsaw) | WS-TEST · **T-0.4** (`conformance/`) | `CONFORMANCE_CMD` | `conformance/run.sh` | `run.sh --layer [--requirements R-A,R-B]`; artifacts land in `conformance/_artifacts/` | | Catalog (ServiceDefinitions) | WS-APPS | `CATALOG_DIR` | `catalog` | `*.yaml` ServiceDefinition files | Artifact locations (git-ignored): JUnit → `results/junit/*.xml`; coverage → `reports/coverage/**`. diff --git a/ci/gate.sh b/ci/gate.sh index 3a54ba4..8df27b4 100755 --- a/ci/gate.sh +++ b/ci/gate.sh @@ -48,6 +48,32 @@ echo >&2 log "==== gate decision ====" BLOCK=0 +# --- R-TEST-03 input: the touched-requirement subset -------------------------- +# Computed ONCE here. ci/touched-requirements.sh exits 3 when it cannot establish a +# reliable diff base; that used to degrade to an empty set, which silently disabled the +# coverage `--require` gate and let the gate print "MERGE GATE: PASS" having checked +# nothing (the 2026-08-31 local false green). An undeterminable subset is now surfaced +# as an UNDECIDED merge decision, never as a pass. +_reason_file="$(mktemp)" +# NOTE: command substitution, NOT `mapfile < <(...)` — process substitution does not +# propagate the child's exit status, and the whole point here is to SEE the exit 3. +TOUCHED_OUT="$(TOUCHED_REASON_FILE="$_reason_file" "${CI_DIR}/touched-requirements.sh")" +TOUCHED_RC=$? +TOUCHED_REASON="$(cat "$_reason_file" 2>/dev/null || true)" +rm -f "$_reason_file" +TOUCHED=() +[ -n "$TOUCHED_OUT" ] && mapfile -t TOUCHED <<< "$TOUCHED_OUT" +UNDECIDED=0 +if [ "$TOUCHED_RC" -eq 3 ]; then + UNDECIDED=1 + echo >&2 + err "R-TEST-03 DECISION INPUT MISSING: the touched-requirement subset could not be" + err " determined (${TOUCHED_REASON:-unknown}). The scoped coverage gate did NOT run." + summary "- **R-TEST-03 subset UNDETERMINED** (\`${TOUCHED_REASON:-unknown}\`) — the scoped coverage gate did not run." +elif [ "${#TOUCHED[@]}" -gt 0 ]; then + log "touched requirements: ${TOUCHED[*]}" +fi + # spec-immutability (R-TEST-06): a FAIL always blocks (the spec was modified/drifted). # A skip means the spec repo wasn't available to verify — surfaced, not green. case "${OUTCOME[spec-immutability]}" in @@ -77,9 +103,11 @@ case "${OUTCOME[conformance]}" in skip) if cluster_available; then err "conformance: skipped despite CLUSTER_AVAILABLE=true -> blocks merge"; BLOCK=1 + elif [ "$UNDECIDED" -eq 1 ]; then + warn "conformance L2/L3 UNVERIFIED, and the touched subset is unknown — nothing certified." else - mapfile -t reqs < <("${CI_DIR}/touched-requirements.sh" || true) - if [ "${#reqs[@]}" -gt 0 ]; then + if [ "${#TOUCHED[@]}" -gt 0 ]; then + reqs=("${TOUCHED[@]}") warn "conformance L2/L3 UNVERIFIED for touched requirements: ${reqs[*]}" warn " (no cluster — Phase 0). These are NOT certified green." summary "- **UNVERIFIED (no cluster):** ${reqs[*]}" @@ -93,15 +121,35 @@ case "${OUTCOME[conformance]}" in fi ;; esac +# An UNDECIDED subset is never a pass. Two outcomes, by reason: +# on-base -> this checkout has no branch commits to gate (you are on main, or the +# branch is already merged; a push-to-main CI run lands here). Legitimate, +# so the job is not failed — but the headline says NOT APPLICABLE, never +# PASS, so nobody reads it as "my branch is green". +# anything -> the environment cannot support a merge decision (no base ref, broken +# else diff). Refusing to decide must not look like deciding yes: BLOCK. +if [ "$UNDECIDED" -eq 1 ] && [ "$TOUCHED_REASON" != "on-base" ]; then + err "cannot compute the R-TEST-03 subset (${TOUCHED_REASON:-unknown}) -> blocks merge" + BLOCK=1 +fi + echo >&2 -if [ "$BLOCK" -eq 0 ]; then - ok "MERGE GATE: PASS (touched-requirement subset green for the layers that can run)" - summary "" - summary "**Result: PASS** — L0/L1 green. L2/L3 status per cluster availability above." - exit 0 -else +if [ "$BLOCK" -ne 0 ]; then err "MERGE GATE: BLOCKED" summary "" summary "**Result: BLOCKED** — see failures above." exit 1 +elif [ "$UNDECIDED" -eq 1 ]; then + warn "MERGE GATE: NOT APPLICABLE — no branch under gate (${TOUCHED_REASON})." + warn " The stages above ran and did not fail, but NO touched-requirement subset was" + warn " checked. This is NOT a green branch: check out the branch you mean to gate" + warn " (or set BASE_REF) and run it again." + summary "" + summary "**Result: NOT APPLICABLE** — no branch under gate (\`${TOUCHED_REASON}\`); the R-TEST-03 subset was not checked." + exit 0 +else + ok "MERGE GATE: PASS (touched-requirement subset green for the layers that can run)" + summary "" + summary "**Result: PASS** — L0/L1 green. L2/L3 status per cluster availability above." + exit 0 fi diff --git a/ci/l0.sh b/ci/l0.sh index 9708d2d..d4dc4cc 100755 --- a/ci/l0.sh +++ b/ci/l0.sh @@ -7,6 +7,9 @@ # Stage 3 L0 : `kyverno test` over platform/kyverno/ — default-deny baseline # present, no plaintext secrets, required labels/egress-class, # ServiceDefinition schema conformance (owner: T-0.5, ADR-0011). +# Stage 3b L0 : conformance/run.sh --layer l0 — the WHOLE-REPO L0 invariants that +# a per-resource policy cannot express (secret hygiene across every +# tracked file, least-privilege RBAC, spec-immutability enforcement). # # Integration seams are env-overridable (see ci/lib.sh + README). When an owning # subtree is still a placeholder, this reports SKIPPED (exit 3) for that sub-stage @@ -73,6 +76,30 @@ else rm -rf "$l0_render" fi +# ---- Stage 3b: L0 STATIC conformance checks --------------------------------- +# docs/14's L0 row is "manifests/ServiceDefinitions obey invariants", which is wider +# than what Kyverno alone can express over single resources: whole-repo invariants +# (no plaintext secrets anywhere in git, least-privilege RBAC across every tracked +# binding, the spec-immutability gate's own behaviour) live in the conformance +# harness's tests/l0-static subtree and run with no cluster. This stage executes them +# through the same single-command runner CI and a laptop both use (R-TEST-02). +log "Stage: L0 static conformance checks (${CONFORMANCE_CMD} --layer l0)" +if ! entrypoint_ready "$CONFORMANCE_CMD"; then + skip "conformance runner not present at '${CONFORMANCE_CMD}' — UNWIRED, not green" + summary "- l0-static: **SKIPPED** (conformance runner missing)" + if [ "$rc" -eq 0 ]; then rc=3; fi +else + load_touched_requirements + if "$CONFORMANCE_CMD" --layer l0 ${REQ_ARGS[@]+"${REQ_ARGS[@]}"}; then + ok "L0 static conformance checks passed" + summary "- l0-static: **passed**" + else + err "L0 static conformance checks failed" + summary "- l0-static: **FAILED**" + rc=1 + fi +fi + case "$rc" in 0) ok "L0 stage complete" ;; 3) skip "L0 stage incomplete (unwired seam / missing tool) — not a pass" ;; diff --git a/ci/l1.sh b/ci/l1.sh index 98707e3..0735146 100755 --- a/ci/l1.sh +++ b/ci/l1.sh @@ -20,19 +20,16 @@ if ! entrypoint_ready "$CONFORMANCE_CMD"; then exit 3 fi -# Touched-requirement subset (R-TEST-03). Empty subset => run all L1 as a sanity net. -mapfile -t reqs < <("${CI_DIR}/touched-requirements.sh" || true) -req_args=() -if [ "${#reqs[@]}" -gt 0 ]; then - log "touched requirements: ${reqs[*]}" - req_args=(--requirements "$(IFS=,; echo "${reqs[*]}")") -fi +# Touched-requirement subset (R-TEST-03). An empty OR undeterminable subset => run all +# L1 as a superset sanity net, and SAY SO — see load_touched_requirements in ci/lib.sh. +load_touched_requirements +req_args=(${REQ_ARGS[@]+"${REQ_ARGS[@]}"}) # RECONCILED (PM 2026-08-16): conformance/run.sh's real interface is `--layer ` # [--requirements R-A,R-B]; it ALWAYS writes artifacts to a FIXED location # (conformance/_artifacts/) and does NOT accept --junit/--coverage. Invoke by layer + # touched subset, then COLLECT its artifacts into the CI artifact dirs. -"$CONFORMANCE_CMD" --layer l1 "${req_args[@]}" +"$CONFORMANCE_CMD" --layer l1 ${req_args[@]+"${req_args[@]}"} CONF_ART="$(dirname "$CONFORMANCE_CMD")/_artifacts" if [ -f "${CONF_ART}/report.junit.xml" ]; then diff --git a/ci/lib.sh b/ci/lib.sh index 0e6cbc2..9a0bf27 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -85,3 +85,38 @@ cluster_available() { } mkartifacts() { mkdir -p "$JUNIT_DIR" "$COVERAGE_DIR"; } + +# --- Touched-requirement subset (R-TEST-03 input) ---------------------------- +# Sets: TOUCHED_RC (0 ok / 3 not determinable), TOUCHED_REQS[] and REQ_ARGS[] +# (ready to splice into a conformance/run.sh invocation). +# +# ci/touched-requirements.sh exits 3 — LOUDLY — when it cannot establish a reliable +# diff base, instead of the old silent degradation to an empty set. An empty set +# means run.sh gets no `--requirements`, which means its coverage `--require` gate +# never runs; treating "could not determine" as "nothing touched" is precisely the +# false green this helper exists to prevent. A stage still runs its FULL layer in +# that case (a superset, so nothing goes untested), but says plainly that the +# SCOPED check did not happen; ci/gate.sh turns that into an UNDECIDED decision. +load_touched_requirements() { + local out rc=0 + out="$("${CI_DIR}/touched-requirements.sh")" || rc=$? + TOUCHED_RC=$rc + TOUCHED_REQS=() + # `if`, not `[ ... ] && ...`: under `set -e` a false AND-list is the caller's last + # command status and would abort the stage script on an EMPTY touched set. + if [ -n "$out" ]; then mapfile -t TOUCHED_REQS <<< "$out"; fi + REQ_ARGS=() + if [ "$rc" -eq 3 ]; then + warn "touched-requirement subset UNDETERMINED (reason logged above)." + warn " Running the FULL layer as a superset sanity net; the SCOPED R-TEST-03" + warn " coverage check is NOT performed here — ci/gate.sh reports that as UNDECIDED." + elif [ "$rc" -ne 0 ]; then + warn "touched-requirements.sh exited ${rc}; proceeding with the full layer." + elif [ "${#TOUCHED_REQS[@]}" -gt 0 ]; then + log "touched requirements: ${TOUCHED_REQS[*]}" + REQ_ARGS=(--requirements "$(IFS=,; echo "${TOUCHED_REQS[*]}")") + else + log "empty touched-requirement set (base WAS established) — running the full layer." + fi + return 0 +} diff --git a/ci/tests/touched-requirements-base.test.sh b/ci/tests/touched-requirements-base.test.sh new file mode 100755 index 0000000..45a7ebf --- /dev/null +++ b/ci/tests/touched-requirements-base.test.sh @@ -0,0 +1,169 @@ +#!/usr/bin/env bash +# ci/tests/touched-requirements-base.test.sh — regression tests for BASE-REF RESOLUTION +# in ci/touched-requirements.sh. +# +# THE BUG THESE LOCK DOWN (2026-08-31, local/CI divergence): +# `make -C ci gate` on a feature branch printed +# "no changed files vs base; empty touched-requirement set" +# and then "MERGE GATE: PASS" — while CI, with a correct base, computed 17 touched +# requirements and BLOCKED on real coverage debt. Root cause: EVERY way of failing to +# establish a diff base degraded silently to an EMPTY touched set and exit 0. An empty +# set means conformance/run.sh is invoked without `--requirements`, which means its +# coverage `--require` gate never runs — so the gate reported PASS having checked +# nothing. A false green is worse than a red. +# +# The fix: an undeterminable base is a LOUD exit 3 (ci/README.md's SKIPPED code) with +# EMPTY stdout and a machine-readable reason code, never exit 0. These tests assert +# that for each way the base can be undeterminable, and assert the good paths still work. +# +# The tests point the real script at THROWAWAY git repos via the REPO_ROOT override +# (ci/lib.sh honours a pre-set REPO_ROOT); the script runs from its true location so it +# still sources the real lib.sh. No network, no cluster, and the working repo is never +# touched. +# +# Run: ci/tests/touched-requirements-base.test.sh (or `make -C ci test`) +set -uo pipefail # NOT -e: these tests deliberately run a script that exits non-zero. + +CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +SCRIPT="${CI_DIR}/touched-requirements.sh" + +PASS=0 FAIL=0 +ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); } +bad() { printf ' FAIL %s\n' "$*" >&2; FAIL=$((FAIL+1)); } + +TMP="$(mktemp -d)" +trap 'rm -rf "$TMP"' EXIT + +# newrepo — a throwaway repo with one baseline commit on `main`. +newrepo() { + local d="$1" + mkdir -p "$d/notes" + git -C "$d" init -q -b main + git -C "$d" config user.email t@t.t + git -C "$d" config user.name t + git -C "$d" config commit.gpgsign false # throwaway: never sign (ignore host gpg config) + echo baseline > "$d/notes/base.txt" + git -C "$d" add -A && git -C "$d" commit -qm baseline +} + +# commit_touching — a commit whose content declares an R-ID. +commit_touching() { + local d="$1" text="$2" + printf 'name: %s\n' "$text" > "$d/notes/decl.yaml" + git -C "$d" add -A && git -C "$d" commit -qm change +} + +# Runs the script in ; sets RC, OUT (stdout), ERRTXT (stderr), REASON. +run_in() { + local d="$1"; shift + local reason_file="${TMP}/reason.$$" + : > "$reason_file" + OUT="$(REPO_ROOT="$d" TOUCHED_REASON_FILE="$reason_file" "$@" "$SCRIPT" 2>"${TMP}/err.$$")" + RC=$? + ERRTXT="$(cat "${TMP}/err.$$")" + REASON="$(cat "$reason_file")" +} + +# assert_refusal