Merge pull request 'fix(ci): CLUSTER_AVAILABLE=true would have faked a green' (#12) from fix/cluster-available-would-fake-a-green into main

Reviewed-on: #12
Reviewed-by: Keith Jackson <[email protected]>

.gitea/workflows/gate.yml

@@ -124,8 +124,13 @@ jobs:
       - name: Merge gate — ci/gate.sh (what `make -C ci gate` runs)
         env:
           GATE_STRICT_SKIPS: "true"
-          GITEA_SETTINGS_TOKEN: ${{ secrets.GITEA_SETTINGS_TOKEN }}
-          # REQUIRE_GITEA_SETTINGS: "true"   # <- uncomment AFTER seeing gitea-settings green
+          SETTINGS_API_TOKEN: ${{ secrets.SETTINGS_API_TOKEN }}
+          # ENABLED 2026-09-29. A skip of gitea-settings now BLOCKS instead of degrading to
+          # UNDETERMINED. Safe because THIS PR's own CI run is the proof: if the stage were
+          # skipping, this line turns that into a failure and the PR goes red instead of
+          # merging. Green here means the token resolved and the live Gitea matched the
+          # declaration in infra/gitea/expected-branch-protection.json.
+          REQUIRE_GITEA_SETTINGS: "true"
         run: ./ci/gate.sh || [ "$?" -eq 3 ]
 
       # NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4

ci/conformance.sh

@@ -61,5 +61,45 @@ if [ -d "${CONF_ART}/coverage" ]; then
   cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
 fi
 
+# ---------------------------------------------------------------------------------------
+# DID IT ACTUALLY RUN? -- the guard without which this stage FAKES A GREEN.
+#
+# run.sh exits 0 in "scaffold mode" when it cannot reach a cluster: it emits a JUnit file
+# whose every testcase is <skipped> and returns success. That is correct FOR run.sh -- an
+# honest skip is its documented no-cluster behaviour. It is catastrophic HERE, because
+# reaching this point means the environment DECLARED CLUSTER_AVAILABLE=true, and this stage
+# would otherwise print "L2/L3 conformance passed" over 54 tests that never executed.
+#
+# MEASURED 2026-09-29, before this guard existed:
+#   CLUSTER_AVAILABLE=true ./ci/conformance.sh   ->  rc=0
+#   "[junit] wrote skipped JUnit for 54 scaffolded test(s)"
+#   "[ ok ] L2/L3 conformance passed for touched subset"
+# One Gitea variable away from a fake green on an enforcing merge gate. That is exactly the
+# HARD RULE 4 violation this repo's whole gate design exists to prevent, and nothing caught
+# it because nobody had ever set the variable.
+#
+# A skip while a cluster is DECLARED is a contradiction: either the declaration is wrong or
+# the cluster is unreachable. The merge-gate decision matrix already says this case is BLOCK
+# ("conformance skipped while CLUSTER_AVAILABLE=true -> BLOCK (it should have run)"). This
+# makes the stage itself honour that instead of reporting a pass.
+JUNIT="${CONF_ART}/report.junit.xml"
+if [ -f "$JUNIT" ]; then
+  total="$(grep -o '<testcase' "$JUNIT" | wc -l | tr -d ' ')"
+  skipped="$(grep -o '<skipped' "$JUNIT" | wc -l | tr -d ' ')"
+  if [ "${total:-0}" -gt 0 ] && [ "${skipped:-0}" -ge "${total:-0}" ]; then
+    err "CLUSTER_AVAILABLE=true, but the L2/L3 suite SKIPPED every one of its ${total} tests"
+    err "  (scaffold mode -- run.sh could not reach a cluster and exited 0)."
+    err "  This stage will NOT report that as a pass. Either the cluster is unreachable from"
+    err "  this runner, or CLUSTER_AVAILABLE is set when it should not be."
+    err "  NOTE: the in-cluster trigger path is still UNIMPLEMENTED -- see"
+    err "  docs/plans/l2-l3-conformance-ci-wiring.md. Setting CLUSTER_AVAILABLE=true does not"
+    err "  by itself make L2/L3 run; it only makes this stage attempt a LOCAL run that the CI"
+    err "  runner has no credential for (automountServiceAccountToken: false, by design)."
+    summary "- conformance-l2-l3: **BLOCKED — declared available but every test skipped** (not a pass)"
+    exit 1
+  fi
+  log "L2/L3 executed: ${total} testcase(s), ${skipped} skipped"
+fi
+
 ok "L2/L3 conformance passed for touched subset"
 summary "- conformance-l2-l3: **passed**"

ci/gitea-settings.sh

@@ -56,14 +56,20 @@ log "Stage: Gitea platform settings match the declaration"
 DECL="${REPO_ROOT}/infra/gitea/expected-branch-protection.json"
 GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
 GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
-TOKEN="${GITEA_SETTINGS_TOKEN:-}"
+# SETTINGS_API_TOKEN, not GITEA_SETTINGS_TOKEN. Gitea RESERVES the `GITEA_` prefix for
+# secret names and rejects a PUT of one with HTTP 400 "invalid secret name" — the same way
+# GitHub reserves `GITHUB_`. The original name could therefore never be stored, and because
+# infra/gitea/set-ci-token.sh sent that PUT to /dev/null and printed "stored" unconditionally,
+# it looked like it had worked. A token was minted, no secret existed, and this stage would
+# have skipped forever. GITEA_SETTINGS_TOKEN is still honoured for a local run.
+TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
 
 [ -f "$DECL" ] || die "declaration missing: ${DECL}"
 require_tool jq
 require_tool curl
 
 if [ -z "$TOKEN" ]; then
-  skip "GITEA_SETTINGS_TOKEN is not set — cannot read the live Gitea rule."
+  skip "SETTINGS_API_TOKEN is not set — cannot read the live Gitea rule."
   skip "  Branch protection and runner registration are therefore UNVERIFIED. They were"
   skip "  wrong, unnoticed, for a month (2026-09-29). Wire it: infra/gitea/set-ci-token.sh"
   summary "- gitea-settings: **SKIPPED — no token** (protection + runners UNVERIFIED; not a pass)"

docs/plans/l2-l3-conformance-ci-wiring.md

@@ -38,8 +38,21 @@ agents. So the credential model must be decided by the operator before anything
 | Runner deployment | `apps/wave-1/gitea-actions.yaml` | act_runner + DinD (Docker executor). **`automountServiceAccountToken: false`** — the runner and its job containers hold NO SA token. Broad egress allowed (`- {}`), so it can *reach* the kube-API VIP `192.168.1.49:6443`, but has no credential. |
 | Secrets encryption | `.sops.yaml` (repo root) | age recipient `age175l8m…mfzg`; encrypts `data`/`stringData` of `*.sops.yaml` under `platform|apps|catalog/**` and `secrets/**`. Private key is the in-cluster `sops-age` Secret Flux uses. |
 
-**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the dev cluster
-is LIVE (Phase 6). So `CLUSTER_AVAILABLE` is the only remaining gate, exactly as the blueprint says.
+**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the cluster is
+LIVE (and it is now PROD; dev was decommissioned 2026-09-12).
+
+> **CORRECTION (2026-09-29): `CLUSTER_AVAILABLE` is NOT "the only remaining gate", and setting it
+> today is actively DANGEROUS.** Measured: with `CLUSTER_AVAILABLE=true` and no reachable cluster,
+> `conformance/run.sh` exits 0 in scaffold mode having marked all 54 tests `<skipped>`, and
+> `ci/conformance.sh` then printed `[ ok ] L2/L3 conformance passed for touched subset` — **rc 0, a
+> fake green**, on a merge gate that is now an enforcing required status check. The CI runner has
+> no cluster credential at all (`automountServiceAccountToken: false`, by design), so the "local
+> run" this stage attempts can never succeed; it can only skip and, before the guard added on
+> 2026-09-29, lie about it.
+>
+> `ci/conformance.sh` now BLOCKS (exit 1) when a cluster is declared but every test skipped. **Do
+> not set `CLUSTER_AVAILABLE=true` until §3's trigger path actually exists** — the variable does
+> not enable L2/L3, it only changes which way this stage fails.
 
 ---
 

infra/gitea/set-ci-token.sh

@@ -10,6 +10,10 @@
 # confirm the check goes green rather than assuming the scope name.
 set -euo pipefail
 HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
+# NOT GITEA_* — Gitea reserves that prefix for secret names and returns HTTP 400
+# "invalid secret name". The first version of this script used GITEA_SETTINGS_TOKEN, sent the
+# PUT to /dev/null, and printed "stored" regardless; the secret never existed and nobody knew.
+SECRET=SETTINGS_API_TOKEN
 NAME="ci-gitea-settings-$(date +%Y%m%d%H%M%S)"
 creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
 U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
@@ -22,12 +26,24 @@ TOKEN="$(basic -X POST -H 'Content-Type: application/json' \
 [ -n "$TOKEN" ] || { echo "token creation failed (check scopes for your Gitea version)" >&2; exit 1; }
 echo "==> got a ${#TOKEN}-character token (value not shown)"
 
-echo "==> storing it as repo Actions secret GITEA_SETTINGS_TOKEN"
-basic -X PUT -H 'Content-Type: application/json' \
+echo "==> storing it as repo Actions secret ${SECRET}"
+code="$(basic -o /tmp/set-ci-token.out -w '%{http_code}' -X PUT -H 'Content-Type: application/json' \
   -d "$(jq -n --arg d "$TOKEN" '{data:$d}')" \
-  "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets/GITEA_SETTINGS_TOKEN" >/dev/null
+  "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets/${SECRET}")"
 unset TOKEN
-echo "==> stored."
+if [ "$code" != 201 ] && [ "$code" != 204 ]; then
+  echo "   PUT failed: HTTP ${code} -- $(head -c 200 /tmp/set-ci-token.out)" >&2
+  rm -f /tmp/set-ci-token.out; exit 1
+fi
+rm -f /tmp/set-ci-token.out
+
+# VERIFY BY READING IT BACK. curl -sS exits 0 on an HTTP error, so without this the script
+# reports success for a store that never happened -- which is exactly what it did.
+names="$(basic "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets" \
+        | jq -rc '[.. | objects | select(has("name")) | .name] | unique')"
+grep -q "$SECRET" <<<"$names" \
+  || { echo "   stored HTTP ${code} but ${SECRET} is NOT in ${names} -- refusing to claim success" >&2; exit 1; }
+echo "==> stored and VERIFIED present: ${names}"
 
 cat <<'NOTE'