Merge pull request 'fix(ci): CLUSTER_AVAILABLE=true would have faked a green' (#12) from fix/cluster-available-would-fake-a-green into main

Reviewed-on: #12
Reviewed-by: Keith Jackson <[email protected]>

.gitea/workflows/gate.yml

@@ -124,8 +124,13 @@ jobs:
       - name: Merge gate — ci/gate.sh (what `make -C ci gate` runs)
         env:
           GATE_STRICT_SKIPS: "true"
-          GITEA_SETTINGS_TOKEN: ${{ secrets.GITEA_SETTINGS_TOKEN }}
-          # REQUIRE_GITEA_SETTINGS: "true"   # <- uncomment AFTER seeing gitea-settings green
+          SETTINGS_API_TOKEN: ${{ secrets.SETTINGS_API_TOKEN }}
+          # ENABLED 2026-09-29. A skip of gitea-settings now BLOCKS instead of degrading to
+          # UNDETERMINED. Safe because THIS PR's own CI run is the proof: if the stage were
+          # skipping, this line turns that into a failure and the PR goes red instead of
+          # merging. Green here means the token resolved and the live Gitea matched the
+          # declaration in infra/gitea/expected-branch-protection.json.
+          REQUIRE_GITEA_SETTINGS: "true"
         run: ./ci/gate.sh || [ "$?" -eq 3 ]
 
       # NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4

DECISIONS-LOG.md

@@ -4964,148 +4964,3 @@ account rubber-stamping the human. That is ceremony, not review. Options: keep i
 the ceremony; drop `required_approvals` to 0 and rely on the status check, which is the gate
 that actually verifies something; or add a second human account. **This is a decision for the
 operator, not a default to be chosen silently.**
-
-## 2026-09-29 (end of session) — **PM HANDOFF.** Six PRs open; read this before touching anything.
-
-The session that began "set a SOPS secret" ended having rebuilt the delivery pipeline. Plex is
-claimed and scanning. `main` is genuinely protected and the gate genuinely enforces. What is
-open is listed below with the reason each is not merged.
-
-### MERGE THESE — both gated green
-
-| PR | What | Risk |
-|---|---|---|
-| #12 | `CLUSTER_AVAILABLE=true` would have FAKED A GREEN over 54 skipped tests — now blocks | low; makes a wrong state loud |
-| #13 | Admission boundary constraining the conformance CI trigger (R-SEC-02) + a fail-open fix in `ci/l0.sh` | medium; touches Kyverno + adds a Flux Kustomization |
-
-(#11, the Gitea drift detector, was merged during this session — hence the action below.)
-
-Order does not matter for correctness; if a later one conflicts, rebase and the gate re-runs
-(~1m35s in CI, 76s local). **Each was gated in the pinned toolbox, not on a host.**
-
-### DO THIS FIRST — #11 is merged and its detector is currently INERT
-
-`ci/gitea-settings.sh` SKIPS until its token exists, and **a skipped detector detects nothing**:
-
-    ./infra/gitea/set-ci-token.sh          # mints + stores GITEA_SETTINGS_TOKEN, never prints it
-
-Then open any PR and confirm the gate prints `gitea-settings: green`. ONLY THEN uncomment
-`REQUIRE_GITEA_SETTINGS: "true"` in `.gitea/workflows/gate.yml`, so a future missing token blocks
-instead of skipping. Doing that before seeing it green locks the repo.
-
-Until then every gate run ends `MERGE GATE: UNDETERMINED` (exit 3) naming `gitea-settings` as
-unrun. That is BY DESIGN and `gate.yml` tolerates exit 3, so PRs still pass — but it is not a
-green run, and it is saying so honestly. The token is what turns it green.
-
-### A DIFFERENT COMPUTER NEEDS THREE THINGS THAT ARE NOT IN GIT
-
-This is the part that will silently waste an hour otherwise. None of these are in the repo, by
-design, and without them the toolbox and the cluster are unreachable:
-
-1. **`~/.config/sops/age/keys.txt`** — the age PRIVATE key. Without it NO `*.sops.yaml` can be
-   read or edited, and `sops` will fail in a way that looks like a broken file. Copy it from this
-   workstation; the public recipient in `.sops.yaml` is only half the pair.
-2. **`infra/ansible/.cache/prod.kubeconfig`** — untracked, so it exists only in the main checkout
-   on this machine. Without it `dev.sh` refuses to start. In a git WORKTREE it is absent even
-   here: use `DEV_KUBECONFIG=/dev/null` for anything that needs no cluster (see `CLAUDE.md` §3).
-3. **A Gitea credential** for `gitea.home.keithjackson.dev` (this session used the stored
-   `~/.git-credentials`). `git push` and every `infra/gitea/*.sh` script needs it.
-
-### STATE, MEASURED — do not re-derive it from this log, re-measure it
-
-`CLAUDE.md` §2b exists because this log was confidently wrong for a month. Verify before relying:
-
-    protected:true  user_can_push:false  enable_status_check:true
-    status_check_contexts:["gate / gate (pull_request)"]  required_approvals:1
-    block_admin_merge_override:true      runner incluster-dind: online      gate ~1m35s
-
-### NEXT WORK, in priority order
-
-1. **Finish the L2/L3 wiring** now that the credential model is decided and #13 lands the
-   boundary. Still missing (`docs/plans/l2-l3-conformance-ci-wiring.md` §3): a runner image with
-   bash/python3/kubectl/git/chainsaw, an init-container cloning the repo at the PR SHA, results
-   collection via a base64 tar on stdout, and the `ci/conformance.sh` rewrite to drive the Job.
-   **Tighten the admission policy's image pattern to the literal digest when that image exists.**
-2. **Do NOT set `vars.CLUSTER_AVAILABLE=true`** until step 1 is done. It does not enable L2/L3; it
-   only changes which way the stage fails (see #12).
-3. The `gitea-settings` detector covers branch protection + runners. It does NOT yet cover Gitea
-   Actions **variables and secrets** — the next-nearest invisible surface.
-4. `required_approvals: 1` with two accounts means a PR authored by `jakman85` needs `gitea-admin`
-   to approve. The operator holds ultimate authority and does not require that review; revisit if
-   it becomes friction.
-
-### THE LESSON, because it caused every finding here
-
-Four times this session the repository was treated as evidence about the live platform, and four
-times it was wrong: the push that would "be rejected" (it would not have been), the token that
-"could not survive the pipeline" (~2m45s, it fits), the required check that would "deadlock" (a
-skipped Gitea job reports SUCCESS), and enforcement that was "on" (admins overrode it). Two
-fail-opens in the gate itself were found the same way — `kyverno test` certifying policies that do
-not load, and `CLUSTER_AVAILABLE` certifying 54 skipped tests as passed.
-
-**Measure, then assert. `CLAUDE.md` §2b is the rule; this entry is the evidence for it.**
-
-## 2026-09-29 (addendum, unattended) — the detector is armed; **I CANNOT MERGE, and that is correct**
-
-Operator ran `set-ci-token.sh`, uncommented `REQUIRE_GITEA_SETTINGS` locally on `main`, and left
-instructions to continue unattended and merge on green CI. Two things came out of that.
-
-### 1. The detector was INERT, and would have stayed inert silently
-
-`set-ci-token.sh` PUT the secret as `GITEA_SETTINGS_TOKEN`. **Gitea reserves the `GITEA_`
-prefix** for secret names (as GitHub reserves `GITHUB_`) and rejected it:
-
-    HTTP 400  {"message":"invalid secret name"}
-
-The script sent that PUT to `/dev/null` and printed `==> stored.` unconditionally; `curl -sS`
-exits 0 on an HTTP error. **A token existed; no secret did.** So `ci/gitea-settings.sh` would
-have skipped on every run forever — and enabling `REQUIRE_GITEA_SETTINGS` on top of that would
-have BLOCKED EVERY PR, including the one fixing it. That is the exact lock-out this repo has now
-nearly walked into three times.
-
-Fixed in **PR #15**: the secret is `SETTINGS_API_TOKEN`, and `set-ci-token.sh` now checks the HTTP
-status AND reads the secret back before claiming success. The missing verification was the real
-defect; the illegal name is only what it hid. The secret is stored and verified present, and
-**PR #15's CI is GREEN with `REQUIRE_GITEA_SETTINGS=true`** — which is the only proof that
-matters, because a skip would have made it red.
-
-The operator's local edit could not be pushed to `main` (protected). It is carried in PR #15.
-
-### 2. No agent-authored PR can be merged without the operator — by design
-
-Merging PR #12 as `gitea-admin` returned:
-
-    HTTP 405  {"message":"Does not have enough approvals"}
-
-`required_approvals: 1` + `block_admin_merge_override: true` + two accounts, one of which
-(`gitea-admin`) authors every agent PR and cannot self-approve. **So "merge on green CI" is not
-something an agent can execute here.** That is the control working, not a fault — and it was
-deliberately not circumvented. Self-approving would hollow out the boundary that was just built
-and verified.
-
-**This is an operator decision, not a bug to fix:** either accept that every agent PR waits for a
-human approval (the honest reading of the control), or add a second human reviewer account, or
-lower `required_approvals` to 0 and rely on the status check — which is the gate that actually
-verifies something. Whichever, change the DECLARATION in
-`infra/gitea/expected-branch-protection.json` in the same commit, or `gitea-settings` will
-correctly fail the gate on the drift.
-
-### Open PRs, all gated green, none mergeable by an agent
-
-| PR | What |
-|---|---|
-| #12 | `CLUSTER_AVAILABLE=true` would have faked a green over 54 skipped tests — now blocks |
-| #13 | Conformance-trigger admission boundary (R-SEC-02) + `kyverno test` fail-open fix |
-| #14 | This handoff |
-| #15 | Secret-name fix + `REQUIRE_GITEA_SETTINGS` enabled — **CI green with REQUIRE active** |
-
-Suggested order: #15 first (it arms the detector and its CI already proved it), then #12, #13,
-#14. If a later one conflicts, rebase; the gate re-runs in ~1m36s.
-
-### Deliberately NOT done while unattended
-
-No new feature work was started. Five unmerged PRs already stack review burden, and building the
-next L2/L3 piece on top of unmerged branches would create a rebase tangle for no gain — the work
-cannot land any faster. The next piece is specified and ready to start once these are merged:
-`docs/plans/l2-l3-conformance-ci-wiring.md` §3, with §4's credential model now decided and its
-boundary landed in #13.

ci/conformance.sh

@@ -61,5 +61,45 @@ if [ -d "${CONF_ART}/coverage" ]; then
   cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
 fi
 
+# ---------------------------------------------------------------------------------------
+# DID IT ACTUALLY RUN? -- the guard without which this stage FAKES A GREEN.
+#
+# run.sh exits 0 in "scaffold mode" when it cannot reach a cluster: it emits a JUnit file
+# whose every testcase is <skipped> and returns success. That is correct FOR run.sh -- an
+# honest skip is its documented no-cluster behaviour. It is catastrophic HERE, because
+# reaching this point means the environment DECLARED CLUSTER_AVAILABLE=true, and this stage
+# would otherwise print "L2/L3 conformance passed" over 54 tests that never executed.
+#
+# MEASURED 2026-09-29, before this guard existed:
+#   CLUSTER_AVAILABLE=true ./ci/conformance.sh   ->  rc=0
+#   "[junit] wrote skipped JUnit for 54 scaffolded test(s)"
+#   "[ ok ] L2/L3 conformance passed for touched subset"
+# One Gitea variable away from a fake green on an enforcing merge gate. That is exactly the
+# HARD RULE 4 violation this repo's whole gate design exists to prevent, and nothing caught
+# it because nobody had ever set the variable.
+#
+# A skip while a cluster is DECLARED is a contradiction: either the declaration is wrong or
+# the cluster is unreachable. The merge-gate decision matrix already says this case is BLOCK
+# ("conformance skipped while CLUSTER_AVAILABLE=true -> BLOCK (it should have run)"). This
+# makes the stage itself honour that instead of reporting a pass.
+JUNIT="${CONF_ART}/report.junit.xml"
+if [ -f "$JUNIT" ]; then
+  total="$(grep -o '<testcase' "$JUNIT" | wc -l | tr -d ' ')"
+  skipped="$(grep -o '<skipped' "$JUNIT" | wc -l | tr -d ' ')"
+  if [ "${total:-0}" -gt 0 ] && [ "${skipped:-0}" -ge "${total:-0}" ]; then
+    err "CLUSTER_AVAILABLE=true, but the L2/L3 suite SKIPPED every one of its ${total} tests"
+    err "  (scaffold mode -- run.sh could not reach a cluster and exited 0)."
+    err "  This stage will NOT report that as a pass. Either the cluster is unreachable from"
+    err "  this runner, or CLUSTER_AVAILABLE is set when it should not be."
+    err "  NOTE: the in-cluster trigger path is still UNIMPLEMENTED -- see"
+    err "  docs/plans/l2-l3-conformance-ci-wiring.md. Setting CLUSTER_AVAILABLE=true does not"
+    err "  by itself make L2/L3 run; it only makes this stage attempt a LOCAL run that the CI"
+    err "  runner has no credential for (automountServiceAccountToken: false, by design)."
+    summary "- conformance-l2-l3: **BLOCKED — declared available but every test skipped** (not a pass)"
+    exit 1
+  fi
+  log "L2/L3 executed: ${total} testcase(s), ${skipped} skipped"
+fi
+
 ok "L2/L3 conformance passed for touched subset"
 summary "- conformance-l2-l3: **passed**"

ci/gitea-settings.sh

@@ -56,14 +56,20 @@ log "Stage: Gitea platform settings match the declaration"
 DECL="${REPO_ROOT}/infra/gitea/expected-branch-protection.json"
 GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
 GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
-TOKEN="${GITEA_SETTINGS_TOKEN:-}"
+# SETTINGS_API_TOKEN, not GITEA_SETTINGS_TOKEN. Gitea RESERVES the `GITEA_` prefix for
+# secret names and rejects a PUT of one with HTTP 400 "invalid secret name" — the same way
+# GitHub reserves `GITHUB_`. The original name could therefore never be stored, and because
+# infra/gitea/set-ci-token.sh sent that PUT to /dev/null and printed "stored" unconditionally,
+# it looked like it had worked. A token was minted, no secret existed, and this stage would
+# have skipped forever. GITEA_SETTINGS_TOKEN is still honoured for a local run.
+TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
 
 [ -f "$DECL" ] || die "declaration missing: ${DECL}"
 require_tool jq
 require_tool curl
 
 if [ -z "$TOKEN" ]; then
-  skip "GITEA_SETTINGS_TOKEN is not set — cannot read the live Gitea rule."
+  skip "SETTINGS_API_TOKEN is not set — cannot read the live Gitea rule."
   skip "  Branch protection and runner registration are therefore UNVERIFIED. They were"
   skip "  wrong, unnoticed, for a month (2026-09-29). Wire it: infra/gitea/set-ci-token.sh"
   summary "- gitea-settings: **SKIPPED — no token** (protection + runners UNVERIFIED; not a pass)"

docs/plans/l2-l3-conformance-ci-wiring.md

@@ -38,8 +38,21 @@ agents. So the credential model must be decided by the operator before anything
 | Runner deployment | `apps/wave-1/gitea-actions.yaml` | act_runner + DinD (Docker executor). **`automountServiceAccountToken: false`** — the runner and its job containers hold NO SA token. Broad egress allowed (`- {}`), so it can *reach* the kube-API VIP `192.168.1.49:6443`, but has no credential. |
 | Secrets encryption | `.sops.yaml` (repo root) | age recipient `age175l8m…mfzg`; encrypts `data`/`stringData` of `*.sops.yaml` under `platform|apps|catalog/**` and `secrets/**`. Private key is the in-cluster `sops-age` Secret Flux uses. |
 
-**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the dev cluster
-is LIVE (Phase 6). So `CLUSTER_AVAILABLE` is the only remaining gate, exactly as the blueprint says.
+**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the cluster is
+LIVE (and it is now PROD; dev was decommissioned 2026-09-12).
+
+> **CORRECTION (2026-09-29): `CLUSTER_AVAILABLE` is NOT "the only remaining gate", and setting it
+> today is actively DANGEROUS.** Measured: with `CLUSTER_AVAILABLE=true` and no reachable cluster,
+> `conformance/run.sh` exits 0 in scaffold mode having marked all 54 tests `<skipped>`, and
+> `ci/conformance.sh` then printed `[ ok ] L2/L3 conformance passed for touched subset` — **rc 0, a
+> fake green**, on a merge gate that is now an enforcing required status check. The CI runner has
+> no cluster credential at all (`automountServiceAccountToken: false`, by design), so the "local
+> run" this stage attempts can never succeed; it can only skip and, before the guard added on
+> 2026-09-29, lie about it.
+>
+> `ci/conformance.sh` now BLOCKS (exit 1) when a cluster is declared but every test skipped. **Do
+> not set `CLUSTER_AVAILABLE=true` until §3's trigger path actually exists** — the variable does
+> not enable L2/L3, it only changes which way this stage fails.
 
 ---
 

infra/gitea/set-ci-token.sh

@@ -10,6 +10,10 @@
 # confirm the check goes green rather than assuming the scope name.
 set -euo pipefail
 HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
+# NOT GITEA_* — Gitea reserves that prefix for secret names and returns HTTP 400
+# "invalid secret name". The first version of this script used GITEA_SETTINGS_TOKEN, sent the
+# PUT to /dev/null, and printed "stored" regardless; the secret never existed and nobody knew.
+SECRET=SETTINGS_API_TOKEN
 NAME="ci-gitea-settings-$(date +%Y%m%d%H%M%S)"
 creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
 U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
@@ -22,12 +26,24 @@ TOKEN="$(basic -X POST -H 'Content-Type: application/json' \
 [ -n "$TOKEN" ] || { echo "token creation failed (check scopes for your Gitea version)" >&2; exit 1; }
 echo "==> got a ${#TOKEN}-character token (value not shown)"
 
-echo "==> storing it as repo Actions secret GITEA_SETTINGS_TOKEN"
-basic -X PUT -H 'Content-Type: application/json' \
+echo "==> storing it as repo Actions secret ${SECRET}"
+code="$(basic -o /tmp/set-ci-token.out -w '%{http_code}' -X PUT -H 'Content-Type: application/json' \
   -d "$(jq -n --arg d "$TOKEN" '{data:$d}')" \
-  "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets/GITEA_SETTINGS_TOKEN" >/dev/null
+  "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets/${SECRET}")"
 unset TOKEN
-echo "==> stored."
+if [ "$code" != 201 ] && [ "$code" != 204 ]; then
+  echo "   PUT failed: HTTP ${code} -- $(head -c 200 /tmp/set-ci-token.out)" >&2
+  rm -f /tmp/set-ci-token.out; exit 1
+fi
+rm -f /tmp/set-ci-token.out
+
+# VERIFY BY READING IT BACK. curl -sS exits 0 on an HTTP error, so without this the script
+# reports success for a store that never happened -- which is exactly what it did.
+names="$(basic "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets" \
+        | jq -rc '[.. | objects | select(has("name")) | .name] | unique')"
+grep -q "$SECRET" <<<"$names" \
+  || { echo "   stored HTTP ${code} but ${SECRET} is NOT in ${names} -- refusing to claim success" >&2; exit 1; }
+echo "==> stored and VERIFIED present: ${names}"
 
 cat <<'NOTE'