jakman85andClaude Opus 5
b52c8d1af6
The override did not work on its first real test, and the reason was invisible from the outside. Run 203 on PR #38: a well-formed `/approve-override` posted by jakman85 in the PR comment box, 29 minutes newer than the head commit, and the stage still blocked with: approval-override: COMMENTS_JSON is not a list SETTINGS_API_TOKEN is minted [read:repository, read:admin], and Gitea puts /issues/{n}/comments behind the *issue* scope — so Gitea answered with an error OBJECT, `jq -e .` accepted it as valid JSON, and the stage fell through to 'no approving review'. A permission failure wearing the costume of a missing override. /pulls/{n}/files and /pulls/{n}/reviews are under read:repository, which is why every other part of the same stage worked. 1. The diagnosis no longer lies. COMMENTS_JSON leaves the must-parse-or-block loop and is checked for being an ARRAY; when it is not, the stage prints Gitea's own .message, names the missing scope and says where to post instead. An unreadable source can only cause a false BLOCK, never a false pass, so a loud warning is the honest behaviour and blocking was not. 2. An override is accepted from a PR REVIEW body as well as a comment, so the hatch is not hostage to one API scope — a review of type 'Comment' works on the token CI has today. The gate names which source it used; PENDING (unsubmitted, author-only) reviews are dropped. 3. set-ci-token.sh requests read:issue, making the ordinary comment box work — on a RE-RUN only, since an existing token's scopes cannot be widened. ci/tests/approval-override.test.sh: 23 -> 38 assertions, covering the reviews source, PENDING reviews, an error object from the comments API, 'an unreadable comments source must not suppress a valid review override', latest-wins across both sources, and that the diagnostic quotes Gitea's message. Also measured and recorded: run 203 shows gitea-settings PASS after the operator ran the live PATCH, with block_admin_merge_override still true — the narrow change held, the approval count moved and the gate's teeth did not. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
233 lines
12 KiB
Bash
Executable File
233 lines
12 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# ci/tests/approval-override.test.sh — proves ci/approval-override.py selects the override it
|
|
# claims to select, and REJECTS the ones it claims to reject.
|
|
#
|
|
# This parser decides whether a human review requirement gets waived. Every rule in it exists
|
|
# because the obvious implementation ("does the body contain the directive?") is wrong in a way
|
|
# that is invisible until abused: a quoted directive would re-arm an override, a bare directive
|
|
# would waive review with no stated reason, and an override posted against commit A would go on
|
|
# covering commit B forever. A guard is worth nothing until somebody has watched it refuse, which
|
|
# is the same reasoning as ci/tests/decisions-log.test.sh one layer up.
|
|
#
|
|
# The parser is pure and offline BY DESIGN — authorisation (the live permission lookup) lives in
|
|
# ci/approval-policy.sh, which cannot be tested without Gitea. That split is what makes this file
|
|
# possible, and it is also why a green run here does NOT prove the stage authorises correctly.
|
|
set -uo pipefail
|
|
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
|
SUT="${CI_DIR}/approval-override.py"
|
|
|
|
PASS=0 FAIL=0
|
|
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
|
|
bad() { printf ' FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
|
|
|
|
HEAD_TIME="2026-10-01T12:00:00Z"
|
|
AFTER="2026-10-01T12:30:00Z"
|
|
BEFORE="2026-10-01T11:30:00Z"
|
|
|
|
# run <comments-json> -> stdout of the parser (stderr is diagnostic only)
|
|
run() {
|
|
COMMENTS_JSON="$1" REVIEWS_JSON="" HEAD_TIME="${2:-$HEAD_TIME}" DIRECTIVE="${3:-/approve-override}" \
|
|
python3 "$SUT" 2>/dev/null
|
|
}
|
|
|
|
# run_rev <reviews-json> — the same, but through the REVIEWS source.
|
|
run_rev() {
|
|
COMMENTS_JSON="" REVIEWS_JSON="$1" HEAD_TIME="${2:-$HEAD_TIME}" DIRECTIVE="${3:-/approve-override}" \
|
|
python3 "$SUT" 2>/dev/null
|
|
}
|
|
|
|
# comment <login> <created_at> <body>
|
|
comment() {
|
|
python3 -c '
|
|
import json, sys
|
|
print(json.dumps([{"user": {"login": sys.argv[1]}, "created_at": sys.argv[2], "body": sys.argv[3]}]))
|
|
' "$1" "$2" "$3"
|
|
}
|
|
|
|
expect_none() {
|
|
local label="$1" out
|
|
out="$(run "$2")"
|
|
if [ -z "$out" ]; then ok "$label"; else bad "$label (got: ${out})"; fi
|
|
}
|
|
|
|
expect_user() {
|
|
local label="$1" json="$2" want="$3" out got
|
|
out="$(run "$json")"
|
|
got="$(cut -f1 <<<"$out")"
|
|
if [ "$got" = "$want" ]; then ok "$label"; else bad "$label (wanted ${want}, got '${got}')"; fi
|
|
}
|
|
|
|
expect_reason() {
|
|
local label="$1" json="$2" want="$3" out got
|
|
out="$(run "$json")"
|
|
got="$(cut -f4- <<<"$out")"
|
|
if [ "$got" = "$want" ]; then ok "$label"; else bad "$label (wanted '${want}', got '${got}')"; fi
|
|
}
|
|
|
|
# check <label> <actual> <wanted>
|
|
check() {
|
|
if [ "$2" = "$3" ]; then ok "$1"; else bad "$1 (wanted '${3}', got '${2}')"; fi
|
|
}
|
|
|
|
# review <login> <submitted_at> <state> <body>
|
|
review() {
|
|
python3 -c '
|
|
import json, sys
|
|
print(json.dumps([{"user": {"login": sys.argv[1]}, "submitted_at": sys.argv[2],
|
|
"state": sys.argv[3], "body": sys.argv[4]}]))
|
|
' "$1" "$2" "$3" "$4"
|
|
}
|
|
|
|
echo "== ci/approval-override.py"
|
|
|
|
# --- the happy path -------------------------------------------------------------------
|
|
expect_user "a directive with a reason, posted after the head commit, is selected" \
|
|
"$(comment jakman85 "$AFTER" '/approve-override sole admin, gate is green')" jakman85
|
|
|
|
expect_reason "the reason is returned verbatim" \
|
|
"$(comment jakman85 "$AFTER" '/approve-override sole admin, gate is green')" \
|
|
'sole admin, gate is green'
|
|
|
|
# --- absence is a normal answer, not an error -----------------------------------------
|
|
expect_none "no comments at all" '[]'
|
|
expect_none "an ordinary comment" "$(comment jakman85 "$AFTER" 'looks good to me')"
|
|
|
|
# --- a reason is MANDATORY ------------------------------------------------------------
|
|
expect_none "a bare directive (no reason) is refused" \
|
|
"$(comment jakman85 "$AFTER" '/approve-override')"
|
|
expect_none "a directive followed by only whitespace is refused" \
|
|
"$(comment jakman85 "$AFTER" '/approve-override ')"
|
|
|
|
# --- the directive must START a line --------------------------------------------------
|
|
expect_none "the directive mid-sentence does not count" \
|
|
"$(comment jakman85 "$AFTER" 'you could just /approve-override this one')"
|
|
expect_none "a QUOTED directive does not re-arm an override" \
|
|
"$(comment jakman85 "$AFTER" '> /approve-override quoting someone else')"
|
|
expect_user "leading whitespace before the directive is fine" \
|
|
"$(comment jakman85 "$AFTER" ' /approve-override indented but mine')" jakman85
|
|
expect_user "the directive on a later line of a longer comment counts" \
|
|
"$(comment jakman85 "$AFTER" 'some preamble
|
|
/approve-override the real reason here')" jakman85
|
|
|
|
# --- staleness: an override must post-date the commit it waives review on -------------
|
|
expect_none "an override OLDER than the head commit is dropped as stale" \
|
|
"$(comment jakman85 "$BEFORE" '/approve-override stale, predates the push')"
|
|
expect_none "an override exactly AT the head commit time is dropped (strictly newer)" \
|
|
"$(comment jakman85 "$HEAD_TIME" '/approve-override same instant')"
|
|
|
|
# --- malformed input is never a free pass --------------------------------------------
|
|
expect_none "an unreadable timestamp drops the candidate" \
|
|
"$(comment jakman85 'not-a-date' '/approve-override bad clock')"
|
|
expect_none "a comment with no author is dropped" \
|
|
'[{"user": {}, "created_at": "2026-10-01T12:30:00Z", "body": "/approve-override no login"}]'
|
|
expect_none "invalid JSON yields no override (and never a traceback into stdout)" \
|
|
'this is not json'
|
|
expect_none "a JSON object instead of a list yields no override" \
|
|
'{"user": "nope"}'
|
|
|
|
# --- latest wins ----------------------------------------------------------------------
|
|
TWO='[
|
|
{"user": {"login": "early"}, "created_at": "2026-10-01T12:10:00Z", "body": "/approve-override first"},
|
|
{"user": {"login": "latest"}, "created_at": "2026-10-01T12:40:00Z", "body": "/approve-override second"}
|
|
]'
|
|
expect_user "the LATEST override supersedes an earlier one" "$TWO" latest
|
|
expect_reason "and brings its own reason" "$TWO" second
|
|
|
|
# --- offsets, not just Z --------------------------------------------------------------
|
|
# Gitea emits RFC3339 WITH AN OFFSET (measured: 2026-10-01T23:09:08-05:00), not a trailing Z.
|
|
# A parser that only handles 'Z' silently drops every real override, so this case is load-bearing.
|
|
expect_user "an RFC3339 offset timestamp (-05:00), as Gitea actually emits, parses" \
|
|
'[{"user": {"login": "jakman85"}, "created_at": "2026-10-01T08:30:00-05:00", "body": "/approve-override offset form"}]' \
|
|
jakman85
|
|
expect_none "and an offset timestamp that is genuinely earlier is still stale" \
|
|
'[{"user": {"login": "jakman85"}, "created_at": "2026-10-01T06:00:00-05:00", "body": "/approve-override offset, stale"}]'
|
|
|
|
# --- the directive is configurable ----------------------------------------------------
|
|
out="$(run "$(comment jakman85 "$AFTER" '/admin-waive because policy says so')" "$HEAD_TIME" '/admin-waive')"
|
|
if [ "$(cut -f1 <<<"$out")" = "jakman85" ]; then
|
|
ok "a custom directive from the declaration is honoured"
|
|
else
|
|
bad "a custom directive from the declaration is honoured (got '${out}')"
|
|
fi
|
|
expect_none "the default directive does not match a custom-directive comment" \
|
|
"$(comment jakman85 "$AFTER" '/admin-waive because policy says so')"
|
|
|
|
# --- no staleness anchor: do not invent one -------------------------------------------
|
|
out="$(COMMENTS_JSON="$(comment jakman85 "$BEFORE" '/approve-override no anchor')" \
|
|
HEAD_TIME="" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
|
|
if [ "$(cut -f1 <<<"$out")" = "jakman85" ]; then
|
|
ok "with no HEAD_TIME the staleness test is skipped, not faked (caller blocks instead)"
|
|
else
|
|
bad "with no HEAD_TIME the candidate should still be returned (got '${out}')"
|
|
fi
|
|
|
|
# --- the REVIEWS source ---------------------------------------------------------------
|
|
# Load-bearing, not redundant. Measured on PR #38: SETTINGS_API_TOKEN is minted
|
|
# [read:repository, read:admin], /issues/{n}/comments needs the ISSUE scope, so a well-formed
|
|
# override in the comment box came back as an error object and was invisible. /pulls/{n}/reviews
|
|
# IS readable with that token, so a review body is the route that works today.
|
|
check "a review body carries an override" \
|
|
"$(cut -f1 <<<"$(run_rev "$(review jakman85 "$AFTER" COMMENT '/approve-override via a review')")")" \
|
|
jakman85
|
|
check "the source is reported as 'review'" \
|
|
"$(cut -f3 <<<"$(run_rev "$(review jakman85 "$AFTER" COMMENT '/approve-override via a review')")")" \
|
|
review
|
|
check "the source is reported as 'comment' for a comment" \
|
|
"$(cut -f3 <<<"$(run "$(comment jakman85 "$AFTER" '/approve-override via a comment')")")" \
|
|
comment
|
|
|
|
out="$(run_rev "$(review jakman85 "$BEFORE" COMMENT '/approve-override stale review')")"
|
|
check "a stale review override is dropped too" "${out:-<none>}" "<none>"
|
|
|
|
out="$(run_rev "$(review jakman85 "$AFTER" PENDING '/approve-override unsubmitted draft')")"
|
|
check "a PENDING (unsubmitted) review is NOT an act of anything" "${out:-<none>}" "<none>"
|
|
|
|
check "an APPROVED review carrying the directive still counts as an override" \
|
|
"$(cut -f1 <<<"$(run_rev "$(review jakman85 "$AFTER" APPROVED '/approve-override belt and braces')")")" \
|
|
jakman85
|
|
|
|
out="$(run_rev "$(review jakman85 "$AFTER" COMMENT 'ordinary review prose, no directive')")"
|
|
check "an ordinary review body is not an override" "${out:-<none>}" "<none>"
|
|
|
|
# --- an unreadable source degrades to 'no candidates', never to a pass ----------------
|
|
# The real-world shape: Gitea answers with {"message": "token does not have ... scope"}.
|
|
ERR_OBJ='{"message":"token does not have at least one of required scope(s): [read:issue]"}'
|
|
out="$(COMMENTS_JSON="$ERR_OBJ" REVIEWS_JSON="" HEAD_TIME="$HEAD_TIME" \
|
|
DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
|
|
check "an error OBJECT from the comments API yields no override" "${out:-<none>}" "<none>"
|
|
|
|
out="$(COMMENTS_JSON="$ERR_OBJ" \
|
|
REVIEWS_JSON="$(review jakman85 "$AFTER" COMMENT '/approve-override reviews still work')" \
|
|
HEAD_TIME="$HEAD_TIME" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
|
|
check "an unreadable COMMENTS source does not suppress a valid REVIEW override" \
|
|
"$(cut -f1 <<<"$out")" jakman85
|
|
|
|
# and it must SAY so, or a scope problem reads as 'you never posted one' (it did, on PR #38)
|
|
errout="$(COMMENTS_JSON="$ERR_OBJ" REVIEWS_JSON="" HEAD_TIME="$HEAD_TIME" \
|
|
DIRECTIVE=/approve-override python3 "$SUT" 2>&1 >/dev/null)"
|
|
case "$errout" in
|
|
*"not a list"*read:issue*) ok "the diagnostic names the scope Gitea complained about" ;;
|
|
*) bad "the diagnostic should quote Gitea's message (got: ${errout})" ;;
|
|
esac
|
|
|
|
# --- latest wins ACROSS the two sources ----------------------------------------------
|
|
out="$(COMMENTS_JSON="$(comment early "2026-10-01T12:10:00Z" '/approve-override from a comment')" \
|
|
REVIEWS_JSON="$(review later "2026-10-01T12:50:00Z" COMMENT '/approve-override from a review')" \
|
|
HEAD_TIME="$HEAD_TIME" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
|
|
check "the latest override wins across both sources (review newer)" "$(cut -f1 <<<"$out")" later
|
|
check "and its source is labelled" "$(cut -f3 <<<"$out")" review
|
|
|
|
out="$(COMMENTS_JSON="$(comment later "2026-10-01T12:50:00Z" '/approve-override from a comment')" \
|
|
REVIEWS_JSON="$(review early "2026-10-01T12:10:00Z" COMMENT '/approve-override from a review')" \
|
|
HEAD_TIME="$HEAD_TIME" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
|
|
check "the latest override wins across both sources (comment newer)" "$(cut -f1 <<<"$out")" later
|
|
check "and that source is labelled too" "$(cut -f3 <<<"$out")" comment
|
|
|
|
# --- both sources absent entirely -----------------------------------------------------
|
|
out="$(COMMENTS_JSON="" REVIEWS_JSON="" HEAD_TIME="$HEAD_TIME" \
|
|
DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
|
|
check "no sources at all yields no override, and no crash" "${out:-<none>}" "<none>"
|
|
|
|
printf '\n %d passed, %d failed\n' "$PASS" "$FAIL"
|
|
[ "$FAIL" -eq 0 ] || exit 1
|