home-network-k3s-impl/decisions/2026-10-01-admin-approval-override.md

date, branch, title

date	branch	title
2026-10-01	fix/admin-approval-override	Administrators can merge their own green PR — required_approvals 1 → 0, plus an attributed /approve-override for sensitive paths

What changed

The problem, measured first. required_approvals: 1 on main was a lock-out, not a protection. Gitea refuses to let a pull request's author approve their own PR, and jakman85 is the only collaborator (permission owner). So no account in existence could approve a PR the operator opened. Measured on the live API, 2026-10-01:

PR #35 (bugfix/arr-to-plex, author jakman85)	gate / gate (pull_request) = success, 1m41s
its reviews	[]
live required_approvals	1
live block_admin_merge_override	true

Green gate, zero reviews, no path to a merge — and the same was true of any PR that would have fixed it.

It was also never the design this repo wrote down. ci/approval-policy.sh's own header and ci/gate.sh:194 both already described the model as "required_approvals drops to 0 and the GATE decides", and infra/gitea/require-gate.sh's closing note named this exact failure and prescribed this exact remedy. decisions/2026-09-30-flux-deploy-notification-inhibited.md had flagged the 1-vs-0 disagreement as unresolved and not that session's to settle. This settles it at 0.

Two layers were blocking, and both are addressed.

Layer 1 — Gitea branch protection. required_approvals: 1 → 0 in infra/gitea/expected-branch-protection.json, with the rationale in the file, plus infra/gitea/allow-admin-self-merge.sh to apply it to the live rule (it reads the target from the declaration, so the script cannot drift from what the gate checks). require-gate.sh was changed to 0 as well — left at 1 it would silently re-impose the lock-out on its next run.

The narrow choice matters: the other way to unblock an owner is block_admin_merge_override: false, and that would have re-opened precisely what infra/gitea/block-admin-override.sh was written to close (PRs #5–#9 merged with zero approvals while the settings read as enforced). This change drops the approval count only. block_admin_merge_override and enable_status_check both stay true, and the script asserts both afterwards and fails loudly if the PATCH disturbed either. A red gate remains unmergeable by everyone, administrators included.

Layer 2 — ci/approval-policy.sh. Sensitive paths (ci/**, infra/gitea/**, CLAUDE.md, kyverno, rbac, sops, Flux wiring) still require a non-author APPROVED review, which the sole administrator still cannot obtain. An administrator may now waive it by commenting on the PR:

/approve-override <reason>

• the directive must start a line (a quoted or mid-sentence mention does not count, or repeating someone else's override would re-arm it);
• a reason is mandatory — a bare directive records that review was waived but not why;
• the commenter's permission is re-checked live against /collaborators/{login}/permission when the gate runs, so an old comment from a demoted account carries nothing;
• it is invalidated by a new commit (the comment must post-date the head commit), mirroring Gitea's own dismiss_stale_approvals;
• the overrider, timestamp and reason are printed in the gate log and the job summary, and a self-waiver by the PR's own author is called out as such.

Selection of the candidate comment is ci/approval-override.py — pure, offline, no authority of its own — so it could be unit-tested; authorisation stays in the fail-closed stage. ci/tests/approval-override.test.sh adds 23 assertions, all passing, each constructing a case the parser must refuse. Every new failure mode blocks: unreadable head-commit timestamp, unreadable permission, parser error.

The cost, stated rather than buried. For an administrator, "a change to the gate cannot be gated by the gate" is now advisory, not structural. That is unavoidable with one human — the alternative is a repository its owner cannot change — and what is bought back is an attributed, reasoned, commit-scoped record of each waiver instead of a silent bypass.

The uncomfortable part, accepted with eyes open (operator decision). Authorisation is by live permission alone, with no allowlist of human accounts. The agent account gitea-admin also reports permission owner on this repo (measured), so an agent can clear the sensitive-path review on its own PR — and agents author most PRs here, so that is the common case, not an edge case. I raised this before implementing; the operator chose permission-only. It is documented in three places rather than left to be discovered: approval-policy.json (_known_consequence_agent_accounts), the stage's header, and CLAUDE.md, which now tells agents plainly not to use it on their own work. The narrowing path needs no code change — add an override_admins list to approval-policy.json and the stage already requires membership in addition to the permission check.

Also updated: CLAUDE.md §1 and the §2 enforcement table (the "approval required" row is now explicitly ⬜ with the reason, rather than a stale ✅), and infra/gitea/README.md gains §6 and marks its historical step 4 as reversed — do not re-apply.

Second pass, 2026-10-02 — the override did not work the first time, and why. The operator ran the live PATCH (required_approvals: 0 confirmed live, with block_admin_merge_override and enable_status_check still true), merged PR #35, and posted /approve-override Need to ensure approval override work as expected on PR #38. The gate re-ran (run 203) and still blocked.

The comment was not at fault: posted by jakman85, directive at the start of the line, reason present, and 29 minutes newer than head commit 616a497, so it cleared the staleness rule. What failed was that CI could not read it. Run 203's log:

approval-override: COMMENTS_JSON is not a list

SETTINGS_API_TOKEN is minted ["read:repository", "read:admin"] by infra/gitea/set-ci-token.sh, and Gitea puts /issues/{n}/comments behind the issue scope. So Gitea returned an error object, jq -e . accepted it as valid JSON, and the stage fell through to "no approving review" — a permission failure wearing the costume of a missing override. /pulls/{n}/files and /pulls/{n}/reviews are under read:repository, which is exactly why every other part of the same stage worked and this one did not.

Three fixes, and the first is the one that mattered most:

1. The diagnosis is no longer misleading. COMMENTS_JSON is out of the must-parse-or-block loop and is explicitly checked for being an array; when it is not, the stage prints Gitea's own .message, names the missing scope, and says where to post the override instead. An unreadable source can only cause a false BLOCK, never a false pass, so degrading it to a loud warning is safe — and blocking on it was never the honest behaviour.
2. The override is accepted from a PR REVIEW body as well as a comment, so the hatch is not hostage to one API scope. A review of type "Comment" works on the token CI has today. The gate names which source it used. PENDING (unsubmitted, author-only draft) reviews are dropped.
3. set-ci-token.sh now requests read:issue, which makes the ordinary comment box work — the thing anybody reaches for first. It takes effect only on a re-run: an existing token's scopes cannot be widened in place.

ci/tests/approval-override.test.sh grows from 23 to 38 assertions, covering the reviews source, PENDING reviews, an error object from the comments API, "an unreadable comments source must not suppress a valid review override", latest-wins across both sources, and that the diagnostic actually quotes Gitea's message.

One adjacent fix, deliberately in scope: ci/tests/gate-decision.test.sh was already broken. It hand-maintains a list of stub stage scripts that "must name EVERY stage gate.sh runs", and it did not name approval-policy — added to gate.sh by PR #17 and never added here. run_stage therefore got exit 127 from a missing file and recorded a FAILED stage in every case, so 15 of 50 assertions failed, including every one expecting a PASS/UNDETERMINED headline. Measured on pristine origin/main (0c7e8dd) before touching it, so the attribution is not a guess.

I fixed it because make -C ci test is the only runner for the suite this change adds — leaving it red would have made the new tests unverifiable in practice. The file's own comment already narrated this exact failure for gitea-settings and did not prevent the repeat, which says the comment was not the fix. So the stub list is now derived from gate.sh's own run_stage lines and asserts it parsed at least 8 of them: a stage added to the gate can no longer be silently missing from its test. make -C ci test goes from FAILED to all suites passed.

What I verified

• The lock-out is real, and PR #35 is the proof. Queried live: head sha 8a094c46, statuses gate / gate (pull_request) = success :: Successful in 1m41s, /pulls/35/reviews → [], /branch_protections/main → required_approvals: 1. I read the rule object, not /branches/main, per §2b.
• jakman85 is the only collaborator: /collaborators → one entry. Permission owner.
• gitea-admin also reports owner — this is why the agent-account consequence above is a statement of fact and not a hypothetical.
• PR #35 touches no sensitive paths (apps/wave-0/plex.yaml, apps/wave-1/{radarr,sonarr}.yaml, catalog/{radarr,sonarr}.yaml, its decisions entry) — so ci/approval-policy.sh was never its blocker. Layer 1 alone unblocks it.
• API shapes verified before depending on them, rather than assumed: /git/commits/{sha} returns .commit.committer.date as RFC3339 with an offset (2026-10-01T23:09:08-05:00), not a trailing Z — a parser handling only Z would have silently dropped every real override, so there is a test for the offset form. PR conversation comments are under /issues/{n}/comments; /pulls/{n}/comments returns diff comments only and would never have seen an override.
• ci/tests/approval-override.test.sh: 23 passed, 0 failed. Covers the happy path, bare directive, mid-sentence and >-quoted directives, stale and equal-timestamp overrides, unparseable timestamps, missing login, invalid JSON, non-list JSON, latest-wins, offset timestamps, and a configurable directive.
• bash -n clean on ci/approval-policy.sh, infra/gitea/allow-admin-self-merge.sh, infra/gitea/require-gate.sh; jq -e . clean on both JSON declarations.
• gate-decision.test.sh's 15 failures are pre-existing, measured not assumed. Extracted origin/main's ci/ at 0c7e8dd into a throwaway directory and ran the suite from there: FAIL, 15 failed / 35 passed. After deriving the stub list from gate.sh: PASS, 50 assertions.
• make -C ci test in the pinned toolbox: ci/tests: all suites passed, rc 0 — approval-override 38, gate-decision 50, touched-requirements-base 31, test-attribution 15, touched-requirements 26, decisions-log pass. It was rc 2 (ci/tests: FAILED) before the stub-list fix.
• The live flip worked, and the drift detector proved it. Run 202 (before): gitea-settings: FAIL, required_approvals: DECLARED 0 but LIVE is 1. Run 203 (after the operator ran the PATCH): gitea-settings: pass, with block_admin_merge_override = true among the keys it checked. So the narrow change held — the approval count moved and the gate's teeth did not.
• The first override attempt failed for a measured reason, not a guessed one. Run 203 log: approval-override: COMMENTS_JSON is not a list. Token scopes read from set-ci-token.sh: [read:repository, read:admin]. The same run successfully read /pulls/38 and /pulls/38/files (author and sensitive paths both resolved) and got past the reviews validation, which is how I know /pulls/{n}/reviews IS readable with that token — from behaviour, not from documentation.
• The comment itself was valid. created_at 2026-10-02T05:31:24Z vs head commit 616a497 committer date 2026-10-02T00:02:20-05:00 → the comment is 29m04s newer, so it passed the staleness rule. Ruling the comment out is what left the token scope as the only candidate.
• ?limit=1000 is not the culprit. Queried /issues/38/comments with limit=1000, limit=50 and no limit using my own credentials: type=array in all three. So the difference is the credential, not the query.
• The root cause is confirmed by Gitea, in its own words. Run 206 (the fix in place): [warn] cannot read PR COMMENTS from Gitea — token does not have at least one of required scope(s), required=[read:issue], token scope=read:admin,read:repository. followed by the instruction to post the override as a review. That is the diagnosis the first attempt should have produced, and gitea-settings: pass in the same run. The stage still ends FAIL, correctly — there is no valid override on the PR yet.
• Toolbox merge gate (./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'), run on the rebased branch: rc 3, MERGE GATE: UNDETERMINED. Green stages: spec-immutability, decisions-log, lint, l0 (23 offline cases, 0 failures), render-golden including the new catalog/render.py --selftest, l1 (3 cases), approval-policy (N/A offline). Skipped: gitea-settings (no SETTINGS_API_TOKEN on this host) and conformance (no cluster). This is NOT a green run and I am not reporting it as one — rc 3 is UNDETERMINED per the CLAUDE.md §3 contract.

What I did NOT verify

• The override has still never been ACCEPTED by a real run. This is the second attempt at this bullet, which is itself the point: the first pass called the hatch unverified, and the first real test found a defect that no amount of local testing would have (an API scope CI has and I did not). What is now measured is that the parse → reject path works end to end and diagnoses itself. The accepting branch — directive found → live permission lookup → exit 0 — has never executed. Posting the override as a review on PR #38 is its first real test, and I would not be surprised by a third round.
• read:issue is the right scope name — Gitea said so itself, so this is no longer a guess. Run 206 printed Gitea's own message: token does not have at least one of required scope(s), required=[read:issue], token scope=read:admin,read:repository. That both confirms the root cause and names the string, which matters because this repo has been burned by a token name that could not exist (GITEA_SETTINGS_TOKEN). What is still unverified is that the token mint accepts it — POST /users/{u}/tokens with that scope has not been run. The review route does not depend on it.
• I did not verify that Gitea lets a PR's own author submit a COMMENT-type review. Gitea blocks author self-approval; "Comment" is expected to be allowed, and that expectation is now load-bearing for the recommended route. It is one click to find out, and I deliberately did not test it by posting an override myself: doing so as gitea-admin is precisely the agent self-override that CLAUDE.md now forbids.
• The whole re-run mechanism is manual. gate.yml triggers on pull_request and push: main only — there is no issue_comment trigger, so posting an override does not re-run the gate. Re-run the job from the Actions UI, or close and reopen the PR (which fires pull_request and leaves the head SHA alone). Do not push an empty commit to re-trigger it: that moves the head commit and makes the override stale by the rule above, so it would have to be posted again. A comment trigger was considered and not built — it would need to check out PR head code while the job holds SETTINGS_API_TOKEN, which is the classic pwn-request shape and not worth it here.
• I have not seen the gate still REFUSE a merge now that approvals are 0. The claim that block_admin_merge_override: true keeps a red gate unmergeable rests on Gitea's documented behaviour and on the 2026-09-29 session, not on a provoked refusal in this one. That probe is written out at the foot of allow-admin-self-merge.sh and should be done, exactly as the push rejection was provoked rather than assumed.
• Clock skew in the staleness check is unhandled. Staleness compares the comment's server-side created_at against the head commit's committer date, which comes from the committer's machine. A commit dated in the future would reject legitimate overrides until real time catches up (fail-closed, so loud, but annoying). Gitea's own dismiss_stale_approvals compares SHAs instead; a comment cannot. Not hit here — PR #35's commit date is ~21 min behind server time — and not fixed.
• ci/tests/approval-override.test.sh does not run in the merge gate. ci/gate.sh runs no ci/tests/*; they run only via make -C ci test. So a regression in the parser would not be caught by CI. PR #33 (merged while this branch was being written, and rebased onto here) fixed the same class of gap for catalog/render.py --selftest by invoking it from ci/render-golden.sh — it did not wire ci/tests/* in, which I confirmed after rebasing rather than assuming. Wiring them is a one-line stage and I did not widen scope to it.
• Authorisation itself is untested code. ci/approval-override.py is unit-tested; the live permission lookup, the override_admins allowlist branch, and every new BLOCK path in ci/approval-policy.sh need Gitea and have never been exercised. The allowlist branch in particular is implemented but unused — override_admins is absent by design.
• The other open PRs will go red once the live value changes, until they are rebased onto the merged declaration: as of writing those are #29 and #37 (both gitea-admin), plus #35. I did not rebase them. Merging them first avoids it entirely; the script's header gives the order. PRs #33, #34 and #36 merged while this branch was being written — this branch is rebased onto 0c7e8dd, so the list was re-measured rather than carried forward.
• No cluster state was read or changed; nothing here touches Flux, Kyverno or any workload.