jakman85andClaude Opus 5
e6f6e4febb
Every check in this repo reads the repository, the rendered manifests, or the cluster.
NONE could see Gitea's own database, and that is where branch protection and runner
registration live. Three failures came out of that blind spot on 2026-09-29, all the same
shape -- a setting that READ as enforced while not being, with nothing able to notice:
* main had NO branch protection for ~a month while DECISIONS-LOG recorded it as a
required status check;
* ZERO Actions runners were registered, so PR #1 -- the first PR this repo ever had --
sat pending for 30.6 minutes with nothing to execute it;
* after enable_status_check and required_approvals were turned on they STILL did not
bite, because block_admin_merge_override was false: PRs #5-#9 merged with zero
approvals recorded against required_approvals: 1.
infra/gitea/*.sh ASSERT the desired state. Nothing DETECTED drift from it. This is the
detector.
ci/gitea-settings.sh runs as a merge-gate stage: it diffs the live rule at
/branch_protections/main against infra/gitea/expected-branch-protection.json and asserts at
least one runner is online. It reads the RULE object, not /branches/main -- the branch view
does not expose block_admin_merge_override and showed all-green while the rule was
overridable by the only account that merges.
Verified by test, not assumed:
no token -> exit 3, skips loudly, does NOT block
live == declaration -> exit 0
injected drift -> exit 1, blocks, naming both the drifted key and the runner count
The skip is DELIBERATELY not in gate.sh's strict-skips loop. Making an absent credential
block every PR is precisely the lock-yourself-out failure this repo keeps walking into, so
a skip denies a PASS (exit 3) and says loudly what is unverified. REQUIRE_GITEA_SETTINGS=true
upgrades it to a block once the token is wired.
gate.yml's `|| [ "$?" -eq 3 ]` tolerance now covers a second stage. That widening is
declared in the file rather than left to be discovered, and it narrows back the moment
REQUIRE_GITEA_SETTINGS is set.
infra/gitea/set-ci-token.sh mints the token and stores it as the repo Actions secret without
ever printing the value.
UNTIL THAT TOKEN EXISTS THIS STAGE SKIPS, AND A SKIPPED DETECTOR DETECTS NOTHING.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
169 lines
8.8 KiB
Markdown
169 lines
8.8 KiB
Markdown
# infra/gitea — platform settings that are NOT GitOps-managed
|
|
|
|
Gitea's own repo settings (branch protection, required status checks, approvals) live
|
|
in Gitea's database, not in a manifest Flux reconciles. They are therefore **drift-prone
|
|
and invisible to the gate** — exactly how the state below went unnoticed.
|
|
|
|
## The gap this closes
|
|
|
|
Verified against the live API on 2026-09-29, against a `DECISIONS-LOG.md` entry
|
|
(2026-08-31) asserting the opposite:
|
|
|
|
| Claimed | Measured |
|
|
|---|---|
|
|
| `main` is a protected branch, gate is a required status check | `branch_protections` → `[]`; `protected:false`, `enable_status_check:false`, `required_approvals:0`, `user_can_push:true` |
|
|
| Runner `incluster-dind` online | **zero** runners registered at repo, org and site level |
|
|
|
|
So `main` accepted direct pushes, and the gate could not run at all. `CLAUDE.md` is the
|
|
rule; this directory is how the rule gets enforced by the machine.
|
|
|
|
## `branch-protection.sh`
|
|
|
|
Blocks direct pushes to `main`. Run it, then run the push probe it prints — **verify,
|
|
do not trust the API response**, since Gitea may let site admins bypass protection.
|
|
|
|
./infra/gitea/branch-protection.sh
|
|
|
|
It authenticates with the credential `git` already has stored for this host, and never
|
|
prints the secret.
|
|
|
|
## `runner-token-rotate.sh`
|
|
|
|
Mints a fresh instance-level registration token from the Gitea API and writes it into
|
|
`apps/wave-1/gitea-actions-runner-secret.sops.yaml` via `sops set`. **The token value is
|
|
never printed.**
|
|
|
|
./infra/gitea/runner-token-rotate.sh
|
|
|
|
A fresh token is usually **not sufficient on its own**. `act_runner` registers exactly once
|
|
and persists its identity in `/data/.runner` on the `act-runner-data` PVC. If Gitea no
|
|
longer holds a matching runner row, the pod starts, sees `.runner`, **skips registration**,
|
|
and polls with an identity Gitea does not recognise — so it never appears in the runner
|
|
list. That is precisely the observed symptom: zero runners, pod running, no error anyone
|
|
looks at. Clearing the PVC is what forces a real re-registration; the script prints the
|
|
commands.
|
|
|
|
## Sequence — DONE 2026-09-29
|
|
|
|
All four steps are complete and each was verified, not assumed:
|
|
|
|
| Step | State |
|
|
|---|---|
|
|
| 1. Runner registered | ✅ `incluster-dind` online. Root cause of its absence was a stale `/data/.runner` identity; clearing the PVC forced re-registration |
|
|
| 2. Gate observed green on a real PR | ✅ `gate / gate (pull_request)` ~1m35s |
|
|
| 3. `enable_status_check: true` | ✅ context `gate / gate (pull_request)` — that one ONLY |
|
|
| 4. Collaborator + approvals | ✅ `jakman85` is owner; `required_approvals: 1` |
|
|
|
|
Direct push to `main` was **provoked and rejected** (`pre-receive hook declined`) as
|
|
`gitea-admin`, a site admin — so push protection is not merely declared, it holds for the
|
|
account agents authenticate as.
|
|
|
|
### 5. Block the admin override — the step the first pass MISSED
|
|
|
|
`enable_status_check` and `required_approvals` were set on 2026-09-29 and **still did not
|
|
bite**, because `block_admin_merge_override` was `false`. Gitea lets repo admins override
|
|
merge requirements without it, and `jakman85` holds `owner` — so the two settings that looked
|
|
like the gate were overridable by precisely the account that merges.
|
|
|
|
**The evidence was sitting in plain sight:** PRs #5 through #9 all merged with
|
|
`required_approvals: 1` in force and **zero approvals recorded on any of them**. Checking the
|
|
*branch* view (`/branches/main`) hid this — it does not expose the field. Read the rule object
|
|
(`/branch_protections/main`) and enumerate its keys.
|
|
|
|
> **Do not probe for field names you assume exist.** This Gitea (1.24.3) has no
|
|
> `apply_to_admins`; asking `jq` for it returns `null`, which reads like "set to false" but
|
|
> means "key absent". Enumerate with `to_entries` and act on what is really there.
|
|
|
|
Now `block_admin_merge_override: true`. **That is still only a setting, not evidence** — it is
|
|
proven when a green PR is REFUSED a merge until approved, and not before.
|
|
|
|
### `require-gate.sh` / `verify-then-enforce.sh`
|
|
|
|
`verify-then-enforce.sh` is the re-runnable form of the whole safe order. It merges nothing —
|
|
merging is a human's reviewed act — but it forces the Flux reconcile and then **aborts** unless
|
|
the runner pod is healthy, a runner is online, and a real workflow goes green on it, before
|
|
enabling enforcement and probing the push. `require-gate.sh` is the enforcement step alone.
|
|
|
|
Re-run `verify-then-enforce.sh` after any future change to the act-runner Deployment.
|
|
|
|
### `expected-branch-protection.json` + `ci/gitea-settings.sh` — the detector
|
|
|
|
`expected-branch-protection.json` **declares** the desired rule. `ci/gitea-settings.sh` runs as
|
|
a merge-gate stage, reads the LIVE rule, and **fails the gate on any divergence** — plus asserts
|
|
at least one runner is online. It is the only thing in this repository that can see Gitea's own
|
|
database.
|
|
|
|
It reads `/branch_protections/main`, not `/branches/main`: the branch view does **not** expose
|
|
`block_admin_merge_override`, and it showed all-green for hours while the rule was overridable.
|
|
|
|
Behaviour, proven by test rather than assumed:
|
|
|
|
| Situation | Exit | Effect |
|
|
|---|---|---|
|
|
| live matches the declaration | 0 | green |
|
|
| any key drifted, or no runner online | 1 | **blocks the merge** |
|
|
| no `GITEA_SETTINGS_TOKEN`, or Gitea unreachable | 3 | skips **loudly**; never a pass, never a block |
|
|
|
|
The skip is deliberately *not* a block: an unwired credential blocking every PR is the
|
|
lock-yourself-out failure this repo keeps hitting. Wire the token with `set-ci-token.sh`, confirm
|
|
the stage goes **green on a real PR**, and only then set `REQUIRE_GITEA_SETTINGS=true` in
|
|
`gate.yml` so a future missing token blocks instead of skipping.
|
|
|
|
**Changing the declaration is how you change policy.** If the live state is what you now want,
|
|
edit this JSON in the same commit — deliberately, in review — rather than letting the two drift.
|
|
|
|
### Why these scripts exist at all, and what is still missing
|
|
|
|
Gitea's repo settings live in **its own database**, not in anything Flux reconciles or the
|
|
conformance gate can read. Nothing detects drift from what this directory declares. That is
|
|
the root cause of the entire 2026-09-29 session: branch protection was off and the runner was
|
|
gone for a month while the log recorded both as working, and **no check existed that could
|
|
have caught it.**
|
|
|
|
These scripts encode the desired state and can re-assert it; `ci/gitea-settings.sh` above is the
|
|
**detector** that turns divergence from a silent, invisible failure into a loud one. It closes the
|
|
root cause of the 2026-09-29 session.
|
|
|
|
It is only as good as its wiring: **until `GITEA_SETTINGS_TOKEN` exists it skips**, and a skipped
|
|
detector detects nothing. That is the remaining step.
|
|
|
|
## Historical sequence (kept for context)
|
|
|
|
### Original ordering notes
|
|
|
|
1. **Get an Actions runner registered** — `runner-token-rotate.sh`, then clear the PVC so
|
|
the runner actually re-registers, then confirm it shows `online` in the API. Do not
|
|
skip the confirmation: the whole reason this went unnoticed is that a runner was
|
|
*recorded* as online without anyone checking.
|
|
2. **Watch the gate actually report on a real PR**, and note the context name it produced.
|
|
3. **Only then set `enable_status_check: true`** with that exact context.
|
|
4. Add the human reviewer as a repo collaborator, then raise `required_approvals` to 1.
|
|
|
|
> **Do not do step 3 before step 1.** A required status check with no runner leaves every
|
|
> PR permanently pending — including the PR that would fix it. That half is measured: PR #1
|
|
> sat `pending` for 30.6 minutes across 122 polls with no runner registered, and never
|
|
> reported.
|
|
|
|
### Measured, once the runner came up (2026-09-29)
|
|
|
|
The first CI jobs this repo ever ran, on PR #1 and PR #2:
|
|
|
|
| Context | Result | Time |
|
|
|---|---|---|
|
|
| `gate / gate (pull_request)` | success | 1m58s (PR #1), 1m34s (PR #2) |
|
|
| `gate / gate (push)` | success | 1m32s (PR #2 only) |
|
|
| `gate / conformance (pull_request)` | **success — "Has been skipped"** | — |
|
|
|
|
**Require `gate / gate (pull_request)`. Do NOT require either `conformance` context —
|
|
but not for the reason an earlier draft of this file gave.** A skipped Gitea Actions job
|
|
does not hang a status check: Gitea reports it as **`success`** with the description
|
|
"Has been skipped". So requiring it would not deadlock anything — it would pass
|
|
*vacuously*, every time, forever, while nothing ran. Under HARD RULE 4 that is worse than
|
|
a deadlock: a deadlock is loud, a permanently-green check that verifies nothing is
|
|
exactly the fail-open shape this repo keeps finding.
|
|
|
|
The `(push)` contexts appear only for branches matching `gate.yml`'s push trigger
|
|
(`feature/**`, `fix/**`, `docs/**`, `chore/**`). `feat/**` is **not** in that list, which
|
|
is why PR #1's `feat/` branch produced only `pull_request` contexts. Require only the
|
|
`pull_request` context, or PRs from non-matching prefixes will never satisfy the rule.
|