home-network-k3s-impl/infra/gitea

infra/gitea — platform settings that are NOT GitOps-managed

Gitea's own repo settings (branch protection, required status checks, approvals) live in Gitea's database, not in a manifest Flux reconciles. They are therefore drift-prone and invisible to the gate — exactly how the state below went unnoticed.

The gap this closes

Verified against the live API on 2026-09-29, against a DECISIONS-LOG.md entry (2026-08-31) asserting the opposite:

Claimed	Measured
main is a protected branch, gate is a required status check	branch_protections → []; protected:false, enable_status_check:false, required_approvals:0, user_can_push:true
Runner incluster-dind online	zero runners registered at repo, org and site level

So main accepted direct pushes, and the gate could not run at all. CLAUDE.md is the rule; this directory is how the rule gets enforced by the machine.

branch-protection.sh

Blocks direct pushes to main. Run it, then run the push probe it prints — verify, do not trust the API response, since Gitea may let site admins bypass protection.

./infra/gitea/branch-protection.sh

It authenticates with the credential git already has stored for this host, and never prints the secret.

runner-token-rotate.sh

Mints a fresh instance-level registration token from the Gitea API and writes it into apps/wave-1/gitea-actions-runner-secret.sops.yaml via sops set. The token value is never printed.

./infra/gitea/runner-token-rotate.sh

A fresh token is usually not sufficient on its own. act_runner registers exactly once and persists its identity in /data/.runner on the act-runner-data PVC. If Gitea no longer holds a matching runner row, the pod starts, sees .runner, skips registration, and polls with an identity Gitea does not recognise — so it never appears in the runner list. That is precisely the observed symptom: zero runners, pod running, no error anyone looks at. Clearing the PVC is what forces a real re-registration; the script prints the commands.

Sequence — DONE 2026-09-29

All four steps are complete and each was verified, not assumed:

Step	State
1. Runner registered	✅ incluster-dind online. Root cause of its absence was a stale /data/.runner identity; clearing the PVC forced re-registration
2. Gate observed green on a real PR	✅ gate / gate (pull_request) ~1m35s
3. enable_status_check: true	✅ context gate / gate (pull_request) — that one ONLY
4. Collaborator + approvals	✅ jakman85 is owner. required_approvals was raised to 1 here and lowered back to 0 on 2026-10-01 — see §6, it was a lock-out

Direct push to main was provoked and rejected (pre-receive hook declined) as gitea-admin, a site admin — so push protection is not merely declared, it holds for the account agents authenticate as.

5. Block the admin override — the step the first pass MISSED

enable_status_check and required_approvals were set on 2026-09-29 and still did not bite, because block_admin_merge_override was false. Gitea lets repo admins override merge requirements without it, and jakman85 holds owner — so the two settings that looked like the gate were overridable by precisely the account that merges.

The evidence was sitting in plain sight: PRs #5 through #9 all merged with required_approvals: 1 in force and zero approvals recorded on any of them. Checking the branch view (/branches/main) hid this — it does not expose the field. Read the rule object (/branch_protections/main) and enumerate its keys.

Do not probe for field names you assume exist. This Gitea (1.24.3) has no apply_to_admins; asking jq for it returns null, which reads like "set to false" but means "key absent". Enumerate with to_entries and act on what is really there.

Now block_admin_merge_override: true. That is still only a setting, not evidence — it is proven when a green PR is REFUSED a merge until approved, and not before.

require-gate.sh / verify-then-enforce.sh

verify-then-enforce.sh is the re-runnable form of the whole safe order. It merges nothing — merging is a human's reviewed act — but it forces the Flux reconcile and then aborts unless the runner pod is healthy, a runner is online, and a real workflow goes green on it, before enabling enforcement and probing the push. require-gate.sh is the enforcement step alone.

Re-run verify-then-enforce.sh after any future change to the act-runner Deployment.

expected-branch-protection.json + ci/gitea-settings.sh — the detector

expected-branch-protection.json declares the desired rule. ci/gitea-settings.sh runs as a merge-gate stage, reads the LIVE rule, and fails the gate on any divergence — plus asserts at least one runner is online. It is the only thing in this repository that can see Gitea's own database.

It reads /branch_protections/main, not /branches/main: the branch view does not expose block_admin_merge_override, and it showed all-green for hours while the rule was overridable.

Behaviour, proven by test rather than assumed:

Situation	Exit	Effect
live matches the declaration	0	green
any key drifted, or no runner online	1	blocks the merge
no SETTINGS_API_TOKEN, or Gitea unreachable	3	skips loudly; never a pass, never a block

The secret is SETTINGS_API_TOKEN. An earlier draft of this table named GITEA_SETTINGS_TOKEN, which cannot exist as a Gitea secret at all — Gitea reserves the GITEA_ prefix and returns HTTP 400 invalid secret name. That is the whole 2026-09-29 inert-detector story, so the wrong name surviving here is worth correcting rather than reading past. Measured in ci/gitea-settings.sh:

TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"

so GITEA_SETTINGS_TOKEN is still honoured — but only as an environment variable for a LOCAL run. It can never be the CI wiring, because the secret that would feed it cannot be created. In CI, only SETTINGS_API_TOKEN exists.

The skip is deliberately not a block: an unwired credential blocking every PR is the lock-yourself-out failure this repo keeps hitting. Wire the token with set-ci-token.sh, confirm the stage goes green on a real PR, and only then set REQUIRE_GITEA_SETTINGS=true in gate.yml so a future missing token blocks instead of skipping.

Changing the declaration is how you change policy. If the live state is what you now want, edit this JSON in the same commit — deliberately, in review — rather than letting the two drift.

6. allow-admin-self-merge.sh — undoing the lock-out step 4 created (2026-10-01)

Step 4 above raised required_approvals to 1. That was a lock-out, not a protection, and the note at the bottom of require-gate.sh had already said why: Gitea will not let an author approve their own PR. With jakman85 the only collaborator, there was no account in existence that could approve a PR jakman85 opened.

Measured 2026-10-01 — PR #35: gate / gate (pull_request) success in 1m41s, reviews [], required_approvals: 1, block_admin_merge_override: true. Unmergeable, permanently, with no action available to its author. The same was true of any PR fixing it.

It was also never the documented design. ci/approval-policy.sh's header and ci/gate.sh:194 both already describe the model as "required_approvals drops to 0 and the GATE decides", and decisions/2026-09-30-flux-deploy-notification-inhibited.md flagged the 1-vs-0 disagreement as unresolved and not that session's to settle. So this is a correction toward the stated intent, not a weakening away from it.

./infra/gitea/allow-admin-self-merge.sh      # required_approvals -> 0 (read from the declaration)

What stays switched on, and why that is the point. The other way to unblock an owner is block_admin_merge_override: false, and it would have been the wrong one — that re-opens exactly what §5 above closed, letting the one account that merges bypass the gate itself. This script touches the approval count only and verifies afterwards that block_admin_merge_override and enable_status_check are both still true, failing loudly if the PATCH disturbed either:

	before	after
required_approvals	1	0
block_admin_merge_override	true	true (asserted)
enable_status_check	true	true (asserted)

So a red gate is still unmergeable by everyone, administrators included. What changed is that a green one no longer needs a signature that cannot exist.

Security-relevant changes are still reviewed — by ci/approval-policy.sh against approval-policy.json sensitive_paths, not by this field. An administrator waives that review only by posting /approve-override <reason> on the PR: reason mandatory, permission re-checked live, invalidated by a new commit, and recorded by account in the gate summary.

Post the override as a PR review (type "Comment"), not in the plain comment box — until the CI token is re-minted. SETTINGS_API_TOKEN is [read:repository, read:admin], and /issues/{n}/comments needs Gitea's issue scope, so Gitea answers with an error object instead of an array and the override is invisible. Measured on PR #38 (2026-10-02): a correct, fresh, well-formed override in the comment box produced approval-policy: FAIL reading as "no approving review" — a permission failure disguised as a missing one. /pulls/{n}/reviews is under read:repository and worked in the same run. The stage now reads both sources, names which one it used, and says plainly when the comments API is unreadable. set-ci-token.sh requests read:issue as of 2026-10-02, so re-run it and the comment box works too — token scopes cannot be widened in place. Note that the agent account gitea-admin also reports permission owner, so it can clear its own PRs that way; that is an accepted consequence of authorising by permission alone, and the route to narrowing it (an override_admins allowlist, already honoured by the code) is written down in approval-policy.json.

Sequencing. ci/gitea-settings.sh compares live against the declaration from the branch under test, and REQUIRE_GITEA_SETTINGS=true. So merge any other open PRs before running this, then run it, then re-run the gate on the branch carrying the new declaration. Any PR left open across the change declares 1 against a live 0 and goes red until rebased — the detector working, not a fault. The script's header spells the order out.

conformance-trigger-kubeconfig.sh — the narrow L2/L3 trigger credential

Implements docs/plans/l2-l3-conformance-ci-wiring.md §3 item 6 (= §5 step 7): mint a ServiceAccount token for conformance/conformance-trigger, wrap only that token in a kubeconfig, store it as the Actions secret CONFORMANCE_TRIGGER_KUBECONFIG, and — separately, and only on request — set vars.CLUSTER_AVAILABLE=true.

./infra/gitea/conformance-trigger-kubeconfig.sh --dry-run   # measure only; writes nothing
./infra/gitea/conformance-trigger-kubeconfig.sh             # mint + store the secret
./infra/gitea/conformance-trigger-kubeconfig.sh --arm-cluster-available   # + the variable

Never a cluster-admin kubeconfig, never a copy of the operator's own. The token is never printed: it goes from the TokenRequest API into a 0600 file, into the request body file (--data @file, so not even into ps), and nowhere else. The script refuses to run under set -x.

Its preconditions are split in two, and the split is the point.

Class	Checks	Blocks
mint-blocking (cluster)	SA/Role/RoleBinding exist; ClusterPolicy conformance-trigger-job-shape is present, Enforce, failurePolicy: Fail, Ready	the mint — nothing is created
arm-blocking (repo)	digest-pinned CONFORMANCE_RUNNER_IMAGE; chainsaw-job.yaml has an init-container and the admitted image; ci/conformance.sh names the trigger kubeconfig in code, not a comment	only CLUSTER_AVAILABLE

Why mint-blocking: Kubernetes puts no use check on a Job's serviceAccountName, so a token that can create jobs in conformance can run arbitrary code as the privileged conformance-runner SA. RBAC cannot close that; the Kyverno policy is the boundary. Minting while it is absent or in Audit hands CI the privileged identity, so the script will not do it.

Why arm-blocking is separate: storing the secret before the runner image / Job template / CI trigger flow exist is harmless (the job is gated off), and is the plan's intended order. So running this today provisions the secret and then refuses to arm the variable, naming the missing item.

CLUSTER_AVAILABLE is not a "turn conformance on" switch. Measured 2026-09-29: set with no trigger path, run.sh scaffold-skipped all 54 tests at rc 0 and ci/conformance.sh printed "L2/L3 conformance passed" — a fake green on an enforcing required check. ci/conformance.sh now blocks that case, so arming early does not enable L2/L3; it only breaks every PR. Hence: --arm-cluster-available plus all preconditions green plus a typed confirmation on a TTY, or nothing happens.

Both writes are verified the way set-ci-token.sh learned to: check %{http_code} and read the object back. For the secret that proves presence by name only (Gitea never returns secret values); for the variable it proves the value is exactly true, which is what gate.yml compares against.

The token expires — it is a TokenRequest token, and the script decodes the JWT exp claim to report the real expiry rather than trusting --duration (the API server caps it). Re-run to rotate. After expiry the trigger gets 401 and the stage fails loudly rather than skipping.

--server exists because the embedded API-server URL must be reachable from the CI runner pod — the VIP — which is not necessarily the address your workstation uses. The script refuses a loopback server outright.

Why these scripts exist at all, and what is still missing

Gitea's repo settings live in its own database, not in anything Flux reconciles or the conformance gate can read. Nothing detects drift from what this directory declares. That is the root cause of the entire 2026-09-29 session: branch protection was off and the runner was gone for a month while the log recorded both as working, and no check existed that could have caught it.

These scripts encode the desired state and can re-assert it; ci/gitea-settings.sh above is the detector that turns divergence from a silent, invisible failure into a loud one. It closes the root cause of the 2026-09-29 session.

It is only as good as its wiring: until SETTINGS_API_TOKEN exists it skips, and a skipped detector detects nothing. (That secret was stored and verified in PR #15; this sentence predates it. The name matters — see the note under the behaviour table above.)

Historical sequence (kept for context)

Original ordering notes

1. Get an Actions runner registered — runner-token-rotate.sh, then clear the PVC so the runner actually re-registers, then confirm it shows online in the API. Do not skip the confirmation: the whole reason this went unnoticed is that a runner was recorded as online without anyone checking.
2. Watch the gate actually report on a real PR, and note the context name it produced.
3. Only then set enable_status_check: true with that exact context.
4. Add the human reviewer as a repo collaborator, then raise required_approvals to 1. — Step 4 was REVERSED on 2026-10-01 (§6). It assumed a reviewer distinct from the author would exist. One never did, so this step produced a lock-out. Do not re-apply it.

Do not do step 3 before step 1. A required status check with no runner leaves every PR permanently pending — including the PR that would fix it. That half is measured: PR #1 sat pending for 30.6 minutes across 122 polls with no runner registered, and never reported.

Measured, once the runner came up (2026-09-29)

The first CI jobs this repo ever ran, on PR #1 and PR #2:

Context	Result	Time
gate / gate (pull_request)	success	1m58s (PR #1), 1m34s (PR #2)
gate / gate (push)	success	1m32s (PR #2 only)
gate / conformance (pull_request)	success — "Has been skipped"	—

Require gate / gate (pull_request). Do NOT require either conformance context — but not for the reason an earlier draft of this file gave. A skipped Gitea Actions job does not hang a status check: Gitea reports it as success with the description "Has been skipped". So requiring it would not deadlock anything — it would pass vacuously, every time, forever, while nothing ran. Under HARD RULE 4 that is worse than a deadlock: a deadlock is loud, a permanently-green check that verifies nothing is exactly the fail-open shape this repo keeps finding.

The (push) contexts appear only for branches matching gate.yml's push trigger (feature/**, fix/**, docs/**, chore/**). feat/** is not in that list, which is why PR #1's feat/ branch produced only pull_request contexts. Require only the pull_request context, or PRs from non-matching prefixes will never satisfy the rule.