home-network-k3s-impl/infra/gitea/README.md

# infra/gitea — platform settings that are NOT GitOps-managed

Gitea's own repo settings (branch protection, required status checks, approvals) live
in Gitea's database, not in a manifest Flux reconciles. They are therefore **drift-prone
and invisible to the gate** — exactly how the state below went unnoticed.

## The gap this closes

Verified against the live API on 2026-09-29, against a `DECISIONS-LOG.md` entry
(2026-08-31) asserting the opposite:

| Claimed | Measured |
|---|---|
| `main` is a protected branch, gate is a required status check | `branch_protections` → `[]`; `protected:false`, `enable_status_check:false`, `required_approvals:0`, `user_can_push:true` |
| Runner `incluster-dind` online | **zero** runners registered at repo, org and site level |

So `main` accepted direct pushes, and the gate could not run at all. `CLAUDE.md` is the
rule; this directory is how the rule gets enforced by the machine.

## `branch-protection.sh`

Blocks direct pushes to `main`. Run it, then run the push probe it prints — **verify,
do not trust the API response**, since Gitea may let site admins bypass protection.

    ./infra/gitea/branch-protection.sh

It authenticates with the credential `git` already has stored for this host, and never
prints the secret.

## `runner-token-rotate.sh`

Mints a fresh instance-level registration token from the Gitea API and writes it into
`apps/wave-1/gitea-actions-runner-secret.sops.yaml` via `sops set`. **The token value is
never printed.**

    ./infra/gitea/runner-token-rotate.sh

A fresh token is usually **not sufficient on its own**. `act_runner` registers exactly once
and persists its identity in `/data/.runner` on the `act-runner-data` PVC. If Gitea no
longer holds a matching runner row, the pod starts, sees `.runner`, **skips registration**,
and polls with an identity Gitea does not recognise — so it never appears in the runner
list. That is precisely the observed symptom: zero runners, pod running, no error anyone
looks at. Clearing the PVC is what forces a real re-registration; the script prints the
commands.

## Sequence — DONE 2026-09-29

All four steps are complete and each was verified, not assumed:

| Step | State |
|---|---|
| 1. Runner registered | ✅ `incluster-dind` online. Root cause of its absence was a stale `/data/.runner` identity; clearing the PVC forced re-registration |
| 2. Gate observed green on a real PR | ✅ `gate / gate (pull_request)` ~1m35s |
| 3. `enable_status_check: true` | ✅ context `gate / gate (pull_request)` — that one ONLY |
| 4. Collaborator + approvals | ✅ `jakman85` is owner. `required_approvals` was raised to 1 here and **lowered back to 0 on 2026-10-01** — see §6, it was a lock-out |

Direct push to `main` was **provoked and rejected** (`pre-receive hook declined`) as
`gitea-admin`, a site admin — so push protection is not merely declared, it holds for the
account agents authenticate as.

### 5. Block the admin override — the step the first pass MISSED

`enable_status_check` and `required_approvals` were set on 2026-09-29 and **still did not
bite**, because `block_admin_merge_override` was `false`. Gitea lets repo admins override
merge requirements without it, and `jakman85` holds `owner` — so the two settings that looked
like the gate were overridable by precisely the account that merges.

**The evidence was sitting in plain sight:** PRs #5 through #9 all merged with
`required_approvals: 1` in force and **zero approvals recorded on any of them**. Checking the
*branch* view (`/branches/main`) hid this — it does not expose the field. Read the rule object
(`/branch_protections/main`) and enumerate its keys.

> **Do not probe for field names you assume exist.** This Gitea (1.24.3) has no
> `apply_to_admins`; asking `jq` for it returns `null`, which reads like "set to false" but
> means "key absent". Enumerate with `to_entries` and act on what is really there.

Now `block_admin_merge_override: true`. **That is still only a setting, not evidence** — it is
proven when a green PR is REFUSED a merge until approved, and not before.

### `require-gate.sh` / `verify-then-enforce.sh`

`verify-then-enforce.sh` is the re-runnable form of the whole safe order. It merges nothing —
merging is a human's reviewed act — but it forces the Flux reconcile and then **aborts** unless
the runner pod is healthy, a runner is online, and a real workflow goes green on it, before
enabling enforcement and probing the push. `require-gate.sh` is the enforcement step alone.

Re-run `verify-then-enforce.sh` after any future change to the act-runner Deployment.

### `expected-branch-protection.json` + `ci/gitea-settings.sh` — the detector

`expected-branch-protection.json` **declares** the desired rule. `ci/gitea-settings.sh` runs as
a merge-gate stage, reads the LIVE rule, and **fails the gate on any divergence** — plus asserts
at least one runner is online. It is the only thing in this repository that can see Gitea's own
database.

It reads `/branch_protections/main`, not `/branches/main`: the branch view does **not** expose
`block_admin_merge_override`, and it showed all-green for hours while the rule was overridable.

Behaviour, proven by test rather than assumed:

| Situation | Exit | Effect |
|---|---|---|
| live matches the declaration | 0 | green |
| any key drifted, or no runner online | 1 | **blocks the merge** |
| no `SETTINGS_API_TOKEN`, or Gitea unreachable | 3 | skips **loudly**; never a pass, never a block |

> **The secret is `SETTINGS_API_TOKEN`.** An earlier draft of this table named
> `GITEA_SETTINGS_TOKEN`, which **cannot exist as a Gitea secret at all** — Gitea reserves the
> `GITEA_` prefix and returns HTTP 400 `invalid secret name`. That is the whole 2026-09-29
> inert-detector story, so the wrong name surviving here is worth correcting rather than
> reading past. Measured in `ci/gitea-settings.sh`:
>
>     TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
>
> so `GITEA_SETTINGS_TOKEN` is still honoured — but **only as an environment variable for a
> LOCAL run**. It can never be the CI wiring, because the secret that would feed it cannot be
> created. In CI, only `SETTINGS_API_TOKEN` exists.

The skip is deliberately *not* a block: an unwired credential blocking every PR is the
lock-yourself-out failure this repo keeps hitting. Wire the token with `set-ci-token.sh`, confirm
the stage goes **green on a real PR**, and only then set `REQUIRE_GITEA_SETTINGS=true` in
`gate.yml` so a future missing token blocks instead of skipping.

**Changing the declaration is how you change policy.** If the live state is what you now want,
edit this JSON in the same commit — deliberately, in review — rather than letting the two drift.

### 6. `allow-admin-self-merge.sh` — undoing the lock-out step 4 created (2026-10-01)

Step 4 above raised `required_approvals` to 1. That was a **lock-out, not a protection**, and the
note at the bottom of `require-gate.sh` had already said why: *Gitea will not let an author approve
their own PR.* With `jakman85` the only collaborator, there was no account in existence that could
approve a PR `jakman85` opened.

Measured 2026-10-01 — **PR #35**: `gate / gate (pull_request)` **success in 1m41s**, reviews `[]`,
`required_approvals: 1`, `block_admin_merge_override: true`. Unmergeable, permanently, with no
action available to its author. The same was true of any PR fixing it.

It was also never the documented design. `ci/approval-policy.sh`'s header and `ci/gate.sh:194`
both already describe the model as *"`required_approvals` drops to 0 and the GATE decides"*, and
`decisions/2026-09-30-flux-deploy-notification-inhibited.md` flagged the 1-vs-0 disagreement as
unresolved and not that session's to settle. So this is a correction toward the stated intent, not
a weakening away from it.

    ./infra/gitea/allow-admin-self-merge.sh      # required_approvals -> 0 (read from the declaration)

**What stays switched on, and why that is the point.** The other way to unblock an owner is
`block_admin_merge_override: false`, and it would have been the wrong one — that re-opens exactly
what §5 above closed, letting the one account that merges bypass the gate itself. This script
touches the approval **count** only and *verifies* afterwards that `block_admin_merge_override` and
`enable_status_check` are both still `true`, failing loudly if the PATCH disturbed either:

| | before | after |
|---|---|---|
| `required_approvals` | 1 | **0** |
| `block_admin_merge_override` | true | true (asserted) |
| `enable_status_check` | true | true (asserted) |

So a **red gate is still unmergeable by everyone, administrators included**. What changed is that
a *green* one no longer needs a signature that cannot exist.

**Security-relevant changes are still reviewed** — by `ci/approval-policy.sh` against
`approval-policy.json` `sensitive_paths`, not by this field. An administrator waives that review
only by posting `/approve-override <reason>` on the PR: reason mandatory, permission re-checked
live, invalidated by a new commit, and recorded by account in the gate summary.

> **Post the override as a PR *review* (type "Comment"), not in the plain comment box** — until
> the CI token is re-minted. `SETTINGS_API_TOKEN` is `[read:repository, read:admin]`, and
> `/issues/{n}/comments` needs Gitea's *issue* scope, so Gitea answers with an error **object**
> instead of an array and the override is invisible. Measured on PR #38 (2026-10-02): a correct,
> fresh, well-formed override in the comment box produced `approval-policy: FAIL` reading as "no
> approving review" — a permission failure disguised as a missing one. `/pulls/{n}/reviews` is
> under `read:repository` and worked in the same run. The stage now reads **both** sources, names
> which one it used, and says plainly when the comments API is unreadable. `set-ci-token.sh`
> requests `read:issue` as of 2026-10-02, so **re-run it** and the comment box works too — token
> scopes cannot be widened in place. Note that the
**agent account `gitea-admin` also reports permission `owner`**, so it can clear its own PRs that
way; that is an accepted consequence of authorising by permission alone, and the route to
narrowing it (an `override_admins` allowlist, already honoured by the code) is written down in
`approval-policy.json`.

**Sequencing.** `ci/gitea-settings.sh` compares live against the declaration *from the branch under
test*, and `REQUIRE_GITEA_SETTINGS=true`. So merge any other open PRs **before** running this, then
run it, then re-run the gate on the branch carrying the new declaration. Any PR left open across
the change declares 1 against a live 0 and goes **red until rebased** — the detector working, not
a fault. The script's header spells the order out.

## `conformance-trigger-kubeconfig.sh` — the narrow L2/L3 trigger credential

Implements `docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 6 (= §5 step 7): mint a
ServiceAccount token for `conformance/conformance-trigger`, wrap **only that token** in a
kubeconfig, store it as the Actions secret `CONFORMANCE_TRIGGER_KUBECONFIG`, and — separately,
and only on request — set `vars.CLUSTER_AVAILABLE=true`.

    ./infra/gitea/conformance-trigger-kubeconfig.sh --dry-run   # measure only; writes nothing
    ./infra/gitea/conformance-trigger-kubeconfig.sh             # mint + store the secret
    ./infra/gitea/conformance-trigger-kubeconfig.sh --arm-cluster-available   # + the variable

Never a cluster-admin kubeconfig, never a copy of the operator's own. The token is never
printed: it goes from the TokenRequest API into a 0600 file, into the request **body file**
(`--data @file`, so not even into `ps`), and nowhere else. The script refuses to run under
`set -x`.

**Its preconditions are split in two, and the split is the point.**

| Class | Checks | Blocks |
|---|---|---|
| **mint-blocking** (cluster) | SA/Role/RoleBinding exist; ClusterPolicy `conformance-trigger-job-shape` is present, `Enforce`, `failurePolicy: Fail`, Ready | the **mint** — nothing is created |
| **arm-blocking** (repo) | digest-pinned `CONFORMANCE_RUNNER_IMAGE`; `chainsaw-job.yaml` has an init-container and the admitted image; `ci/conformance.sh` names the trigger kubeconfig in **code, not a comment** | only `CLUSTER_AVAILABLE` |

Why mint-blocking: Kubernetes puts **no `use` check on a Job's `serviceAccountName`**, so a
token that can `create jobs` in `conformance` can run arbitrary code as the privileged
`conformance-runner` SA. RBAC cannot close that; the Kyverno policy is the boundary. Minting
while it is absent or in `Audit` hands CI the privileged identity, so the script will not do it.

Why arm-blocking is separate: storing the secret before the runner image / Job template / CI
trigger flow exist is harmless (the job is gated off), and is the plan's intended order. So
running this today provisions the secret and then **refuses to arm the variable**, naming the
missing item.

**`CLUSTER_AVAILABLE` is not a "turn conformance on" switch.** Measured 2026-09-29: set with no
trigger path, `run.sh` scaffold-skipped all 54 tests at rc 0 and `ci/conformance.sh` printed
"L2/L3 conformance passed" — a fake green on an enforcing required check. `ci/conformance.sh`
now blocks that case, so arming early does not enable L2/L3; it only breaks every PR. Hence:
`--arm-cluster-available` plus all preconditions green plus a typed confirmation on a TTY, or
nothing happens.

Both writes are verified the way `set-ci-token.sh` learned to: check `%{http_code}` **and**
read the object back. For the secret that proves presence by name only (Gitea never returns
secret values); for the variable it proves the value is exactly `true`, which is what
`gate.yml` compares against.

The token **expires** — it is a TokenRequest token, and the script decodes the JWT `exp` claim
to report the real expiry rather than trusting `--duration` (the API server caps it). Re-run to
rotate. After expiry the trigger gets 401 and the stage fails loudly rather than skipping.

`--server` exists because the embedded API-server URL must be reachable **from the CI runner
pod** — the VIP — which is not necessarily the address your workstation uses. The script
refuses a loopback server outright.

### Why these scripts exist at all, and what is still missing

Gitea's repo settings live in **its own database**, not in anything Flux reconciles or the
conformance gate can read. Nothing detects drift from what this directory declares. That is
the root cause of the entire 2026-09-29 session: branch protection was off and the runner was
gone for a month while the log recorded both as working, and **no check existed that could
have caught it.**

These scripts encode the desired state and can re-assert it; `ci/gitea-settings.sh` above is the
**detector** that turns divergence from a silent, invisible failure into a loud one. It closes the
root cause of the 2026-09-29 session.

It is only as good as its wiring: **until `SETTINGS_API_TOKEN` exists it skips**, and a skipped
detector detects nothing. (That secret was stored and verified in PR #15; this sentence predates
it. The name matters — see the note under the behaviour table above.)

## Historical sequence (kept for context)

### Original ordering notes

1. **Get an Actions runner registered** — `runner-token-rotate.sh`, then clear the PVC so
   the runner actually re-registers, then confirm it shows `online` in the API. Do not
   skip the confirmation: the whole reason this went unnoticed is that a runner was
   *recorded* as online without anyone checking.
2. **Watch the gate actually report on a real PR**, and note the context name it produced.
3. **Only then set `enable_status_check: true`** with that exact context.
4. Add the human reviewer as a repo collaborator, then raise `required_approvals` to 1.
   — **Step 4 was REVERSED on 2026-10-01 (§6).** It assumed a reviewer distinct from the author
   would exist. One never did, so this step produced a lock-out. Do not re-apply it.

> **Do not do step 3 before step 1.** A required status check with no runner leaves every
> PR permanently pending — including the PR that would fix it. That half is measured: PR #1
> sat `pending` for 30.6 minutes across 122 polls with no runner registered, and never
> reported.

### Measured, once the runner came up (2026-09-29)

The first CI jobs this repo ever ran, on PR #1 and PR #2:

| Context | Result | Time |
|---|---|---|
| `gate / gate (pull_request)` | success | 1m58s (PR #1), 1m34s (PR #2) |
| `gate / gate (push)` | success | 1m32s (PR #2 only) |
| `gate / conformance (pull_request)` | **success — "Has been skipped"** | — |

**Require `gate / gate (pull_request)`. Do NOT require either `conformance` context —
but not for the reason an earlier draft of this file gave.** A skipped Gitea Actions job
does not hang a status check: Gitea reports it as **`success`** with the description
"Has been skipped". So requiring it would not deadlock anything — it would pass
*vacuously*, every time, forever, while nothing ran. Under HARD RULE 4 that is worse than
a deadlock: a deadlock is loud, a permanently-green check that verifies nothing is
exactly the fail-open shape this repo keeps finding.

The `(push)` contexts appear only for branches matching `gate.yml`'s push trigger
(`feature/**`, `fix/**`, `docs/**`, `chore/**`). `feat/**` is **not** in that list, which
is why PR #1's `feat/` branch produced only `pull_request` contexts. Require only the
`pull_request` context, or PRs from non-matching prefixes will never satisfy the rule.