jakman85andClaude Opus 5
3284612f06
docs/plans/l2-l3-conformance-ci-wiring.md §3 item 6, as an operator script:
infra/gitea/conformance-trigger-kubeconfig.sh. UNEXECUTED here -- minting a
cluster credential and changing Gitea variables are operator actions.
It mints a token for conformance/conformance-trigger (TokenRequest), wraps ONLY
that token with the cluster CA and server into a kubeconfig, and stores it as the
Actions secret CONFORMANCE_TRIGGER_KUBECONFIG. Never cluster-admin, never a copy
of the operator's own kubeconfig. The token is never printed and never reaches
argv: it goes into a 0600 file and into the PUT body FILE (--data @file), and the
script refuses to run under `set -x` because xtrace would print it.
Two traps this repo already paid for are designed around:
* GITEA_ is a RESERVED secret-name prefix (HTTP 400 "invalid secret name"), so
the name is CONFORMANCE_TRIGGER_KUBECONFIG -- which is also what gate.yml and
merge-gate.yaml already reference.
* `curl -sS` exits 0 on an HTTP error. Every write checks %{http_code} AND reads
the object back. The secret read-back proves presence by NAME only (Gitea
never returns secret values) and the script says so rather than implying its
content was checked; the variable read-back asserts the value is exactly
"true", which is what gate.yml compares against.
PRECONDITIONS ARE SPLIT IN TWO, and the split is the security argument:
mint-blocking (cluster) -- SA/Role/RoleBinding present, and ClusterPolicy
conformance-trigger-job-shape present + Enforce + failurePolicy Fail + Ready.
Any failure and NOTHING IS MINTED. Kubernetes places no `use` check on a Job's
serviceAccountName, so a `create jobs` token in that namespace IS the privileged
conformance-runner identity unless admission bounds its shape. RBAC cannot close
that (plan §4); minting before the boundary is live hands CI the escalation the
boundary exists to close.
arm-blocking (repo) -- digest-pinned CONFORMANCE_RUNNER_IMAGE, an initContainer
plus the admitted image in chainsaw-job.yaml, and CONFORMANCE_TRIGGER_KUBECONFIG
referenced in ci/conformance.sh CODE rather than a comment. These block only
vars.CLUSTER_AVAILABLE, because storing the secret early is harmless (the job is
gated off) and is the plan's own order.
All three arm-blocking checks FAIL in the tree today, so a live run would store
the secret and then REFUSE to arm CLUSTER_AVAILABLE, naming each missing item.
That is the intended behaviour: measured 2026-09-29, CLUSTER_AVAILABLE=true with
no trigger path scaffold-skipped all 54 tests at rc 0 and the stage printed "L2/L3
conformance passed". ci/conformance.sh now blocks that case, so arming early does
not enable L2/L3 -- it only breaks every PR. Arming needs the explicit flag, every
precondition green, and a typed confirmation on a TTY.
Before storing anything, the credential is PROVEN narrow: auth whoami must be
system:serviceaccount:conformance:conformance-trigger, a real `get jobs` round-trip
must succeed, and an auth can-i matrix asserts both halves -- granted (create/get
jobs, get pods, get pods/log) and DENIED (delete jobs, create pods/exec, get
secrets, create jobs in default, get nodes, '*' '*' --all-namespaces). A mismatch
aborts before any store, and says explicitly not to widen
conformance/incluster/rbac.yaml to make it pass.
Verified only by `bash -n` (clean) and reading. The Gitea variables API shape
(POST creates, PUT updates, body field `value` not `data`) is from documentation,
not measurement: the script tries POST then PUT, prints the response body on
failure, and treats the read-back as the authority. Kyverno readiness is accepted
from either .status.ready or a Ready condition, and "neither present" blocks
rather than assuming ready.
Gate (pinned toolbox): MERGE GATE: UNDETERMINED (rc 3) -- spec-immutability and
gitea-settings unrun, the expected local result.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
222 lines
12 KiB
Markdown
222 lines
12 KiB
Markdown
# infra/gitea — platform settings that are NOT GitOps-managed
|
|
|
|
Gitea's own repo settings (branch protection, required status checks, approvals) live
|
|
in Gitea's database, not in a manifest Flux reconciles. They are therefore **drift-prone
|
|
and invisible to the gate** — exactly how the state below went unnoticed.
|
|
|
|
## The gap this closes
|
|
|
|
Verified against the live API on 2026-09-29, against a `DECISIONS-LOG.md` entry
|
|
(2026-08-31) asserting the opposite:
|
|
|
|
| Claimed | Measured |
|
|
|---|---|
|
|
| `main` is a protected branch, gate is a required status check | `branch_protections` → `[]`; `protected:false`, `enable_status_check:false`, `required_approvals:0`, `user_can_push:true` |
|
|
| Runner `incluster-dind` online | **zero** runners registered at repo, org and site level |
|
|
|
|
So `main` accepted direct pushes, and the gate could not run at all. `CLAUDE.md` is the
|
|
rule; this directory is how the rule gets enforced by the machine.
|
|
|
|
## `branch-protection.sh`
|
|
|
|
Blocks direct pushes to `main`. Run it, then run the push probe it prints — **verify,
|
|
do not trust the API response**, since Gitea may let site admins bypass protection.
|
|
|
|
./infra/gitea/branch-protection.sh
|
|
|
|
It authenticates with the credential `git` already has stored for this host, and never
|
|
prints the secret.
|
|
|
|
## `runner-token-rotate.sh`
|
|
|
|
Mints a fresh instance-level registration token from the Gitea API and writes it into
|
|
`apps/wave-1/gitea-actions-runner-secret.sops.yaml` via `sops set`. **The token value is
|
|
never printed.**
|
|
|
|
./infra/gitea/runner-token-rotate.sh
|
|
|
|
A fresh token is usually **not sufficient on its own**. `act_runner` registers exactly once
|
|
and persists its identity in `/data/.runner` on the `act-runner-data` PVC. If Gitea no
|
|
longer holds a matching runner row, the pod starts, sees `.runner`, **skips registration**,
|
|
and polls with an identity Gitea does not recognise — so it never appears in the runner
|
|
list. That is precisely the observed symptom: zero runners, pod running, no error anyone
|
|
looks at. Clearing the PVC is what forces a real re-registration; the script prints the
|
|
commands.
|
|
|
|
## Sequence — DONE 2026-09-29
|
|
|
|
All four steps are complete and each was verified, not assumed:
|
|
|
|
| Step | State |
|
|
|---|---|
|
|
| 1. Runner registered | ✅ `incluster-dind` online. Root cause of its absence was a stale `/data/.runner` identity; clearing the PVC forced re-registration |
|
|
| 2. Gate observed green on a real PR | ✅ `gate / gate (pull_request)` ~1m35s |
|
|
| 3. `enable_status_check: true` | ✅ context `gate / gate (pull_request)` — that one ONLY |
|
|
| 4. Collaborator + approvals | ✅ `jakman85` is owner; `required_approvals: 1` |
|
|
|
|
Direct push to `main` was **provoked and rejected** (`pre-receive hook declined`) as
|
|
`gitea-admin`, a site admin — so push protection is not merely declared, it holds for the
|
|
account agents authenticate as.
|
|
|
|
### 5. Block the admin override — the step the first pass MISSED
|
|
|
|
`enable_status_check` and `required_approvals` were set on 2026-09-29 and **still did not
|
|
bite**, because `block_admin_merge_override` was `false`. Gitea lets repo admins override
|
|
merge requirements without it, and `jakman85` holds `owner` — so the two settings that looked
|
|
like the gate were overridable by precisely the account that merges.
|
|
|
|
**The evidence was sitting in plain sight:** PRs #5 through #9 all merged with
|
|
`required_approvals: 1` in force and **zero approvals recorded on any of them**. Checking the
|
|
*branch* view (`/branches/main`) hid this — it does not expose the field. Read the rule object
|
|
(`/branch_protections/main`) and enumerate its keys.
|
|
|
|
> **Do not probe for field names you assume exist.** This Gitea (1.24.3) has no
|
|
> `apply_to_admins`; asking `jq` for it returns `null`, which reads like "set to false" but
|
|
> means "key absent". Enumerate with `to_entries` and act on what is really there.
|
|
|
|
Now `block_admin_merge_override: true`. **That is still only a setting, not evidence** — it is
|
|
proven when a green PR is REFUSED a merge until approved, and not before.
|
|
|
|
### `require-gate.sh` / `verify-then-enforce.sh`
|
|
|
|
`verify-then-enforce.sh` is the re-runnable form of the whole safe order. It merges nothing —
|
|
merging is a human's reviewed act — but it forces the Flux reconcile and then **aborts** unless
|
|
the runner pod is healthy, a runner is online, and a real workflow goes green on it, before
|
|
enabling enforcement and probing the push. `require-gate.sh` is the enforcement step alone.
|
|
|
|
Re-run `verify-then-enforce.sh` after any future change to the act-runner Deployment.
|
|
|
|
### `expected-branch-protection.json` + `ci/gitea-settings.sh` — the detector
|
|
|
|
`expected-branch-protection.json` **declares** the desired rule. `ci/gitea-settings.sh` runs as
|
|
a merge-gate stage, reads the LIVE rule, and **fails the gate on any divergence** — plus asserts
|
|
at least one runner is online. It is the only thing in this repository that can see Gitea's own
|
|
database.
|
|
|
|
It reads `/branch_protections/main`, not `/branches/main`: the branch view does **not** expose
|
|
`block_admin_merge_override`, and it showed all-green for hours while the rule was overridable.
|
|
|
|
Behaviour, proven by test rather than assumed:
|
|
|
|
| Situation | Exit | Effect |
|
|
|---|---|---|
|
|
| live matches the declaration | 0 | green |
|
|
| any key drifted, or no runner online | 1 | **blocks the merge** |
|
|
| no `GITEA_SETTINGS_TOKEN`, or Gitea unreachable | 3 | skips **loudly**; never a pass, never a block |
|
|
|
|
The skip is deliberately *not* a block: an unwired credential blocking every PR is the
|
|
lock-yourself-out failure this repo keeps hitting. Wire the token with `set-ci-token.sh`, confirm
|
|
the stage goes **green on a real PR**, and only then set `REQUIRE_GITEA_SETTINGS=true` in
|
|
`gate.yml` so a future missing token blocks instead of skipping.
|
|
|
|
**Changing the declaration is how you change policy.** If the live state is what you now want,
|
|
edit this JSON in the same commit — deliberately, in review — rather than letting the two drift.
|
|
|
|
## `conformance-trigger-kubeconfig.sh` — the narrow L2/L3 trigger credential
|
|
|
|
Implements `docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 6 (= §5 step 7): mint a
|
|
ServiceAccount token for `conformance/conformance-trigger`, wrap **only that token** in a
|
|
kubeconfig, store it as the Actions secret `CONFORMANCE_TRIGGER_KUBECONFIG`, and — separately,
|
|
and only on request — set `vars.CLUSTER_AVAILABLE=true`.
|
|
|
|
./infra/gitea/conformance-trigger-kubeconfig.sh --dry-run # measure only; writes nothing
|
|
./infra/gitea/conformance-trigger-kubeconfig.sh # mint + store the secret
|
|
./infra/gitea/conformance-trigger-kubeconfig.sh --arm-cluster-available # + the variable
|
|
|
|
Never a cluster-admin kubeconfig, never a copy of the operator's own. The token is never
|
|
printed: it goes from the TokenRequest API into a 0600 file, into the request **body file**
|
|
(`--data @file`, so not even into `ps`), and nowhere else. The script refuses to run under
|
|
`set -x`.
|
|
|
|
**Its preconditions are split in two, and the split is the point.**
|
|
|
|
| Class | Checks | Blocks |
|
|
|---|---|---|
|
|
| **mint-blocking** (cluster) | SA/Role/RoleBinding exist; ClusterPolicy `conformance-trigger-job-shape` is present, `Enforce`, `failurePolicy: Fail`, Ready | the **mint** — nothing is created |
|
|
| **arm-blocking** (repo) | digest-pinned `CONFORMANCE_RUNNER_IMAGE`; `chainsaw-job.yaml` has an init-container and the admitted image; `ci/conformance.sh` names the trigger kubeconfig in **code, not a comment** | only `CLUSTER_AVAILABLE` |
|
|
|
|
Why mint-blocking: Kubernetes puts **no `use` check on a Job's `serviceAccountName`**, so a
|
|
token that can `create jobs` in `conformance` can run arbitrary code as the privileged
|
|
`conformance-runner` SA. RBAC cannot close that; the Kyverno policy is the boundary. Minting
|
|
while it is absent or in `Audit` hands CI the privileged identity, so the script will not do it.
|
|
|
|
Why arm-blocking is separate: storing the secret before the runner image / Job template / CI
|
|
trigger flow exist is harmless (the job is gated off), and is the plan's intended order. So
|
|
running this today provisions the secret and then **refuses to arm the variable**, naming the
|
|
missing item.
|
|
|
|
**`CLUSTER_AVAILABLE` is not a "turn conformance on" switch.** Measured 2026-09-29: set with no
|
|
trigger path, `run.sh` scaffold-skipped all 54 tests at rc 0 and `ci/conformance.sh` printed
|
|
"L2/L3 conformance passed" — a fake green on an enforcing required check. `ci/conformance.sh`
|
|
now blocks that case, so arming early does not enable L2/L3; it only breaks every PR. Hence:
|
|
`--arm-cluster-available` plus all preconditions green plus a typed confirmation on a TTY, or
|
|
nothing happens.
|
|
|
|
Both writes are verified the way `set-ci-token.sh` learned to: check `%{http_code}` **and**
|
|
read the object back. For the secret that proves presence by name only (Gitea never returns
|
|
secret values); for the variable it proves the value is exactly `true`, which is what
|
|
`gate.yml` compares against.
|
|
|
|
The token **expires** — it is a TokenRequest token, and the script decodes the JWT `exp` claim
|
|
to report the real expiry rather than trusting `--duration` (the API server caps it). Re-run to
|
|
rotate. After expiry the trigger gets 401 and the stage fails loudly rather than skipping.
|
|
|
|
`--server` exists because the embedded API-server URL must be reachable **from the CI runner
|
|
pod** — the VIP — which is not necessarily the address your workstation uses. The script
|
|
refuses a loopback server outright.
|
|
|
|
### Why these scripts exist at all, and what is still missing
|
|
|
|
Gitea's repo settings live in **its own database**, not in anything Flux reconciles or the
|
|
conformance gate can read. Nothing detects drift from what this directory declares. That is
|
|
the root cause of the entire 2026-09-29 session: branch protection was off and the runner was
|
|
gone for a month while the log recorded both as working, and **no check existed that could
|
|
have caught it.**
|
|
|
|
These scripts encode the desired state and can re-assert it; `ci/gitea-settings.sh` above is the
|
|
**detector** that turns divergence from a silent, invisible failure into a loud one. It closes the
|
|
root cause of the 2026-09-29 session.
|
|
|
|
It is only as good as its wiring: **until `GITEA_SETTINGS_TOKEN` exists it skips**, and a skipped
|
|
detector detects nothing. That is the remaining step.
|
|
|
|
## Historical sequence (kept for context)
|
|
|
|
### Original ordering notes
|
|
|
|
1. **Get an Actions runner registered** — `runner-token-rotate.sh`, then clear the PVC so
|
|
the runner actually re-registers, then confirm it shows `online` in the API. Do not
|
|
skip the confirmation: the whole reason this went unnoticed is that a runner was
|
|
*recorded* as online without anyone checking.
|
|
2. **Watch the gate actually report on a real PR**, and note the context name it produced.
|
|
3. **Only then set `enable_status_check: true`** with that exact context.
|
|
4. Add the human reviewer as a repo collaborator, then raise `required_approvals` to 1.
|
|
|
|
> **Do not do step 3 before step 1.** A required status check with no runner leaves every
|
|
> PR permanently pending — including the PR that would fix it. That half is measured: PR #1
|
|
> sat `pending` for 30.6 minutes across 122 polls with no runner registered, and never
|
|
> reported.
|
|
|
|
### Measured, once the runner came up (2026-09-29)
|
|
|
|
The first CI jobs this repo ever ran, on PR #1 and PR #2:
|
|
|
|
| Context | Result | Time |
|
|
|---|---|---|
|
|
| `gate / gate (pull_request)` | success | 1m58s (PR #1), 1m34s (PR #2) |
|
|
| `gate / gate (push)` | success | 1m32s (PR #2 only) |
|
|
| `gate / conformance (pull_request)` | **success — "Has been skipped"** | — |
|
|
|
|
**Require `gate / gate (pull_request)`. Do NOT require either `conformance` context —
|
|
but not for the reason an earlier draft of this file gave.** A skipped Gitea Actions job
|
|
does not hang a status check: Gitea reports it as **`success`** with the description
|
|
"Has been skipped". So requiring it would not deadlock anything — it would pass
|
|
*vacuously*, every time, forever, while nothing ran. Under HARD RULE 4 that is worse than
|
|
a deadlock: a deadlock is loud, a permanently-green check that verifies nothing is
|
|
exactly the fail-open shape this repo keeps finding.
|
|
|
|
The `(push)` contexts appear only for branches matching `gate.yml`'s push trigger
|
|
(`feature/**`, `fix/**`, `docs/**`, `chore/**`). `feat/**` is **not** in that list, which
|
|
is why PR #1's `feat/` branch produced only `pull_request` contexts. Require only the
|
|
`pull_request` context, or PRs from non-matching prefixes will never satisfy the rule.
|