From fe862ab527afd539f63b5b926b126a2b024938bd Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 29 Sep 2026 00:43:47 -0500 Subject: [PATCH] infra(gitea): recover the Actions runner before anything requires the gate No runner is registered at repo, org or site level, so gate.yml has never executed. PR #1 sat pending for 30.6 minutes across 122 polls and never reported. Requiring the gate as a status check in that state would lock the repo permanently. Adds runner-token-rotate.sh: mints a fresh instance-level registration token from the Gitea API and writes it into the SOPS secret via sops set, without ever printing the value. Records the likely root cause, which a new token alone does NOT fix. act_runner registers once and persists its identity in /data/.runner on the act-runner-data PVC. If Gitea no longer holds a matching runner row, the pod starts, sees .runner, SKIPS registration entirely, and polls with an identity Gitea does not recognise. Symptom: zero runners, pod running, nothing obviously broken. Clearing the PVC is what forces a genuine re-registration. The README now sequences runner-first, enforcement-second, and requires CONFIRMING the runner reports online rather than assuming it. That assumption is the whole defect: DECISIONS-LOG recorded the runner as online on 2026-08-31 and nobody checked since. Co-Authored-By: Claude Opus 5 (1M context) --- infra/gitea/README.md | 24 ++++++++++-- infra/gitea/runner-token-rotate.sh | 61 ++++++++++++++++++++++++++++++ 2 files changed, 81 insertions(+), 4 deletions(-) create mode 100755 infra/gitea/runner-token-rotate.sh diff --git a/infra/gitea/README.md b/infra/gitea/README.md index 411de5c..d8655e0 100644 --- a/infra/gitea/README.md +++ b/infra/gitea/README.md @@ -27,12 +27,28 @@ do not trust the API response**, since Gitea may let site admins bypass protecti It authenticates with the credential `git` already has stored for this host, and never prints the secret. +## `runner-token-rotate.sh` + +Mints a fresh instance-level registration token from the Gitea API and writes it into +`apps/wave-1/gitea-actions-runner-secret.sops.yaml` via `sops set`. **The token value is +never printed.** + + ./infra/gitea/runner-token-rotate.sh + +A fresh token is usually **not sufficient on its own**. `act_runner` registers exactly once +and persists its identity in `/data/.runner` on the `act-runner-data` PVC. If Gitea no +longer holds a matching runner row, the pod starts, sees `.runner`, **skips registration**, +and polls with an identity Gitea does not recognise — so it never appears in the runner +list. That is precisely the observed symptom: zero runners, pod running, no error anyone +looks at. Clearing the PVC is what forces a real re-registration; the script prints the +commands. + ## Sequence — order matters -1. **Get an Actions runner registered.** It is GitOps-managed - (`apps/wave-1/gitea-actions.yaml`, `apps/wave-1/gitea-actions-runner-secret.sops.yaml`), - so it is deployed but not registering — suspect a stale registration token in that - SOPS secret. +1. **Get an Actions runner registered** — `runner-token-rotate.sh`, then clear the PVC so + the runner actually re-registers, then confirm it shows `online` in the API. Do not + skip the confirmation: the whole reason this went unnoticed is that a runner was + *recorded* as online without anyone checking. 2. **Watch the gate actually report on a real PR**, and note the context name it produced. 3. **Only then set `enable_status_check: true`** with that exact context. 4. Add the human reviewer as a repo collaborator, then raise `required_approvals` to 1. diff --git a/infra/gitea/runner-token-rotate.sh b/infra/gitea/runner-token-rotate.sh new file mode 100755 index 0000000..b1d6dfd --- /dev/null +++ b/infra/gitea/runner-token-rotate.sh @@ -0,0 +1,61 @@ +#!/usr/bin/env bash +# Mint a fresh Gitea Actions registration token and write it into the SOPS secret. +# +# The token value is NEVER printed. It goes from the Gitea API into `sops set` and +# nowhere else. (It is briefly visible in this host's process list as a sops argument; +# acceptable on a single-operator workstation, and far better than echoing it.) +# +# This ALONE may not bring the runner back -- see "IMPORTANT" at the end of the output. +set -euo pipefail +HOST=gitea.home.keithjackson.dev +SECRET=apps/wave-1/gitea-actions-runner-secret.sops.yaml +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +cd "$REPO_ROOT" +[ -f "$SECRET" ] || { echo "not in the repo root, or $SECRET is missing" >&2; exit 1; } + +creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)" +U="$(sed -n 's/^username=//p' <<<"$creds")" +P="$(sed -n 's/^password=//p' <<<"$creds")" +[ -n "$U" ] && [ -n "$P" ] || { echo "no stored git credential for $HOST" >&2; exit 1; } +echo "==> minting an instance-level registration token as '${U}'" + +TOKEN="$(curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \ + "https://${HOST}/api/v1/admin/runners/registration-token" | jq -r '.token // empty')" +[ -n "$TOKEN" ] || { echo "Gitea did not return a token" >&2; exit 1; } +echo "==> got a ${#TOKEN}-character token (value not shown)" + +echo "==> writing it into ${SECRET} via sops" +DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run \ + sops set "$SECRET" '["stringData"]["token"]' "\"${TOKEN}\"" +unset TOKEN + +echo "==> done. Verify the file is still encrypted (no plaintext token):" +grep -q 'ENC\[' "$SECRET" && echo " OK - stringData is still SOPS-encrypted" \ + || { echo " REFUSING TO CONTINUE: $SECRET is not encrypted" >&2; exit 1; } + +cat <<'NOTE' + +Next: commit this on a branch, open a PR, and merge it so Flux applies the new Secret. + +IMPORTANT -- a new token is usually NOT sufficient on its own. +act_runner registers exactly ONCE and then persists its identity in /data/.runner on the +act-runner-data PVC. If Gitea no longer has a matching runner row (DB restore, the row was +deleted, the instance was rebuilt), the pod starts, sees .runner, SKIPS registration, and +polls with an identity Gitea does not recognise -- so it never appears in the runner list, +which is exactly the observed symptom (zero runners, pod running). + +So after the Secret lands, force a genuine re-registration by clearing that state: + + kubectl --context k3s-prod -n gitea-actions scale deploy/act-runner --replicas=0 + kubectl --context k3s-prod -n gitea-actions delete pvc act-runner-data + kubectl --context k3s-prod -n gitea-actions scale deploy/act-runner --replicas=1 + +(The PVC holds only runner registration state -- no build artifacts, nothing durable. +Deleting it is the supported way to re-register. Flux recreates it from the manifest.) + +Then confirm it actually registered -- do not assume: + + curl -s -u '' https://gitea.home.keithjackson.dev/api/v1/admin/actions/runners | jq + +A runner must appear with status "online" and the labels ubuntu-latest / homelab. +NOTE