Compare commits

...
Author SHA1 Message Date
jakman85andClaude Opus 5 e72050c690 docs: correct CI/platform documentation that was stale, wrong, or load-bearing
gate / gate (push) Successful in 1m37s
gate / gate (pull_request) Successful in 1m36s
gate / conformance (push) Has been skipped
gate / conformance (pull_request) Has been skipped
The 2026-09-29 session found the merge gate did not exist -- no branch protection, no
runner -- while the tree asserted both. This corrects the documents that carried, and in
several cases CAUSED, that belief.

CLAUDE.md gains section 2b: never infer platform state from this repository. It carries
the specific API and kubectl checks for protection, runners, CI status, workloads and
Flux revisions, and the rule that a green tick is not proof anything ran (a skipped Gitea
job reports success). This is the rule whose absence produced every other finding.

.gitea/workflows/README.md and merge-gate.yaml both said "there is no remote git host,
this file is dormant". Gitea had been live for weeks. Both were written as plans and never
revisited once the plan completed. The Transition section is now marked COMPLETE and says
what is genuinely still outstanding (CLUSTER_AVAILABLE unset, so L2/L3 never runs in CI).

DECISIONS-LOG 2026-08-31 is annotated IN PLACE. Anyone reading chronologically previously
hit "required status check; runner incluster-dind online" with nothing marking it false
until a month later. The correction now sits on the bullet itself.

ci/README.md, clusters/README.md and both conformance READMEs carry a status banner: they
speak as though no cluster exists, which makes a live L2/L3 coverage gap read as an
expected phase. That framing is why the gap went unexamined.

apps/wave-1/gitea-actions-runner-secret.sops.yaml: fixed a wrong path, replaced advice to
`sops -d` into a terminal (which CLAUDE.md forbids) with the rotation script, and recorded
that a fresh token alone does not re-register a runner holding a stale /data/.runner.

apps/wave-0/plex-claim-secret.sops.yaml: the header claimed the file ships the placeholder.
It has not since the claim -- it holds the spent token, which is inert and deliberately
left in place.

catalog/plex.yaml and the plex secret both asserted a "~4-minute TTL" as fact. It was
uncited, upstream documents no expiry, and it was the basis for a since-disproven claim
that the pipeline was too slow to fit inside it. Measured: ~2m45s.

Both .sops.yaml files were verified to still decrypt after the comment edits
(`sops -d > /dev/null`), and render==golden confirms the catalog comment does not
propagate into the rendered manifest.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 02:14:26 -05:00
gitea-admin 27d0ef3324 Merge pull request 'chore(plex): fresh plex.tv claim token' (#4) from feat/plex-claim-token-013124 into main
gate / gate (push) Successful in 1m33s
gate / conformance (push) Has been skipped
2026-09-29 06:33:04 +00:00
jakman85 bbd4f682a6 chore(plex): fresh plex.tv claim token
gate / gate (pull_request) Successful in 1m35s
gate / conformance (pull_request) Has been skipped
Injected inside the plex.tv claim window per runbooks/plex-claim.md.
2026-09-29 01:31:24 -05:00
jakman85 0391139240 Merge pull request 'feat(plex): wire PLEX_CLAIM from a SOPS Secret (placeholder value)' (#1) from feat/plex-claim-sops-secret into main
gate / gate (push) Successful in 1m34s
gate / conformance (push) Has been skipped
Reviewed-on: #1
2026-09-29 06:18:14 +00:00
jakman85 e9f441b5ce Merge pull request 'docs(gitea): a skipped job reports SUCCESS, not pending' (#3) from fix/conformance-context-is-fail-open into main
gate / gate (push) Successful in 1m34s
gate / conformance (push) Has been skipped
Reviewed-on: #3
2026-09-29 06:12:30 +00:00
jakman85andClaude Opus 5 234cc6a71a docs(gitea): a skipped job reports SUCCESS, not pending -- correct the deadlock claim
gate / gate (push) Successful in 1m33s
gate / gate (pull_request) Successful in 1m32s
gate / conformance (push) Has been skipped
gate / conformance (pull_request) Has been skipped
The runner came up on 2026-09-29 and the first CI jobs this repo has ever run went
green, which falsified a claim merged in #2.

MEASURED:
  gate / gate (pull_request)          success   1m58s (PR 1), 1m34s (PR 2)
  gate / gate (push)                  success   1m32s (PR 2 only)
  gate / conformance (pull_request)   success   "Has been skipped"

#2 said listing the conformance context as a required check would deadlock every PR,
because the job is `if: false` and never runs. That is wrong. Gitea reports a skipped
job as SUCCESS. Requiring it would not deadlock anything -- it would pass VACUOUSLY,
forever, while nothing ran.

The conclusion is unchanged (never require it) but the reason inverts, and the corrected
reason matters more: under HARD RULE 4 a permanently-green check that verifies nothing is
worse than a deadlock. A deadlock is loud. This is the exact fail-open shape the
2026-09-18 audit and this one both keep finding.

Also records that the (push) contexts only appear for branches matching gate.yml's push
trigger -- feature/**, fix/**, docs/**, chore/** -- and NOT feat/**, which is why PR 1's
feat/ branch produced only pull_request contexts. Require the pull_request context alone,
or PRs from non-matching prefixes never satisfy the rule.

CLAUDE.md's enforcement table is updated: the runner half is genuinely fixed and observed
green, the branch-protection half is not. Adds the warning that a green tick is not by
itself evidence that anything ran.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 01:07:13 -05:00
jakman85 69a0d1a1c7 Merge pull request 'docs: delivery contract for agents + record that the merge gate does not exist' (#2) from docs/ci-delivery-flow-for-agents into main
gate / gate (push) Successful in 1m34s
gate / conformance (push) Has been skipped
Reviewed-on: #2
2026-09-29 05:50:31 +00:00
jakman85 d3a0d587a4 Added gitea action runner secret
gate / gate (push) Successful in 1m32s
gate / gate (pull_request) Successful in 1m34s
gate / conformance (push) Has been skipped
gate / conformance (pull_request) Has been skipped
2026-09-29 00:50:02 -05:00
jakman85andClaude Opus 5 fe862ab527 infra(gitea): recover the Actions runner before anything requires the gate
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
gate / gate (pull_request) Successful in 1m32s
gate / conformance (pull_request) Has been skipped
No runner is registered at repo, org or site level, so gate.yml has never executed.
PR #1 sat pending for 30.6 minutes across 122 polls and never reported. Requiring the
gate as a status check in that state would lock the repo permanently.

Adds runner-token-rotate.sh: mints a fresh instance-level registration token from the
Gitea API and writes it into the SOPS secret via sops set, without ever printing the
value.

Records the likely root cause, which a new token alone does NOT fix. act_runner
registers once and persists its identity in /data/.runner on the act-runner-data PVC.
If Gitea no longer holds a matching runner row, the pod starts, sees .runner, SKIPS
registration entirely, and polls with an identity Gitea does not recognise. Symptom:
zero runners, pod running, nothing obviously broken. Clearing the PVC is what forces a
genuine re-registration.

The README now sequences runner-first, enforcement-second, and requires CONFIRMING the
runner reports online rather than assuming it. That assumption is the whole defect:
DECISIONS-LOG recorded the runner as online on 2026-08-31 and nobody checked since.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 00:43:47 -05:00
jakman85andClaude Opus 5 a2665c0511 infra(gitea): script the branch protection that was never applied
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
gate / gate (pull_request) Successful in 1m33s
gate / conformance (pull_request) Has been skipped
Gitea repo settings live in its own database, not in a manifest Flux reconciles, so
they are drift-prone and invisible to the conformance gate -- which is how main came
to accept direct pushes while the log recorded it as protected.

Adds infra/gitea/branch-protection.sh (blocks direct pushes to main) and a README
carrying the measured before-state and the ORDER the remaining steps must happen in.

The script deliberately leaves enable_status_check:false and required_approvals:0.
Both would deadlock today: there is no runner to satisfy a required context, and the
only human account is not yet a repo collaborator, so no one could supply an approval.
The README sequences both.

It prints a push probe to run afterwards. The API response is not proof -- Gitea may
let site admins bypass protection, and that has to be observed, not assumed.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 00:20:12 -05:00
jakman85andClaude Opus 5 5a9e455149 docs: delivery contract for agents + record that the merge gate does not exist
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
gate / gate (pull_request) Successful in 1m33s
gate / conformance (pull_request) Has been skipped
Going to land the plex claim wiring through the documented PR flow surfaced that
the flow has never run. PR #1 is the first pull request this repo has ever had.

Verified against the live Gitea, not inferred from the docs:
  * branch_protections is [] -- main is protected:false, enable_status_check:false,
    status_check_contexts:[], required_approvals:0, user_can_push:true
  * ZERO Actions runners registered at repo, org and site level

DECISIONS-LOG 2026-08-31 asserts a "required status check" and a runner
"incluster-dind online". Both are false of the live system. The gate is neither
enforced nor able to run, and that went unnoticed because the log claimed
otherwise -- the same fail-open class as the two holes found on 2026-09-18.

Adds CLAUDE.md, the first agent-facing delivery doc here: the
branch -> PR -> gate -> approval -> merge rule, stated alongside the honest fact
that enforcement is currently weaker than the rule, so agents follow the rule
rather than what they can get away with. Carries the toolbox gate invocation, the
two git-worktree workarounds, the rc 0/1/3 contract, and secret handling.

Records the ordering trap: enabling a required status check before a runner exists
deadlocks every PR permanently, including the one that would fix it.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 00:11:58 -05:00
14 changed files with 548 additions and 37 deletions

No files matched your search

+28 -14
View File
@@ -40,9 +40,15 @@ and ADR-0011: **CI holds no broad kubeconfig.** L2/L3 run as an in-cluster **Cha
*collects* its artifacts, using a narrow trigger credential referenced **by secret name only**
(`secrets.CONFORMANCE_TRIGGER_KUBECONFIG`) — never a plaintext secret, never a cluster-admin kubeconfig.
## Honest SKIP — never fake green (Phase 0)
## Honest SKIP — never fake green
**No cluster exists in Phase 0.** The gate is deliberately built so unavailable checks **SKIP loudly,
**NOTE (2026-09-29):** this section is written in the past's voice — "Phase 0", "no cluster
exists". **A cluster has existed for weeks** and Flux has been reconciling against it. The
skip logic below is still correct and still honest, but read it as describing a **live
coverage gap** (L2/L3 is not wired into CI) rather than an expected phase. Its being
phrased as a phase is part of why the gap went unexamined.
Originally: The gate is deliberately built so unavailable checks **SKIP loudly,
they do not pass**:
- The `conformance` job is `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}` — with no cluster it is
@@ -77,21 +83,29 @@ Both dirs are git-ignored (see [`.gitignore`](../../.gitignore)).
## Triggers & branch model (CONTRIBUTING.md)
- `pull_request` → `main`: **PRs are the enforced merge gate into `main`.**
- `push` → `feature/**`, `fix/**`, `docs/**`, `chore/**`: fast feedback on the branch. **No work on `main`.**
- `push` → `main` only. The per-branch prefix list was removed: `act_runner` runs at capacity 1,
so a branch push fires a redundant run that serialises *ahead* of the PR run for the same tree.
**No work on `main`** — and since 2026-09-29 that is enforced, not conventional: `main` is a
protected branch and a direct push is rejected by the pre-receive hook (verified, not assumed).
## Transition: generic-CI-now → Gitea-Actions-after-Gitea-is-up
## Transition: COMPLETE (2026-09-29)
Per [`CONTRIBUTING.md`](../../../home-network-k3s/CONTRIBUTING.md) and
[`docs/05` §"local bootstrap → self-hosted Gitea"](../../../home-network-k3s/docs/05-gitops-and-delivery.md):
This section used to describe a migration as pending. It has happened. Gitea is live at
`gitea.home.keithjackson.dev`, the runner `incluster-dind` is registered and online, and
`gate.yml` runs on every PR.
1. **Now (interim):** there is **no remote git host** on this network. This workflow file is dormant —
but the **identical gate runs locally today** via `make -C ci gate` (R-TEST-02). Development is
local feature-branch → `main`, the gate enforced by convention + the local runner.
2. **Gitea comes online early** (one of the first services onboarded) at `gitea.home.keithjackson.dev`.
3. **CI moves to Gitea Actions:** this workflow activates unchanged. The only operational steps are
enabling a Gitea Actions runner and setting `vars.CLUSTER_AVAILABLE=true` +
`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` once a dev/test cluster exists. PRs then become the
**platform-enforced** merge gate — the same no-work-in-main rule, now enforced by machine.
**What took a month to notice:** the migration was recorded as done on 2026-08-31 without
anyone checking the platform. In reality `main` had **no branch protection** and **zero
runners were registered** until 2026-09-29. Every "the gate enforces this" statement in
between rested on an assertion nobody verified. PR #1 — the first pull request this
repository ever had — sat `pending` for 30.6 minutes because nothing existed to run it.
So: **do not read this file, or the DECISIONS-LOG, as evidence about the live platform.**
Query the API. `infra/gitea/` holds the scripts and the measured state.
Still genuinely outstanding: `vars.CLUSTER_AVAILABLE` and
`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` are unset, so L2/L3 conformance does not run in
CI. That is a real coverage gap, not a phase.
## Configuration (set in Gitea; referenced by name only)
+8 -5
View File
@@ -11,11 +11,14 @@
# v
# merge-gate decision [R-TEST-03: touched subset green]
#
# Transition (CONTRIBUTING.md / docs/05 §"local bootstrap -> self-hosted Gitea"):
# Today there is no Gitea host, so this file is dormant. The SAME gate is runnable
# now via `make -C ci gate` (R-TEST-02). Once Gitea is self-hosted and Gitea
# Actions is enabled, this workflow becomes the ENFORCED PR merge gate into main —
# no code change required, only turning on the runner and setting CLUSTER_AVAILABLE.
# STATUS (corrected 2026-09-29): Gitea IS hosted and Gitea Actions IS enabled; the runner
# `incluster-dind` is online and gate.yml is the live PR gate. The text that used to sit here
# ("Today there is no Gitea host, so this file is dormant") was stale by weeks -- it was written
# as a plan and never revisited once the plan completed.
#
# THIS file is still dormant, but for the reason in the DORMANT block below -- its spec checkout
# and L2/L3 seams are unwired -- NOT because there is no host. Do not read a stale plan as a
# statement about the live platform; query the API instead (see infra/gitea/).
#
# CI holds NO broad kubeconfig. L2/L3 run as an in-cluster Chainsaw Job whose own
# scoped ServiceAccount holds cluster access (docs/14, R-SEC-02); CI uses only a
+146
View File
@@ -0,0 +1,146 @@
# CLAUDE.md — how work lands in this repo
Read this before doing anything durable. It is the delivery contract for AI coding
sessions and agents. `DECISIONS-LOG.md`'s last entry is the current PM handoff and
takes precedence on *what* to work on; this file governs *how* it lands.
## 1. Never push to `main`. Ever.
Every durable change reaches `main` exactly one way:
branch -> push the branch -> open a PR -> CI gate green -> human approval -> merge
There is no other path. Specifically:
- **Do not `git push origin main`.**
- **Do not `git push origin HEAD:main`.** Same thing wearing a hat.
- **Do not merge your own PR.** A human approves and clicks merge.
- **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
> **Many runbooks under `runbooks/` still say `git push origin main` IS the deploy.**
> That wording is **stale** — it predates the PR flow. It describes what Flux does
> (main is what the cluster follows), not what you are permitted to do. Honour this
> file, not that line. Fixing those runbooks is tracked work.
## 2. Enforcement is currently WEAKER than this rule. Follow the rule anyway.
Verified against the live Gitea on 2026-09-29:
| Expected | Actual |
|---|---|
| `main` protected, direct push rejected | **No branch protection.** `protected: false`, `user_can_push: true` |
| PR blocked until the gate passes | **`enable_status_check: false`**, `status_check_contexts: []` |
| Gate runs on every PR | **Fixed 2026-09-29.** Runner `incluster-dind` is online; `gate / gate (pull_request)` runs and passes in ~1m30s–2m |
| Approval required before merge | **`required_approvals: 0`** |
So nothing mechanically stops you from pushing to `main` right now. **That is a gap
being closed, not permission.** `DECISIONS-LOG.md` (2026-08-31) claimed the gate was a
"required status check" with a runner online; neither was true, which is how this went
unnoticed for a month. The runner half is now genuinely fixed and observed green. The
protection half is not — see `infra/gitea/`.
**Consequence for you:** CI now runs, but it does not yet *block* anything. Always run
the toolbox gate yourself and quote its numbers. Never describe a PR as "green" on the
strength of a check you have not seen report — and note that a **skipped** Gitea job
reports as `success`, so a green tick is not by itself evidence that anything ran.
## 2b. Never infer platform state from this repository
**This repo is not evidence about the live platform.** Neither is `DECISIONS-LOG.md`.
Both describe what someone believed at the time of writing, and both have been
confidently wrong for a month at a stretch.
On 2026-09-29 the log stated the merge gate was a required status check with a runner
online. The API said `branch_protections: []` and **zero** runners. Everything downstream
of that belief — "the gate enforces this", "your push would be rejected", "CI covers it" —
was wrong, and nothing in the tree could have revealed it, because the tree was the source
of the error.
**So, before asserting that any gate, protection, runner, secret, or workload is in a
given state: query the thing itself.**
| Claim about... | Check |
|---|---|
| branch protection / required checks | `GET /api/v1/repos/{owner}/{repo}/branches/main` |
| a runner exists | `GET /api/v1/admin/actions/runners` |
| CI actually ran | the commit's `/statuses` — and read the **description**; a *skipped* Gitea job reports `success` with "Has been skipped" |
| a workload / Secret / env | `kubectl --context k3s-prod ...` (read-only is fine for agents) |
| Flux applied your commit | compare `.status.lastAppliedRevision` against the GitRepository's artifact revision |
Quote the measured field, not the prose. If you cannot measure it, say so explicitly —
"unverified" is a legitimate report, and a far better one than a confident guess that
matches the docs.
**A green tick is not proof anything ran.** Read what actually executed.
## 3. Gate before you push
./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
Exit codes are a contract — do not flatten them:
| rc | Meaning |
|---|---|
| 0 | PASS |
| 1 | BLOCKED — something failed |
| 3 | UNDETERMINED — a stage did not run. **Never report this as green.** |
`spec-immutability` reaches rc 3 whenever the sibling spec repo is not checked out
beside this one; that single skip is expected and tolerated. Any *other* skip is a
real gap in your run.
**Never gate on the host.** `promtool` and `kustomize` are not on the host `$PATH`,
so checks SKIP and you get a false rc 3. Only the pinned toolbox result counts.
### Running the gate from a git worktree
Two unrelated failures, both with known workarounds — put both in any agent brief:
1. **`dev.sh` refuses to start:** `infra/ansible/.cache/prod.kubeconfig` is untracked
and exists only in the main checkout. Do **not** copy it (it is a credential).
Use `DEV_KUBECONFIG=/dev/null` — the gate contacts no cluster, so L2/L3 scaffold-skip
normally.
2. **Every `git` call inside the container fails:** a worktree's `.git` is a pointer to a
gitdir outside the mounted root. `dev.sh` takes no extra-mount flag, so inject one via
the `ENGINE` hook — a shim that intercepts `run` and re-execs
`podman run -v /path/to/main/.git:/path/to/main/.git:ro "$@"`. A traceback in
`tracked_files()` is this bug, not a broken tree.
## 4. Branch naming
Use `feat/`, `fix/`, `docs/`, `chore/`. Note that `.gitea/workflows/gate.yml`'s **push**
trigger lists `feature/**` — not `feat/**` — so branch pushes get no fast-feedback run.
The `pull_request` trigger fires regardless of branch name, so the PR gate is unaffected.
## 5. Cluster access
`kubectl` does not work on the operator workstation; the toolchain lives only in the
pinned container. Always name the context explicitly:
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
`dev.sh` merges both clusters and picks a current-context **per invocation**; that
default has already changed once. Never rely on it. Do not run two `dev.sh`
invocations concurrently.
**Mutating `kubectl` and `flux reconcile` are operator actions, not agent actions.**
## 6. Secrets
- **Never `sops -d` a file into a terminal, a log, or a transcript.**
- **Never print a Secret's `.data`/`.stringData`** when checking whether it exists —
list key names or object names only. A password has been leaked into a transcript
this way before.
- Encrypted files are `*.sops.yaml` or live under `secrets/`. The root `.sops.yaml` is
the single creation-rules file; do not add a second one in a subtree.
- A short-lived credential (e.g. a plex.tv claim token, ~minutes) cannot survive the PR
flow. Do not design around committing one — read the expiry from the issuing page and
treat a burnt token as free to retry.
## 7. Before you finish
Append a handoff entry to `DECISIONS-LOG.md` describing what you changed, what you
verified, and what you did **not** verify. State gaps plainly — an honest
"UNVERIFIED" is worth more than an optimistic "green", and this repo has already been
burned by the difference.
+72
View File
@@ -1192,6 +1192,13 @@ fresh agent should read this entry + `infra/PHASE1-STATUS.md` + `docs/plans/` an
Platform-admins require TOTP MFA.
- **Gitea Actions CI is LIVE + enforcing**: `.gitea/workflows/gate.yml` runs `make -C ci gate` on push/PR to main;
required status check; runner `incluster-dind` online (ns gitea-actions).
> **[CORRECTION, 2026-09-29 — THIS BULLET WAS FALSE WHEN WRITTEN AND STAYED FALSE FOR A MONTH.]**
> Measured against the live Gitea API on 2026-09-29: `branch_protections` returned `[]` (`main`
> `protected:false`, `enable_status_check:false`, `user_can_push:true`) and **zero** Actions runners
> were registered at repo, org or site level. The gate was neither enforced nor able to run. PR #1
> — the first pull request this repo ever had — was opened on 2026-09-29 and sat `pending` for
> 30.6 minutes with nothing to execute it. Both halves of this bullet were asserted, not verified.
> See the 2026-09-29 entries for the root cause and the fix.
### What changed this session (durable)
1. **SSO cluster-wide break→fix — root cause Cilium CVE-2025-30162.** The 1.16→1.20 security fix makes the shared
@@ -4716,3 +4723,68 @@ asleep. `git worktree remove` preserves branches and their commits, so only that
6. **Five `public` services have no working public path** — waiting on WAN forwarding, deliberately.
7. Backlog cherry-picks: `feat/egress-gateway-staleness-alert`, two `rescued/*` fixes, two rendered
manifests nothing applies (the keycloak one would stand up a conflicting instance).
## 2026-09-29 — The CI/CD merge gate does not exist: no runner, no branch protection. **PM HANDOFF.**
Went to land the plex claim wiring through the documented PR flow and discovered the flow has
never run. **PR #1 is the first pull request this repository has ever had.** It is still queued.
### What is actually true of the live Gitea (verified via API, 2026-09-29)
| Claimed (DECISIONS-LOG 2026-08-31) | Measured |
|---|---|
| "required status check" on `main` | `branch_protections` returns `[]`. `protected: false`, `enable_status_check: false`, `status_check_contexts: []`, `required_approvals: 0`, `user_can_push: true` |
| "runner `incluster-dind` online (ns gitea-actions)" | **Zero runners registered** at repo, org and site level |
Both halves of that entry are false. Nothing enforces the gate, and the gate cannot run.
PR #1's two contexts (`gate / gate`, `gate / conformance`) sat `pending` for 14+ minutes with
nothing to execute them, and will sit there indefinitely.
This is a **fail-open governance hole**, the same class as the two found on 2026-09-18 — and it
was invisible precisely because the log asserted the opposite. Every "the gate is enforcing"
statement since 2026-08-31 rested on an unverified claim.
### Ordering trap — read before fixing
**Do not enable required status checks until a runner is live.** Requiring
`gate / gate (pull_request)` with no runner deadlocks every PR permanently, including the PR
that would fix it. Sequence: runner first, verify the gate actually runs and passes on a real
PR, *then* turn on enforcement with the context name CI actually reported.
Related: `gate.yml`'s `conformance` job is `if: false` by design (honest skip, never fake green),
but it still publishes a `gate / conformance (pull_request)` context. If that context is ever
listed as required, it deadlocks every PR for the same reason. Wire it or exclude it.
### Delivery contract written down
Added `CLAUDE.md` at the repo root — the first agent-facing delivery doc here. It states the
branch -> PR -> gate -> approval -> merge rule, records that enforcement is currently weaker than
the rule (so agents follow the rule rather than what they can get away with), and carries the
toolbox-gate invocation, the two git-worktree workarounds, the rc 0/1/3 contract, and the secret
handling rules.
It also flags that **37 of 39 runbooks still instruct `git push origin main`**. Those were
accurate before the PR flow and are stale now. Correcting them is tracked, not done.
### Also found, not fixed
- `gate.yml`'s push trigger lists `feature/**`; this repo's branches use `feat/**`, so branch
pushes get no fast-feedback run. PR trigger is unaffected.
- The gitea-actions runner is GitOps-managed (`apps/wave-1/gitea-actions.yaml` +
`gitea-actions-runner-secret.sops.yaml`), so it is deployed but not registering — most likely a
stale registration token in that SOPS secret. Needs in-cluster diagnosis (operator).
- **Plex is still unclaimed and it was never checked** whether it is already claimed. Do that
before spending a token — `runbooks/plex-claim.md` §First commands.
- `runbooks/plex-claim.md` asserts a "~4-minute TTL" for a plex.tv claim token in four places,
**uncited**, and the whole stopwatch procedure is built on it. Upstream `pms-docker`'s README
says nothing about expiry. The number is plausible community folklore, not a measured fact.
The authoritative reading is the countdown rendered on plex.tv/claim at issue time.
### Next steps
1. **Operator:** diagnose the gitea-actions runner; get one registered.
2. **Operator:** enable branch protection on `main` — push blocked, approvals required — and only
then add the gate as a required status check.
3. Land `feat/plex-claim-sops-secret` (PR #1) once the gate can actually run.
4. Claim plex, after checking it is not already claimed.
5. Correct the 37 runbooks' `git push origin main` wording.
+17 -9
View File
@@ -7,12 +7,20 @@
# substitution). A braced token quoted from the upstream shell script would be a live hazard here,
# not prose, so the upstream conditions below are described in words.
#
# THE VALUE SHIPPED WITH THIS FILE IS A PLACEHOLDER, DELIBERATELY — the literal string
# `replace-me-with-a-fresh-plex-tv-claim-token`, not a credential. A https://plex.tv/claim token has
# a ~4-minute TTL, far shorter than commit -> push -> Flux reconcile -> pod-roll, so no token
# committed here is ever still valid when the pod starts. This file exists to make the Secret — and
# therefore the env reference — ALWAYS RESOLVABLE IN THE FLUX GRAPH, not to carry a working token.
# The real token is injected by the operator inside that 4-minute window: runbooks/plex-claim.md.
# THIS FILE NO LONGER HOLDS THE PLACEHOLDER. Corrected 2026-09-29: the server was CLAIMED on that
# date, and the value here is the SPENT token from that claim. It is inert and safe to leave --
# upstream gates the exchange on PlexOnlineToken being ABSENT from /config, and it is now present on
# the plex-config PVC, so this value is a NO-OP on every start whatever it holds. Deleting the Secret
# would NOT be safe (see below), which is why a spent token stays here rather than being blanked.
#
# The claim token is SHORT-LIVED -- minutes. Read the countdown that plex.tv/claim renders when it
# issues one; do NOT budget against a number written in this tree. Earlier revisions of this comment
# asserted a "~4-minute TTL" as established fact with no citation, and upstream plexinc/pms-docker
# documents no expiry at all. That uncited number was load-bearing for the whole procedure design.
#
# This file exists to make the Secret — and therefore the env reference — ALWAYS RESOLVABLE IN THE
# FLUX GRAPH, not to carry a working token. Re-claiming (only needed after a /config wipe) is
# scripted: infra/plex/claim-ship.sh, measured at ~2m45s end to end.
#
# WHY SHIPPING A PLACEHOLDER IS SAFE (verified against plexinc/pms-docker branch `master`, the
# s6-overlay v2 script root/etc/cont-init.d/40-plex-first-run, cross-checked byte-identical against
@@ -46,7 +54,7 @@ metadata:
app.kubernetes.io/name: plex
type: Opaque
stringData:
PLEX_CLAIM: ENC[AES256_GCM,data:8O9nvFw+6OR+53mXvHfe321aSmYUQcZuAqLKZpcSlUwNyeU+Nbp/tRPBlg==,iv:EbvvOsXlXjAFOewnWbK28ErmUdWcJJN0Q5Gm53El3ZM=,tag:wXaInGFQ+wqUt765vvjCDA==,type:str]
PLEX_CLAIM: ENC[AES256_GCM,data:nLmzeUBWRhRjUakrGiGzQpQuqZNcIYvRH6M=,iv:z0on/dArsNejNLrp3V1nSiNLFzKcREhPuMO1634CwQw=,tag:Ih23CT1JYna/5lfYQUb1Dw==,type:str]
sops:
kms: []
gcp_kms: []
@@ -62,8 +70,8 @@ sops:
QUtYdHEvN042ajNoNktOLzlYQmQrc1kK4NHozMbXVzNGYbUBRbK6a9Uw/sVPAfe6
RW5KWzZ4I/hloqjN3zzVISoSMieMkBGDnF5qZS5N6FqIc6/HrsDD8A==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-09-29T03:25:20Z"
mac: ENC[AES256_GCM,data:u5uVmOdXN78UaKUMkPZzqeW84222+tcG57rFKIKcvRKwJkzMTOuSFOeQpKh4eK4KZXVKK7+Q3mFf5G8a+jjuKjYbjSCpR38AF3TJv3xrj8fhnqR+8QHRp8tsgA0jJZBS55N7KKDyOztLFo49sI7mNnlrUwhWTQUkbLgN1LxF5MA=,iv:P4W9G2mI/3VTkvSrDZvnkJ0Qxb1nFmXcF35CrSNhako=,tag:yVsg7xsUALUoCJH0cHfQjw==,type:str]
lastmodified: "2026-09-29T06:31:16Z"
mac: ENC[AES256_GCM,data:62BmBAq45CD2g6LDyx+Ho4bTfrhJcaro/RB/iM+OaRc4a/D83MhjbcBL11Uz/O726inqhkvCKDJ/fFemoc50JDbtC79cYMaxrGJNjFM0sdckWrowPvh6ie7HHHUBRBaefl+gNfnc+pbvcncvRzwA4RyzcZse1wbxKfHN8FjEopE=,iv:h0cMShs3tIgnY6eruUYymYrcLFzQZLk/uXQJ/Jsd5ek=,tag:nZ9Pau7p4xWT1QCN1vv2vA==,type:str]
pgp: []
encrypted_regex: ^(data|stringData)$
version: 3.9.4
@@ -4,7 +4,13 @@
# against Gitea. Flux decrypts it via the sops-age key at reconcile (apps Kustomization
# decryption is wired in clusters/dev/kustomization.yaml). Rotate by resetting the token
# in Gitea (admin -> actions -> runners) and re-encrypting a fresh value here.
# Recover/inspect: infra/devcontainer/dev.sh run sops -d apps/gitea-actions-runner-secret.sops.yaml
# ROTATE: ./infra/gitea/runner-token-rotate.sh -- it mints a fresh instance-level token from the
# Gitea API and writes it here via `sops set`, without the value ever being printed.
# Do NOT `sops -d` this file into a terminal, a log, or an agent transcript. To check only that
# it still decrypts, redirect: `sops -d <file> > /dev/null`.
# NOTE: a fresh token alone may not re-register the runner. act_runner registers ONCE and keeps
# its identity in /data/.runner on the act-runner-data PVC; if Gitea no longer holds a matching
# row it skips registration silently. Clearing that PVC is what forces a real re-registration.
apiVersion: v1
kind: Secret
metadata:
@@ -14,7 +20,7 @@ metadata:
app.kubernetes.io/name: act-runner
type: Opaque
stringData:
token: ENC[AES256_GCM,data:W48mEtBqGeukX13lMhrhrer+zB21cLVp0acy3m1Xy/oo89DqO8FX/Q==,iv:wfTUU7RiFLdEc1fmEFS9DfE2SmGCFLPK17m0IKspft0=,tag:UhoXA47p7386KHcYNbPzCg==,type:str]
token: ENC[AES256_GCM,data:f574mAUkLvMGq0TYycwCd4tduMuBgQLiTg/140kuRDgFUYYm1ZoSjQ==,iv:Tn7gFWBLm2hHk/BbfVX7qIZegwZp9ah1cMou5i2O91E=,tag:TkdSiMI4sdCNn2JuN++Pig==,type:str]
sops:
kms: []
gcp_kms: []
@@ -30,8 +36,8 @@ sops:
MDZLUE52QzJXbFlrNWxOcVhJbDNnY00KxCpFkbhhCuk4Fa8CC3C4+R9AcsUnaXQR
UjAfwKFoXAneXUih7TTXMbR/6+VgQBww5Bmwpn7jVLnzP0N0pIZVTA==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-08-26T04:48:14Z"
mac: ENC[AES256_GCM,data:b99/zkL332RIT3FYBa1sR3o9+pwlmrUgKvVHawbjWYSJZ7ERUfO+LXuhKJ0HKrp6i1JfkurZlRO27iBcfrBisAXS9KLaqmsCJf4kKfFc4skANrmKCzprrXUmZexr1uiE8eDUzjNCLNxSCZfnFR6d/AkMXJqi6Hsj9/O+wH2lWsc=,iv:3DIPszSdxK81ERO3o/nTs33qQYHaAKLNcGP4qlZViFc=,tag:/B6VKkzMUwh0byNUicS5wQ==,type:str]
lastmodified: "2026-09-29T05:48:35Z"
mac: ENC[AES256_GCM,data:nobqr794nrdRYFrd8Ct8bw9M8NtftJXawRGk9zrL+pTWahwlWrwXVtOAxMGEPVojz3WemsxTJESzmzy0gcT1QcrjpYTsF2MVHcr6+umk8q9gAh2enyxy/0QkYiSt6XjvRVqfqleLanEkeIq1FPf2zhSLSJaiw4/oZTfkCHD1/Zg=,iv:JxaP0htmGcvE4ZU1YAmGF4A6JkejV29NNygyMl1OKF8=,tag:G70NvcPyVuagXhugWoUfTw==,type:str]
pgp: []
encrypted_regex: ^(data|stringData)$
version: 3.9.4
+11 -5
View File
@@ -38,11 +38,17 @@ env:
# UNCLAIMED, so this is wired; it was previously left out on the reasoning below, and only one
# half of that reasoning survived contact with the upstream source.
#
# THE 4-MINUTE TTL IS STILL TRUE, and it is now the OPERATIONAL constraint rather than a reason to
# abstain. A https://plex.tv/claim token expires in ~4 minutes — far shorter than commit -> push ->
# Flux reconcile -> pod-roll if that cycle is left to its own timers. So the committed value is a
# documented PLACEHOLDER and the live token is injected by the operator inside that window, with an
# explicit `flux reconcile` to collapse the wait: runbooks/plex-claim.md is the timed procedure.
# A SHORT TTL IS THE OPERATIONAL CONSTRAINT, not a reason to abstain. A https://plex.tv/claim token
# expires in MINUTES — read the countdown the claim page renders; do not budget against a number
# written in this tree. (This comment previously asserted "~4 minutes" as established fact. It was
# UNCITED, upstream plexinc/pms-docker documents no expiry at all, and that number was load-bearing
# for the whole procedure design — including a since-disproven claim that the delivery pipeline was
# too slow to fit inside it. Measured 2026-09-29: the full loop is ~2m45s and fits.)
#
# The live token is injected by the operator inside that window, with an explicit `flux reconcile`
# to collapse the wait. Scripted end to end as infra/plex/claim-ship.sh; runbooks/plex-claim.md is
# the procedure. SERVER CLAIMED 2026-09-29 — this env is now a permanent no-op (see below), and the
# committed value is the spent token from that claim rather than the original placeholder.
#
# ONE-SHOT, AND SAFE TO SIT HERE EXPIRED. Verified from plexinc/pms-docker's own first-run script
# (branch `master`, root/etc/cont-init.d/40-plex-first-run, confirmed byte-identical to the file in
+14
View File
@@ -1,5 +1,19 @@
# `ci/` — portable merge gate (runnable today, without Gitea)
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
> This document speaks as though no cluster exists and the platform is scaffolding. **A
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
> coverage hole, not a phase.
>
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
> documentation rather than measurement, and `main` was unprotected with zero runners
> registered the whole time. **Query the live API before asserting platform state.**
**Task T-0.2 · R-TEST-02 (reproducible, single-command) · Owner WS-GITOPS (CI).**
The same merge gate the Gitea Actions workflow runs
+14
View File
@@ -1,5 +1,19 @@
# clusters/
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
> This document speaks as though no cluster exists and the platform is scaffolding. **A
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
> coverage hole, not a phase.
>
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
> documentation rather than measurement, and `main` was unprotected with zero runners
> registered the whole time. **Query the live API before asserting platform state.**
Flux entrypoints and the environment-overlay model (**R-ENV-02**, **R-GITOPS-01/02/03**).
Governing spec: [`docs/05-gitops-and-delivery.md`](../../home-network-k3s/docs/05-gitops-and-delivery.md),
+14
View File
@@ -1,5 +1,19 @@
# conformance/ — the home-k3s conformance harness (WS-TEST)
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
> This document speaks as though no cluster exists and the platform is scaffolding. **A
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
> coverage hole, not a phase.
>
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
> documentation rather than measurement, and `main` was unprotected with zero runners
> registered the whole time. **Query the live API before asserting platform state.**
This subtree is the **definition of "done"** for the whole platform. Nothing is accepted
on a demo, a screenshot, or "it reconciled": every requirement is proven by an
**automated, reproducible, behavior-based test** tagged with the requirement ID(s) it
+14
View File
@@ -1,5 +1,19 @@
# In-cluster conformance Job (L2/L3)
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
> This document speaks as though no cluster exists and the platform is scaffolding. **A
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
> coverage hole, not a phase.
>
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
> documentation rather than measurement, and `main` was unprotected with zero runners
> registered the whole time. **Query the live API before asserting platform state.**
Per **ADR-0011** and **docs/14 "Where each layer runs"**: L0/L1 run in CI without a
cluster, but **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job**
on the dev/test cluster that **CI triggers and waits for**. CI holds **no** cluster
+82
View File
@@ -0,0 +1,82 @@
# infra/gitea — platform settings that are NOT GitOps-managed
Gitea's own repo settings (branch protection, required status checks, approvals) live
in Gitea's database, not in a manifest Flux reconciles. They are therefore **drift-prone
and invisible to the gate** — exactly how the state below went unnoticed.
## The gap this closes
Verified against the live API on 2026-09-29, against a `DECISIONS-LOG.md` entry
(2026-08-31) asserting the opposite:
| Claimed | Measured |
|---|---|
| `main` is a protected branch, gate is a required status check | `branch_protections` → `[]`; `protected:false`, `enable_status_check:false`, `required_approvals:0`, `user_can_push:true` |
| Runner `incluster-dind` online | **zero** runners registered at repo, org and site level |
So `main` accepted direct pushes, and the gate could not run at all. `CLAUDE.md` is the
rule; this directory is how the rule gets enforced by the machine.
## `branch-protection.sh`
Blocks direct pushes to `main`. Run it, then run the push probe it prints — **verify,
do not trust the API response**, since Gitea may let site admins bypass protection.
./infra/gitea/branch-protection.sh
It authenticates with the credential `git` already has stored for this host, and never
prints the secret.
## `runner-token-rotate.sh`
Mints a fresh instance-level registration token from the Gitea API and writes it into
`apps/wave-1/gitea-actions-runner-secret.sops.yaml` via `sops set`. **The token value is
never printed.**
./infra/gitea/runner-token-rotate.sh
A fresh token is usually **not sufficient on its own**. `act_runner` registers exactly once
and persists its identity in `/data/.runner` on the `act-runner-data` PVC. If Gitea no
longer holds a matching runner row, the pod starts, sees `.runner`, **skips registration**,
and polls with an identity Gitea does not recognise — so it never appears in the runner
list. That is precisely the observed symptom: zero runners, pod running, no error anyone
looks at. Clearing the PVC is what forces a real re-registration; the script prints the
commands.
## Sequence — order matters
1. **Get an Actions runner registered** — `runner-token-rotate.sh`, then clear the PVC so
the runner actually re-registers, then confirm it shows `online` in the API. Do not
skip the confirmation: the whole reason this went unnoticed is that a runner was
*recorded* as online without anyone checking.
2. **Watch the gate actually report on a real PR**, and note the context name it produced.
3. **Only then set `enable_status_check: true`** with that exact context.
4. Add the human reviewer as a repo collaborator, then raise `required_approvals` to 1.
> **Do not do step 3 before step 1.** A required status check with no runner leaves every
> PR permanently pending — including the PR that would fix it. That half is measured: PR #1
> sat `pending` for 30.6 minutes across 122 polls with no runner registered, and never
> reported.
### Measured, once the runner came up (2026-09-29)
The first CI jobs this repo ever ran, on PR #1 and PR #2:
| Context | Result | Time |
|---|---|---|
| `gate / gate (pull_request)` | success | 1m58s (PR #1), 1m34s (PR #2) |
| `gate / gate (push)` | success | 1m32s (PR #2 only) |
| `gate / conformance (pull_request)` | **success — "Has been skipped"** | — |
**Require `gate / gate (pull_request)`. Do NOT require either `conformance` context —
but not for the reason an earlier draft of this file gave.** A skipped Gitea Actions job
does not hang a status check: Gitea reports it as **`success`** with the description
"Has been skipped". So requiring it would not deadlock anything — it would pass
*vacuously*, every time, forever, while nothing ran. Under HARD RULE 4 that is worse than
a deadlock: a deadlock is loud, a permanently-green check that verifies nothing is
exactly the fail-open shape this repo keeps finding.
The `(push)` contexts appear only for branches matching `gate.yml`'s push trigger
(`feature/**`, `fix/**`, `docs/**`, `chore/**`). `feat/**` is **not** in that list, which
is why PR #1's `feat/` branch produced only `pull_request` contexts. Require only the
`pull_request` context, or PRs from non-matching prefixes will never satisfy the rule.
+57
View File
@@ -0,0 +1,57 @@
#!/usr/bin/env bash
# Enable branch protection on main for home-network-k3s-impl.
#
# Sets enable_push:false -> no direct pushes to main; changes land only by merging a PR.
#
# enable_status_check is deliberately FALSE. There is currently NO Gitea Actions runner
# registered (verified 2026-09-29: zero at repo, org and site level), so requiring the
# gate context would deadlock EVERY PR permanently -- including the PR that fixes it.
# Turn it on only after a runner is live and you have seen the gate actually report on
# a real PR, using the context name CI itself produced.
#
# required_approvals is 0 for the same class of reason: jakman85 is a site admin but NOT
# a repo collaborator, so requiring an approval now could leave nobody able to approve.
# Add the collaborator first, then raise this to 1.
set -euo pipefail
HOST=gitea.home.keithjackson.dev
OWNER=Home-Network
REPO=home-network-k3s-impl
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"
P="$(sed -n 's/^password=//p' <<<"$creds")"
[ -n "$U" ] && [ -n "$P" ] || { echo "no stored credential for $HOST" >&2; exit 1; }
echo "authenticating as '${U}' (secret not shown)"
echo "==> creating branch protection on 'main'"
jq -n '{
branch_name: "main",
enable_push: false,
enable_status_check: false,
status_check_contexts: [],
required_approvals: 0,
block_on_rejected_reviews: true,
block_on_outdated_branch: false,
dismiss_stale_approvals: true
}' | curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
-X POST -H 'Content-Type: application/json' --data @- \
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections" \
| jq -r 'if .branch_name then "created" else "API said: \(.message // tostring)" end'
echo "==> verifying (this is the part that matters)"
curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \
| jq -r '{protected, user_can_push, required_approvals, enable_status_check, status_check_contexts}'
cat <<'NOTE'
Now prove it actually blocks you -- do not take the API's word for it. From a clean
checkout on main, with nothing to lose:
git commit --allow-empty -m 'protection probe'
git push origin main # EXPECT: rejected by the remote
git reset --hard HEAD~1 # tidy up either way
If that push SUCCEEDS, Gitea is letting site admins bypass protection and this is not
yet doing what you want.
NOTE
+61
View File
@@ -0,0 +1,61 @@
#!/usr/bin/env bash
# Mint a fresh Gitea Actions registration token and write it into the SOPS secret.
#
# The token value is NEVER printed. It goes from the Gitea API into `sops set` and
# nowhere else. (It is briefly visible in this host's process list as a sops argument;
# acceptable on a single-operator workstation, and far better than echoing it.)
#
# This ALONE may not bring the runner back -- see "IMPORTANT" at the end of the output.
set -euo pipefail
HOST=gitea.home.keithjackson.dev
SECRET=apps/wave-1/gitea-actions-runner-secret.sops.yaml
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
cd "$REPO_ROOT"
[ -f "$SECRET" ] || { echo "not in the repo root, or $SECRET is missing" >&2; exit 1; }
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"
P="$(sed -n 's/^password=//p' <<<"$creds")"
[ -n "$U" ] && [ -n "$P" ] || { echo "no stored git credential for $HOST" >&2; exit 1; }
echo "==> minting an instance-level registration token as '${U}'"
TOKEN="$(curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
"https://${HOST}/api/v1/admin/runners/registration-token" | jq -r '.token // empty')"
[ -n "$TOKEN" ] || { echo "Gitea did not return a token" >&2; exit 1; }
echo "==> got a ${#TOKEN}-character token (value not shown)"
echo "==> writing it into ${SECRET} via sops"
DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run \
sops set "$SECRET" '["stringData"]["token"]' "\"${TOKEN}\""
unset TOKEN
echo "==> done. Verify the file is still encrypted (no plaintext token):"
grep -q 'ENC\[' "$SECRET" && echo " OK - stringData is still SOPS-encrypted" \
|| { echo " REFUSING TO CONTINUE: $SECRET is not encrypted" >&2; exit 1; }
cat <<'NOTE'
Next: commit this on a branch, open a PR, and merge it so Flux applies the new Secret.
IMPORTANT -- a new token is usually NOT sufficient on its own.
act_runner registers exactly ONCE and then persists its identity in /data/.runner on the
act-runner-data PVC. If Gitea no longer has a matching runner row (DB restore, the row was
deleted, the instance was rebuilt), the pod starts, sees .runner, SKIPS registration, and
polls with an identity Gitea does not recognise -- so it never appears in the runner list,
which is exactly the observed symptom (zero runners, pod running).
So after the Secret lands, force a genuine re-registration by clearing that state:
kubectl --context k3s-prod -n gitea-actions scale deploy/act-runner --replicas=0
kubectl --context k3s-prod -n gitea-actions delete pvc act-runner-data
kubectl --context k3s-prod -n gitea-actions scale deploy/act-runner --replicas=1
(The PVC holds only runner registration state -- no build artifacts, nothing durable.
Deleting it is the supported way to re-register. Flux recreates it from the manifest.)
Then confirm it actually registered -- do not assume:
curl -s -u '<user>' https://gitea.home.keithjackson.dev/api/v1/admin/actions/runners | jq
A runner must appear with status "online" and the labels ubuntu-latest / homelab.
NOTE