diff --git a/CLAUDE.md b/CLAUDE.md index 3e7739b..e9b7e83 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -23,22 +23,23 @@ There is no other path. Specifically: > (main is what the cluster follows), not what you are permitted to do. Honour this > file, not that line. Fixing those runbooks is tracked work. -## 2. Enforcement is currently WEAKER than this rule. Follow the rule anyway. +## 2. Enforcement (as of 2026-09-29: the rule is now MACHINE-ENFORCED) Verified against the live Gitea on 2026-09-29: | Expected | Actual | |---|---| -| `main` protected, direct push rejected | **No branch protection.** `protected: false`, `user_can_push: true` | -| PR blocked until the gate passes | **`enable_status_check: false`**, `status_check_contexts: []` | -| Gate runs on every PR | **Fixed 2026-09-29.** Runner `incluster-dind` is online; `gate / gate (pull_request)` runs and passes in ~1m30s–2m | -| Approval required before merge | **`required_approvals: 0`** | +| `main` protected, direct push rejected | ✅ `protected: true`, `user_can_push: false` — **verified by a real rejected push**, not by the API's word | +| PR blocked until the gate passes | ✅ `enable_status_check: true`, context `gate / gate (pull_request)` | +| Gate runs on every PR | ✅ runner `incluster-dind` online; ~1m35s | +| Approval required before merge | ✅ `required_approvals: 1` | -So nothing mechanically stops you from pushing to `main` right now. **That is a gap -being closed, not permission.** `DECISIONS-LOG.md` (2026-08-31) claimed the gate was a -"required status check" with a runner online; neither was true, which is how this went -unnoticed for a month. The runner half is now genuinely fixed and observed green. The -protection half is not — see `infra/gitea/`. +All four were FALSE as recently as 2026-09-29, while this repo's own log asserted they were +true. They are true now because each was measured, and the push rejection was provoked rather +than assumed. **Re-measure before relying on any of them** — see §2b. + +Still not covered: `vars.CLUSTER_AVAILABLE` is unset, so **L2/L3 conformance does not run in +CI**. A green PR therefore proves the L0/L1 offline subset, not runtime behaviour. **Consequence for you:** CI now runs, but it does not yet *block* anything. Always run the toolbox gate yourself and quote its numbers. Never describe a PR as "green" on the diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 4187c0a..deb0c8e 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -4870,3 +4870,57 @@ That theory was mine and it was wrong: the pipeline was never too slow, it simpl FOLLOW-UP note in `apps/wave-1/gitea-actions.yaml`. It deletes both hand-rolled wait loops and the envsubst hazard. Land it ALONE — it is the merge gate for the whole repo. 6. Create the Plex libraries over `/media/movies`, `/media/tv`, `/media/photos`. + +## 2026-09-29 (later) — the merge gate is now real, enforced, and proven. **PM HANDOFF.** + +Closes the gap opened by the earlier entry today. Eight PRs, all gated; every claim below was +measured against the live API or cluster, not inferred from this repository. + +### Final state + +| | Verified | +|---|---| +| `main` protected | `protected:true`, `user_can_push:false` — and a direct push as site-admin `gitea-admin` was **provoked and rejected** (`pre-receive hook declined`) | +| Gate required | `enable_status_check:true`, contexts `["gate / gate (pull_request)"]` | +| Approvals | `required_approvals:1`; `jakman85` holds `owner` | +| Runner | `incluster-dind` online, now a **native sidecar** (`initContainers`, `restartPolicy: Always`, TLS-verified `docker info` startupProbe) | +| Gate time | ~1m35s | +| Plex | claimed 2026-09-29; `PlexOnlineToken` present on the `plex-config` PVC | + +### What was corrected in the documentation + +`CLAUDE.md` §2b — *never infer platform state from this repository* — is the durable outcome. +The 2026-08-31 entry asserting an enforced gate is now annotated **in place**. The "Phase 0 / +no cluster exists" framing across `ci/`, `clusters/` and `conformance/` READMEs is banner-marked +as describing a live gap rather than a stage. All 37 runbooks now describe the PR flow. Two +`.sops.yaml` headers and `catalog/plex.yaml` were corrected (comment edits verified not to break +the SOPS MAC, and not to move the rendered golden). + +### A claim of mine that shipped wrong, and is now corrected in place + +PR #7 stated the sidecar conversion would remove the one-per-start dockerd line +`http: TLS handshake error from 127.0.0.1: EOF`. **It does not.** Exactly one still appears, +immediately after `API listen on [::]:2376` — it is now the startupProbe's first attempt, +landing microseconds after dockerd begins listening and cut off by `timeoutSeconds: 1`; the +next poll succeeds. Cosmetic, and deliberately not chased: raising the timeout to hide one +benign startup line would slow every subsequent poll. The manifest comment now says so. + +### The root cause nobody has fixed yet + +Gitea's repo settings live in its own database — invisible to Flux and unreadable by the +conformance gate. **No check exists that could have caught "protection off, runner gone."** +`infra/gitea/` declares and can re-assert the desired state, but it is not a detector. A +conformance check that reads the live Gitea API and fails on divergence is the highest-value +remaining work in this repo, and it is the difference between this having been fixed and this +being unable to recur. + +### Next steps + +1. **Write that Gitea-settings conformance check.** Everything else here is a symptom. +2. Wire `vars.CLUSTER_AVAILABLE` + `secrets.CONFORMANCE_TRIGGER_KUBECONFIG` so **L2/L3 runs in + CI**. A green PR currently proves the L0/L1 offline subset only — runtime behaviour is + unverified on every merge. +3. Re-run `infra/gitea/verify-then-enforce.sh` after any future act-runner change. +4. `required_approvals: 1` means a PR you open as `jakman85` needs `gitea-admin` to approve. + If that friction bites, either open agent work as `gitea-admin` and review as `jakman85`, or + drop back to relying on the status check alone. diff --git a/apps/wave-1/gitea-actions.yaml b/apps/wave-1/gitea-actions.yaml index 4da8b03..097bd65 100644 --- a/apps/wave-1/gitea-actions.yaml +++ b/apps/wave-1/gitea-actions.yaml @@ -118,8 +118,16 @@ spec: # and only THEN begins listening, so the port can accept a connection while the certs # are absent or incomplete -- and act_runner hard-exits on a missing # /certs/client/ca.pem. A full TLS-verified `docker info` proves BOTH that the certs - # are usable and that the daemon answers. It also removes the one-per-start - # "TLS handshake error ... EOF" that the old `nc -z` probe logged in dockerd. + # are usable and that the daemon answers. + # + # CORRECTION (measured 2026-09-29, after this conversion shipped): the commit that + # introduced this claimed the conversion would REMOVE the one-per-start dockerd line + # `http: TLS handshake error from 127.0.0.1: EOF`. IT DOES NOT. Exactly one still + # appears, immediately after "API listen on [::]:2376" -- it is now this probe's FIRST + # attempt, landing microseconds after dockerd begins listening and cut off by + # timeoutSeconds: 1. The next poll succeeds. It is cosmetic, exactly as cosmetic as it + # was before, and deliberately NOT chased: raising timeoutSeconds to hide one benign + # startup line would slow every subsequent poll. Left documented rather than silenced. # # periodSeconds 1 x failureThreshold 60 preserves the previous 60s cap. initContainers: diff --git a/infra/gitea/README.md b/infra/gitea/README.md index 4b9dbf7..345c741 100644 --- a/infra/gitea/README.md +++ b/infra/gitea/README.md @@ -43,7 +43,46 @@ list. That is precisely the observed symptom: zero runners, pod running, no erro looks at. Clearing the PVC is what forces a real re-registration; the script prints the commands. -## Sequence — order matters +## Sequence — DONE 2026-09-29 + +All four steps are complete and each was verified, not assumed: + +| Step | State | +|---|---| +| 1. Runner registered | ✅ `incluster-dind` online. Root cause of its absence was a stale `/data/.runner` identity; clearing the PVC forced re-registration | +| 2. Gate observed green on a real PR | ✅ `gate / gate (pull_request)` ~1m35s | +| 3. `enable_status_check: true` | ✅ context `gate / gate (pull_request)` — that one ONLY | +| 4. Collaborator + approvals | ✅ `jakman85` is owner; `required_approvals: 1` | + +Direct push to `main` was **provoked and rejected** (`pre-receive hook declined`) as +`gitea-admin`, a site admin — so protection is not merely declared, it holds for the account +agents authenticate as. + +### `require-gate.sh` / `verify-then-enforce.sh` + +`verify-then-enforce.sh` is the re-runnable form of the whole safe order. It merges nothing — +merging is a human's reviewed act — but it forces the Flux reconcile and then **aborts** unless +the runner pod is healthy, a runner is online, and a real workflow goes green on it, before +enabling enforcement and probing the push. `require-gate.sh` is the enforcement step alone. + +Re-run `verify-then-enforce.sh` after any future change to the act-runner Deployment. + +### Why these scripts exist at all, and what is still missing + +Gitea's repo settings live in **its own database**, not in anything Flux reconciles or the +conformance gate can read. Nothing detects drift from what this directory declares. That is +the root cause of the entire 2026-09-29 session: branch protection was off and the runner was +gone for a month while the log recorded both as working, and **no check existed that could +have caught it.** + +These scripts encode the desired state and can re-assert it, but they are not a detector. The +durable fix is a conformance check that reads the live Gitea API and fails when it diverges +from what is declared here — turning a class of silent, invisible failure into a loud one. +**That check does not exist yet and is the highest-value remaining work in this directory.** + +## Historical sequence (kept for context) + +### Original ordering notes 1. **Get an Actions runner registered** — `runner-token-rotate.sh`, then clear the PVC so the runner actually re-registers, then confirm it shows `online` in the API. Do not diff --git a/infra/gitea/require-gate.sh b/infra/gitea/require-gate.sh new file mode 100755 index 0000000..7490e15 --- /dev/null +++ b/infra/gitea/require-gate.sh @@ -0,0 +1,50 @@ +#!/usr/bin/env bash +# Items 1 + 2: make the gate a REQUIRED status check and require one approval. +# +# RUN THIS LAST. Prerequisites, in order: +# a) PR #7 (dind native sidecar) is MERGED, and +# b) you have SEEN a real workflow run go green on the converted pod. +# The runner Deployment is the merge gate for the whole repo. Turning on a required check +# while the runner is broken means no PR can merge -- including the revert. (Recoverable: +# an owner can set enable_status_check back to false. But do not rely on that.) +# +# Requires exactly ONE context: `gate / gate (pull_request)`. +# * NOT `gate / conformance (*)` -- that job is `if: false`, and a skipped Gitea job reports +# SUCCESS ("Has been skipped"). Requiring it would pass VACUOUSLY forever while nothing +# ran, which under HARD RULE 4 is worse than a deadlock: a deadlock is loud. +# * NOT the `(push)` variant -- after PR #6 there are no per-branch push runs at all, so a +# PR would never produce that context. +set -euo pipefail +HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl +creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)" +U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")" +api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; } + +echo "==> updating branch protection on main" +jq -n '{ + enable_push: false, + enable_status_check: true, + status_check_contexts: ["gate / gate (pull_request)"], + required_approvals: 1, + block_on_rejected_reviews: true, + dismiss_stale_approvals: true, + block_on_outdated_branch: false +}' | api -X PATCH -H 'Content-Type: application/json' --data @- \ + "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main" >/dev/null + +echo "==> verifying" +api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \ + | jq -r '{protected,user_can_push,enable_status_check,status_check_contexts,required_approvals}' + +cat <<'NOTE' + +NOTE ON required_approvals: 1 -- Gitea will not let an author approve their own PR. + * Agent-opened PRs (author gitea-admin) are approved by jakman85. Fine. + * PRs YOU open as jakman85 need gitea-admin to approve. If that is awkward, either open + agent work as gitea-admin and review as jakman85, or set required_approvals back to 0 + and rely on the status check alone. + +Then prove it, rather than trusting the API: + git commit --allow-empty -m probe && git push origin main # EXPECT: rejected + git reset --hard HEAD~1 +NOTE diff --git a/infra/gitea/verify-then-enforce.sh b/infra/gitea/verify-then-enforce.sh new file mode 100755 index 0000000..65b02ed --- /dev/null +++ b/infra/gitea/verify-then-enforce.sh @@ -0,0 +1,83 @@ +#!/usr/bin/env bash +# verify-then-enforce.sh — run this AFTER you have merged PRs 8, 5, 6, 7 in that order. +# +# It does NOT merge anything. Merging is your reviewed act (CLAUDE.md: a human approves and +# merges). This script does the mechanical half: prove the converted runner actually works, +# and only then turn on enforcement. +# +# WHY THIS ORDER. The act-runner Deployment IS the merge gate for this repository. Enabling a +# required status check while the runner is broken means NO PR can merge -- including the +# revert. So PR #7 lands while the gate is still advisory, is PROVEN green on a real run, and +# only then does enforcement go on. +# +# Aborts at the first checkpoint that does not pass. Safe to re-run. +set -euo pipefail +HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl +REPO_ROOT=/var/home/keith/source/home-network-k3s-impl +D="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +cd "$REPO_ROOT" +creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)" +U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")" +api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; } +A="https://${HOST}/api/v1/repos/${OWNER}/${REPO}" +say() { printf '\n\033[1m==> %s\033[0m\n' "$1"; } +die() { printf '\n\033[31mABORT: %s\033[0m\n' "$1" >&2; exit 1; } + +say "PRECHECK — all four PRs must be merged first" +for n in 8 5 6 7; do + m=$(api "$A/pulls/$n" | jq -r .merged) + printf ' PR #%s merged=%s\n' "$n" "$m" + [ "$m" = true ] || die "PR #$n is not merged yet. Merge 8, 5, 6, 7 in that order, then re-run." +done + +say "STEP 1 — force Flux to apply the sidecar now, not on the 10m interval" +./infra/devcontainer/dev.sh run bash -c \ + 'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-1 --with-source' + +say "STEP 2 — CHECKPOINT: the converted pod must come up healthy" +./infra/devcontainer/dev.sh run bash -c ' + kubectl --context k3s-prod -n gitea-actions rollout status deploy/act-runner --timeout=180s + echo "--- sidecar (initContainers) ---" + kubectl --context k3s-prod -n gitea-actions get deploy act-runner \ + -o jsonpath="{range .spec.template.spec.initContainers[*]}{.name} restartPolicy={.restartPolicy}{\"\n\"}{end}" + kubectl --context k3s-prod -n gitea-actions get pod -l app.kubernetes.io/name=act-runner' \ + || die "converted act-runner pod is not healthy -- REVERT PR #7 before anything else" + +say "STEP 3 — CHECKPOINT: a runner must be ONLINE again" +n=0 +for i in $(seq 1 40); do + n=$(api "https://${HOST}/api/v1/admin/actions/runners" | jq -r '[(.runners//[])[]|select(.status=="online")]|length') + printf '\r online runners: %s (%ss) ' "$n" "$((i*5))" + [ "${n:-0}" -ge 1 ] && { echo; break; } + sleep 5 +done +[ "${n:-0}" -ge 1 ] || die "no runner online after the conversion -- REVERT PR #7" + +say "STEP 4 — CHECKPOINT: a REAL workflow must go green ON THE CONVERTED POD" +MAIN_SHA=$(api "$A/branches/main" | jq -r .commit.id) +echo " watching the post-merge push run on main (${MAIN_SHA:0:8})..." +st="" +for i in $(seq 1 60); do + st=$(api "$A/commits/$MAIN_SHA/status" | jq -r .state) + printf '\r main gate: %-8s (%ss) ' "$st" "$((i*10))" + case "$st" in success) echo; break;; failure|error) die "gate FAILED on the converted pod -- REVERT PR #7";; esac + sleep 10 +done +[ "$st" = success ] || die "gate never completed on the converted pod -- REVERT PR #7; do NOT enable enforcement" +echo " the converted sidecar ran a real job green. Enforcement is now safe." + +say "STEP 5 — require the gate + one approval" +"$D/require-gate.sh" + +say "STEP 6 — prove a direct push is rejected (do not trust the API's word)" +git fetch -q origin && git checkout -q main && git reset -q --hard origin/main +git commit -q --allow-empty -m 'protection probe' +if git push origin main 2>&1 | grep -q 'protected branch'; then + echo " CONFIRMED: push to main rejected by the pre-receive hook." +else + echo " WARNING: the push was NOT rejected -- protection is not doing what it claims." +fi +git reset -q --hard origin/main + +say "FINAL STATE" +api "$A/branches/main" | jq -r '{protected,user_can_push,enable_status_check,status_check_contexts,required_approvals}' diff --git a/infra/plex/claim-ship.sh b/infra/plex/claim-ship.sh index 4dce82a..a3f035e 100755 --- a/infra/plex/claim-ship.sh +++ b/infra/plex/claim-ship.sh @@ -26,9 +26,10 @@ creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)" U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")" api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; } -# feat/ prefix ON PURPOSE. act_runner capacity is 1, and gate.yml's PUSH trigger covers -# feature/** fix/** docs/** chore/** but NOT feat/** — so a chore/ branch would fire a -# second, redundant run that SERIALISES ahead of the PR run and doubles the wall clock. +# Branch prefix is now cosmetic. It used to matter: gate.yml's push trigger covered +# feature/** fix/** docs/** chore/** but not feat/**, and with act_runner at capacity 1 a +# matching prefix fired a redundant run that serialised ahead of the PR run. Those +# per-branch push triggers were deleted, so no prefix produces a duplicate run any more. BRANCH="feat/plex-claim-token-$(date +%H%M%S)" el "committing on $BRANCH"