From b2d98839584ee843a6588746f429a3ae842d3cb1 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 29 Sep 2026 02:28:55 -0500 Subject: [PATCH] docs(ci): close out the hardening -- land the scripts, correct a claim that shipped wrong The branch-protection + PR-gate + sidecar sequence is complete and verified. This records the finished state and fixes what the sequence itself proved wrong. CORRECTS A FALSE CLAIM I SHIPPED. PR #7 said the sidecar conversion would remove the one-per-start dockerd line "http: TLS handshake error from 127.0.0.1: EOF". It does NOT. Exactly one still appears, immediately after "API listen on [::]:2376" -- it is now the startupProbe's FIRST attempt, landing microseconds after dockerd begins listening and cut off by timeoutSeconds: 1; the next poll succeeds. Cosmetic, and deliberately not chased: raising the timeout to hide one benign startup line would slow every subsequent poll. The manifest comment now states this instead of the wrong prediction. Lands infra/gitea/require-gate.sh and verify-then-enforce.sh, which drove the rollout from a session temp dir. verify-then-enforce.sh merges nothing -- merging is a human's reviewed act -- but forces the reconcile and ABORTS unless the pod is healthy, a runner is online, and a real workflow goes green on it, before enabling enforcement and probing the push. Re-run it after any future act-runner change. Trims the now-obsolete feat/ rationale from claim-ship.sh: per-branch push triggers were deleted, so no prefix produces a duplicate run any more. CLAUDE.md's enforcement table flips from "weaker than this rule" to machine-enforced, with all four rows marked verified and the push rejection noted as PROVOKED rather than assumed. It also now states what a green PR does NOT prove: CLUSTER_AVAILABLE is unset, so L2/L3 never runs in CI and only the L0/L1 offline subset is covered. infra/gitea/README.md marks the sequence done and names the root cause that remains unfixed: Gitea repo settings live in its own database, invisible to Flux and unreadable by the conformance gate, so NOTHING detects drift from what this directory declares. These scripts assert desired state; they are not a detector. The check that reads the live API and fails on divergence is the highest-value work left, and it is the difference between this being fixed and this being unable to recur. Co-Authored-By: Claude Opus 5 (1M context) --- CLAUDE.md | 21 ++++---- DECISIONS-LOG.md | 54 +++++++++++++++++++ apps/wave-1/gitea-actions.yaml | 12 ++++- infra/gitea/README.md | 41 ++++++++++++++- infra/gitea/require-gate.sh | 50 ++++++++++++++++++ infra/gitea/verify-then-enforce.sh | 83 ++++++++++++++++++++++++++++++ infra/plex/claim-ship.sh | 7 +-- 7 files changed, 252 insertions(+), 16 deletions(-) create mode 100755 infra/gitea/require-gate.sh create mode 100755 infra/gitea/verify-then-enforce.sh diff --git a/CLAUDE.md b/CLAUDE.md index 3e7739b..e9b7e83 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -23,22 +23,23 @@ There is no other path. Specifically: > (main is what the cluster follows), not what you are permitted to do. Honour this > file, not that line. Fixing those runbooks is tracked work. -## 2. Enforcement is currently WEAKER than this rule. Follow the rule anyway. +## 2. Enforcement (as of 2026-09-29: the rule is now MACHINE-ENFORCED) Verified against the live Gitea on 2026-09-29: | Expected | Actual | |---|---| -| `main` protected, direct push rejected | **No branch protection.** `protected: false`, `user_can_push: true` | -| PR blocked until the gate passes | **`enable_status_check: false`**, `status_check_contexts: []` | -| Gate runs on every PR | **Fixed 2026-09-29.** Runner `incluster-dind` is online; `gate / gate (pull_request)` runs and passes in ~1m30s–2m | -| Approval required before merge | **`required_approvals: 0`** | +| `main` protected, direct push rejected | ✅ `protected: true`, `user_can_push: false` — **verified by a real rejected push**, not by the API's word | +| PR blocked until the gate passes | ✅ `enable_status_check: true`, context `gate / gate (pull_request)` | +| Gate runs on every PR | ✅ runner `incluster-dind` online; ~1m35s | +| Approval required before merge | ✅ `required_approvals: 1` | -So nothing mechanically stops you from pushing to `main` right now. **That is a gap -being closed, not permission.** `DECISIONS-LOG.md` (2026-08-31) claimed the gate was a -"required status check" with a runner online; neither was true, which is how this went -unnoticed for a month. The runner half is now genuinely fixed and observed green. The -protection half is not — see `infra/gitea/`. +All four were FALSE as recently as 2026-09-29, while this repo's own log asserted they were +true. They are true now because each was measured, and the push rejection was provoked rather +than assumed. **Re-measure before relying on any of them** — see §2b. + +Still not covered: `vars.CLUSTER_AVAILABLE` is unset, so **L2/L3 conformance does not run in +CI**. A green PR therefore proves the L0/L1 offline subset, not runtime behaviour. **Consequence for you:** CI now runs, but it does not yet *block* anything. Always run the toolbox gate yourself and quote its numbers. Never describe a PR as "green" on the diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 4187c0a..deb0c8e 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -4870,3 +4870,57 @@ That theory was mine and it was wrong: the pipeline was never too slow, it simpl FOLLOW-UP note in `apps/wave-1/gitea-actions.yaml`. It deletes both hand-rolled wait loops and the envsubst hazard. Land it ALONE — it is the merge gate for the whole repo. 6. Create the Plex libraries over `/media/movies`, `/media/tv`, `/media/photos`. + +## 2026-09-29 (later) — the merge gate is now real, enforced, and proven. **PM HANDOFF.** + +Closes the gap opened by the earlier entry today. Eight PRs, all gated; every claim below was +measured against the live API or cluster, not inferred from this repository. + +### Final state + +| | Verified | +|---|---| +| `main` protected | `protected:true`, `user_can_push:false` — and a direct push as site-admin `gitea-admin` was **provoked and rejected** (`pre-receive hook declined`) | +| Gate required | `enable_status_check:true`, contexts `["gate / gate (pull_request)"]` | +| Approvals | `required_approvals:1`; `jakman85` holds `owner` | +| Runner | `incluster-dind` online, now a **native sidecar** (`initContainers`, `restartPolicy: Always`, TLS-verified `docker info` startupProbe) | +| Gate time | ~1m35s | +| Plex | claimed 2026-09-29; `PlexOnlineToken` present on the `plex-config` PVC | + +### What was corrected in the documentation + +`CLAUDE.md` §2b — *never infer platform state from this repository* — is the durable outcome. +The 2026-08-31 entry asserting an enforced gate is now annotated **in place**. The "Phase 0 / +no cluster exists" framing across `ci/`, `clusters/` and `conformance/` READMEs is banner-marked +as describing a live gap rather than a stage. All 37 runbooks now describe the PR flow. Two +`.sops.yaml` headers and `catalog/plex.yaml` were corrected (comment edits verified not to break +the SOPS MAC, and not to move the rendered golden). + +### A claim of mine that shipped wrong, and is now corrected in place + +PR #7 stated the sidecar conversion would remove the one-per-start dockerd line +`http: TLS handshake error from 127.0.0.1: EOF`. **It does not.** Exactly one still appears, +immediately after `API listen on [::]:2376` — it is now the startupProbe's first attempt, +landing microseconds after dockerd begins listening and cut off by `timeoutSeconds: 1`; the +next poll succeeds. Cosmetic, and deliberately not chased: raising the timeout to hide one +benign startup line would slow every subsequent poll. The manifest comment now says so. + +### The root cause nobody has fixed yet + +Gitea's repo settings live in its own database — invisible to Flux and unreadable by the +conformance gate. **No check exists that could have caught "protection off, runner gone."** +`infra/gitea/` declares and can re-assert the desired state, but it is not a detector. A +conformance check that reads the live Gitea API and fails on divergence is the highest-value +remaining work in this repo, and it is the difference between this having been fixed and this +being unable to recur. + +### Next steps + +1. **Write that Gitea-settings conformance check.** Everything else here is a symptom. +2. Wire `vars.CLUSTER_AVAILABLE` + `secrets.CONFORMANCE_TRIGGER_KUBECONFIG` so **L2/L3 runs in + CI**. A green PR currently proves the L0/L1 offline subset only — runtime behaviour is + unverified on every merge. +3. Re-run `infra/gitea/verify-then-enforce.sh` after any future act-runner change. +4. `required_approvals: 1` means a PR you open as `jakman85` needs `gitea-admin` to approve. + If that friction bites, either open agent work as `gitea-admin` and review as `jakman85`, or + drop back to relying on the status check alone. diff --git a/apps/wave-1/gitea-actions.yaml b/apps/wave-1/gitea-actions.yaml index 4da8b03..097bd65 100644 --- a/apps/wave-1/gitea-actions.yaml +++ b/apps/wave-1/gitea-actions.yaml @@ -118,8 +118,16 @@ spec: # and only THEN begins listening, so the port can accept a connection while the certs # are absent or incomplete -- and act_runner hard-exits on a missing # /certs/client/ca.pem. A full TLS-verified `docker info` proves BOTH that the certs - # are usable and that the daemon answers. It also removes the one-per-start - # "TLS handshake error ... EOF" that the old `nc -z` probe logged in dockerd. + # are usable and that the daemon answers. + # + # CORRECTION (measured 2026-09-29, after this conversion shipped): the commit that + # introduced this claimed the conversion would REMOVE the one-per-start dockerd line + # `http: TLS handshake error from 127.0.0.1: EOF`. IT DOES NOT. Exactly one still + # appears, immediately after "API listen on [::]:2376" -- it is now this probe's FIRST + # attempt, landing microseconds after dockerd begins listening and cut off by + # timeoutSeconds: 1. The next poll succeeds. It is cosmetic, exactly as cosmetic as it + # was before, and deliberately NOT chased: raising timeoutSeconds to hide one benign + # startup line would slow every subsequent poll. Left documented rather than silenced. # # periodSeconds 1 x failureThreshold 60 preserves the previous 60s cap. initContainers: diff --git a/infra/gitea/README.md b/infra/gitea/README.md index 4b9dbf7..345c741 100644 --- a/infra/gitea/README.md +++ b/infra/gitea/README.md @@ -43,7 +43,46 @@ list. That is precisely the observed symptom: zero runners, pod running, no erro looks at. Clearing the PVC is what forces a real re-registration; the script prints the commands. -## Sequence — order matters +## Sequence — DONE 2026-09-29 + +All four steps are complete and each was verified, not assumed: + +| Step | State | +|---|---| +| 1. Runner registered | ✅ `incluster-dind` online. Root cause of its absence was a stale `/data/.runner` identity; clearing the PVC forced re-registration | +| 2. Gate observed green on a real PR | ✅ `gate / gate (pull_request)` ~1m35s | +| 3. `enable_status_check: true` | ✅ context `gate / gate (pull_request)` — that one ONLY | +| 4. Collaborator + approvals | ✅ `jakman85` is owner; `required_approvals: 1` | + +Direct push to `main` was **provoked and rejected** (`pre-receive hook declined`) as +`gitea-admin`, a site admin — so protection is not merely declared, it holds for the account +agents authenticate as. + +### `require-gate.sh` / `verify-then-enforce.sh` + +`verify-then-enforce.sh` is the re-runnable form of the whole safe order. It merges nothing — +merging is a human's reviewed act — but it forces the Flux reconcile and then **aborts** unless +the runner pod is healthy, a runner is online, and a real workflow goes green on it, before +enabling enforcement and probing the push. `require-gate.sh` is the enforcement step alone. + +Re-run `verify-then-enforce.sh` after any future change to the act-runner Deployment. + +### Why these scripts exist at all, and what is still missing + +Gitea's repo settings live in **its own database**, not in anything Flux reconciles or the +conformance gate can read. Nothing detects drift from what this directory declares. That is +the root cause of the entire 2026-09-29 session: branch protection was off and the runner was +gone for a month while the log recorded both as working, and **no check existed that could +have caught it.** + +These scripts encode the desired state and can re-assert it, but they are not a detector. The +durable fix is a conformance check that reads the live Gitea API and fails when it diverges +from what is declared here — turning a class of silent, invisible failure into a loud one. +**That check does not exist yet and is the highest-value remaining work in this directory.** + +## Historical sequence (kept for context) + +### Original ordering notes 1. **Get an Actions runner registered** — `runner-token-rotate.sh`, then clear the PVC so the runner actually re-registers, then confirm it shows `online` in the API. Do not diff --git a/infra/gitea/require-gate.sh b/infra/gitea/require-gate.sh new file mode 100755 index 0000000..7490e15 --- /dev/null +++ b/infra/gitea/require-gate.sh @@ -0,0 +1,50 @@ +#!/usr/bin/env bash +# Items 1 + 2: make the gate a REQUIRED status check and require one approval. +# +# RUN THIS LAST. Prerequisites, in order: +# a) PR #7 (dind native sidecar) is MERGED, and +# b) you have SEEN a real workflow run go green on the converted pod. +# The runner Deployment is the merge gate for the whole repo. Turning on a required check +# while the runner is broken means no PR can merge -- including the revert. (Recoverable: +# an owner can set enable_status_check back to false. But do not rely on that.) +# +# Requires exactly ONE context: `gate / gate (pull_request)`. +# * NOT `gate / conformance (*)` -- that job is `if: false`, and a skipped Gitea job reports +# SUCCESS ("Has been skipped"). Requiring it would pass VACUOUSLY forever while nothing +# ran, which under HARD RULE 4 is worse than a deadlock: a deadlock is loud. +# * NOT the `(push)` variant -- after PR #6 there are no per-branch push runs at all, so a +# PR would never produce that context. +set -euo pipefail +HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl +creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)" +U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")" +api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; } + +echo "==> updating branch protection on main" +jq -n '{ + enable_push: false, + enable_status_check: true, + status_check_contexts: ["gate / gate (pull_request)"], + required_approvals: 1, + block_on_rejected_reviews: true, + dismiss_stale_approvals: true, + block_on_outdated_branch: false +}' | api -X PATCH -H 'Content-Type: application/json' --data @- \ + "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main" >/dev/null + +echo "==> verifying" +api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \ + | jq -r '{protected,user_can_push,enable_status_check,status_check_contexts,required_approvals}' + +cat <<'NOTE' + +NOTE ON required_approvals: 1 -- Gitea will not let an author approve their own PR. + * Agent-opened PRs (author gitea-admin) are approved by jakman85. Fine. + * PRs YOU open as jakman85 need gitea-admin to approve. If that is awkward, either open + agent work as gitea-admin and review as jakman85, or set required_approvals back to 0 + and rely on the status check alone. + +Then prove it, rather than trusting the API: + git commit --allow-empty -m probe && git push origin main # EXPECT: rejected + git reset --hard HEAD~1 +NOTE diff --git a/infra/gitea/verify-then-enforce.sh b/infra/gitea/verify-then-enforce.sh new file mode 100755 index 0000000..65b02ed --- /dev/null +++ b/infra/gitea/verify-then-enforce.sh @@ -0,0 +1,83 @@ +#!/usr/bin/env bash +# verify-then-enforce.sh — run this AFTER you have merged PRs 8, 5, 6, 7 in that order. +# +# It does NOT merge anything. Merging is your reviewed act (CLAUDE.md: a human approves and +# merges). This script does the mechanical half: prove the converted runner actually works, +# and only then turn on enforcement. +# +# WHY THIS ORDER. The act-runner Deployment IS the merge gate for this repository. Enabling a +# required status check while the runner is broken means NO PR can merge -- including the +# revert. So PR #7 lands while the gate is still advisory, is PROVEN green on a real run, and +# only then does enforcement go on. +# +# Aborts at the first checkpoint that does not pass. Safe to re-run. +set -euo pipefail +HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl +REPO_ROOT=/var/home/keith/source/home-network-k3s-impl +D="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +cd "$REPO_ROOT" +creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)" +U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")" +api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; } +A="https://${HOST}/api/v1/repos/${OWNER}/${REPO}" +say() { printf '\n\033[1m==> %s\033[0m\n' "$1"; } +die() { printf '\n\033[31mABORT: %s\033[0m\n' "$1" >&2; exit 1; } + +say "PRECHECK — all four PRs must be merged first" +for n in 8 5 6 7; do + m=$(api "$A/pulls/$n" | jq -r .merged) + printf ' PR #%s merged=%s\n' "$n" "$m" + [ "$m" = true ] || die "PR #$n is not merged yet. Merge 8, 5, 6, 7 in that order, then re-run." +done + +say "STEP 1 — force Flux to apply the sidecar now, not on the 10m interval" +./infra/devcontainer/dev.sh run bash -c \ + 'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-1 --with-source' + +say "STEP 2 — CHECKPOINT: the converted pod must come up healthy" +./infra/devcontainer/dev.sh run bash -c ' + kubectl --context k3s-prod -n gitea-actions rollout status deploy/act-runner --timeout=180s + echo "--- sidecar (initContainers) ---" + kubectl --context k3s-prod -n gitea-actions get deploy act-runner \ + -o jsonpath="{range .spec.template.spec.initContainers[*]}{.name} restartPolicy={.restartPolicy}{\"\n\"}{end}" + kubectl --context k3s-prod -n gitea-actions get pod -l app.kubernetes.io/name=act-runner' \ + || die "converted act-runner pod is not healthy -- REVERT PR #7 before anything else" + +say "STEP 3 — CHECKPOINT: a runner must be ONLINE again" +n=0 +for i in $(seq 1 40); do + n=$(api "https://${HOST}/api/v1/admin/actions/runners" | jq -r '[(.runners//[])[]|select(.status=="online")]|length') + printf '\r online runners: %s (%ss) ' "$n" "$((i*5))" + [ "${n:-0}" -ge 1 ] && { echo; break; } + sleep 5 +done +[ "${n:-0}" -ge 1 ] || die "no runner online after the conversion -- REVERT PR #7" + +say "STEP 4 — CHECKPOINT: a REAL workflow must go green ON THE CONVERTED POD" +MAIN_SHA=$(api "$A/branches/main" | jq -r .commit.id) +echo " watching the post-merge push run on main (${MAIN_SHA:0:8})..." +st="" +for i in $(seq 1 60); do + st=$(api "$A/commits/$MAIN_SHA/status" | jq -r .state) + printf '\r main gate: %-8s (%ss) ' "$st" "$((i*10))" + case "$st" in success) echo; break;; failure|error) die "gate FAILED on the converted pod -- REVERT PR #7";; esac + sleep 10 +done +[ "$st" = success ] || die "gate never completed on the converted pod -- REVERT PR #7; do NOT enable enforcement" +echo " the converted sidecar ran a real job green. Enforcement is now safe." + +say "STEP 5 — require the gate + one approval" +"$D/require-gate.sh" + +say "STEP 6 — prove a direct push is rejected (do not trust the API's word)" +git fetch -q origin && git checkout -q main && git reset -q --hard origin/main +git commit -q --allow-empty -m 'protection probe' +if git push origin main 2>&1 | grep -q 'protected branch'; then + echo " CONFIRMED: push to main rejected by the pre-receive hook." +else + echo " WARNING: the push was NOT rejected -- protection is not doing what it claims." +fi +git reset -q --hard origin/main + +say "FINAL STATE" +api "$A/branches/main" | jq -r '{protected,user_can_push,enable_status_check,status_check_contexts,required_approvals}' diff --git a/infra/plex/claim-ship.sh b/infra/plex/claim-ship.sh index 4dce82a..a3f035e 100755 --- a/infra/plex/claim-ship.sh +++ b/infra/plex/claim-ship.sh @@ -26,9 +26,10 @@ creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)" U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")" api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; } -# feat/ prefix ON PURPOSE. act_runner capacity is 1, and gate.yml's PUSH trigger covers -# feature/** fix/** docs/** chore/** but NOT feat/** — so a chore/ branch would fire a -# second, redundant run that SERIALISES ahead of the PR run and doubles the wall clock. +# Branch prefix is now cosmetic. It used to matter: gate.yml's push trigger covered +# feature/** fix/** docs/** chore/** but not feat/**, and with act_runner at capacity 1 a +# matching prefix fired a redundant run that serialised ahead of the PR run. Those +# per-branch push triggers were deleted, so no prefix produces a duplicate run any more. BRANCH="feat/plex-claim-token-$(date +%H%M%S)" el "committing on $BRANCH" -- 2.54.0