diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..c59709d --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,115 @@ +# CLAUDE.md — how work lands in this repo + +Read this before doing anything durable. It is the delivery contract for AI coding +sessions and agents. `DECISIONS-LOG.md`'s last entry is the current PM handoff and +takes precedence on *what* to work on; this file governs *how* it lands. + +## 1. Never push to `main`. Ever. + +Every durable change reaches `main` exactly one way: + + branch -> push the branch -> open a PR -> CI gate green -> human approval -> merge + +There is no other path. Specifically: + +- **Do not `git push origin main`.** +- **Do not `git push origin HEAD:main`.** Same thing wearing a hat. +- **Do not merge your own PR.** A human approves and clicks merge. +- **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`; + anything applied by hand is drift that the next reconcile reverts or, worse, adopts. + +> **Many runbooks under `runbooks/` still say `git push origin main` IS the deploy.** +> That wording is **stale** — it predates the PR flow. It describes what Flux does +> (main is what the cluster follows), not what you are permitted to do. Honour this +> file, not that line. Fixing those runbooks is tracked work. + +## 2. Enforcement is currently WEAKER than this rule. Follow the rule anyway. + +Verified against the live Gitea on 2026-09-29: + +| Expected | Actual | +|---|---| +| `main` protected, direct push rejected | **No branch protection.** `protected: false`, `user_can_push: true` | +| PR blocked until the gate passes | **`enable_status_check: false`**, `status_check_contexts: []` | +| Gate runs on every PR | **No Actions runner is registered** (repo, org and site level all report zero) | +| Approval required before merge | **`required_approvals: 0`** | + +So nothing mechanically stops you from pushing to `main` right now. **That is a gap +being closed, not permission.** `DECISIONS-LOG.md` (2026-08-31) claims the gate is a +"required status check" with a runner online; that claim is not true of the live +system, which is how this went unnoticed. + +**Consequence for you:** because CI cannot run, the **local toolbox gate is the only +real gate**. Run it and quote its numbers in your report. Never describe a PR as +"green" on the strength of CI that did not execute. + +## 3. Gate before you push + + ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh' + +Exit codes are a contract — do not flatten them: + +| rc | Meaning | +|---|---| +| 0 | PASS | +| 1 | BLOCKED — something failed | +| 3 | UNDETERMINED — a stage did not run. **Never report this as green.** | + +`spec-immutability` reaches rc 3 whenever the sibling spec repo is not checked out +beside this one; that single skip is expected and tolerated. Any *other* skip is a +real gap in your run. + +**Never gate on the host.** `promtool` and `kustomize` are not on the host `$PATH`, +so checks SKIP and you get a false rc 3. Only the pinned toolbox result counts. + +### Running the gate from a git worktree + +Two unrelated failures, both with known workarounds — put both in any agent brief: + +1. **`dev.sh` refuses to start:** `infra/ansible/.cache/prod.kubeconfig` is untracked + and exists only in the main checkout. Do **not** copy it (it is a credential). + Use `DEV_KUBECONFIG=/dev/null` — the gate contacts no cluster, so L2/L3 scaffold-skip + normally. +2. **Every `git` call inside the container fails:** a worktree's `.git` is a pointer to a + gitdir outside the mounted root. `dev.sh` takes no extra-mount flag, so inject one via + the `ENGINE` hook — a shim that intercepts `run` and re-execs + `podman run -v /path/to/main/.git:/path/to/main/.git:ro "$@"`. A traceback in + `tracked_files()` is this bug, not a broken tree. + +## 4. Branch naming + +Use `feat/`, `fix/`, `docs/`, `chore/`. Note that `.gitea/workflows/gate.yml`'s **push** +trigger lists `feature/**` — not `feat/**` — so branch pushes get no fast-feedback run. +The `pull_request` trigger fires regardless of branch name, so the PR gate is unaffected. + +## 5. Cluster access + +`kubectl` does not work on the operator workstation; the toolchain lives only in the +pinned container. Always name the context explicitly: + + ./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +`dev.sh` merges both clusters and picks a current-context **per invocation**; that +default has already changed once. Never rely on it. Do not run two `dev.sh` +invocations concurrently. + +**Mutating `kubectl` and `flux reconcile` are operator actions, not agent actions.** + +## 6. Secrets + +- **Never `sops -d` a file into a terminal, a log, or a transcript.** +- **Never print a Secret's `.data`/`.stringData`** when checking whether it exists — + list key names or object names only. A password has been leaked into a transcript + this way before. +- Encrypted files are `*.sops.yaml` or live under `secrets/`. The root `.sops.yaml` is + the single creation-rules file; do not add a second one in a subtree. +- A short-lived credential (e.g. a plex.tv claim token, ~minutes) cannot survive the PR + flow. Do not design around committing one — read the expiry from the issuing page and + treat a burnt token as free to retry. + +## 7. Before you finish + +Append a handoff entry to `DECISIONS-LOG.md` describing what you changed, what you +verified, and what you did **not** verify. State gaps plainly — an honest +"UNVERIFIED" is worth more than an optimistic "green", and this repo has already been +burned by the difference. diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index d1ce8c9..bee4174 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -4716,3 +4716,68 @@ asleep. `git worktree remove` preserves branches and their commits, so only that 6. **Five `public` services have no working public path** — waiting on WAN forwarding, deliberately. 7. Backlog cherry-picks: `feat/egress-gateway-staleness-alert`, two `rescued/*` fixes, two rendered manifests nothing applies (the keycloak one would stand up a conflicting instance). + +## 2026-09-29 — The CI/CD merge gate does not exist: no runner, no branch protection. **PM HANDOFF.** + +Went to land the plex claim wiring through the documented PR flow and discovered the flow has +never run. **PR #1 is the first pull request this repository has ever had.** It is still queued. + +### What is actually true of the live Gitea (verified via API, 2026-09-29) + +| Claimed (DECISIONS-LOG 2026-08-31) | Measured | +|---|---| +| "required status check" on `main` | `branch_protections` returns `[]`. `protected: false`, `enable_status_check: false`, `status_check_contexts: []`, `required_approvals: 0`, `user_can_push: true` | +| "runner `incluster-dind` online (ns gitea-actions)" | **Zero runners registered** at repo, org and site level | + +Both halves of that entry are false. Nothing enforces the gate, and the gate cannot run. +PR #1's two contexts (`gate / gate`, `gate / conformance`) sat `pending` for 14+ minutes with +nothing to execute them, and will sit there indefinitely. + +This is a **fail-open governance hole**, the same class as the two found on 2026-09-18 — and it +was invisible precisely because the log asserted the opposite. Every "the gate is enforcing" +statement since 2026-08-31 rested on an unverified claim. + +### Ordering trap — read before fixing + +**Do not enable required status checks until a runner is live.** Requiring +`gate / gate (pull_request)` with no runner deadlocks every PR permanently, including the PR +that would fix it. Sequence: runner first, verify the gate actually runs and passes on a real +PR, *then* turn on enforcement with the context name CI actually reported. + +Related: `gate.yml`'s `conformance` job is `if: false` by design (honest skip, never fake green), +but it still publishes a `gate / conformance (pull_request)` context. If that context is ever +listed as required, it deadlocks every PR for the same reason. Wire it or exclude it. + +### Delivery contract written down + +Added `CLAUDE.md` at the repo root — the first agent-facing delivery doc here. It states the +branch -> PR -> gate -> approval -> merge rule, records that enforcement is currently weaker than +the rule (so agents follow the rule rather than what they can get away with), and carries the +toolbox-gate invocation, the two git-worktree workarounds, the rc 0/1/3 contract, and the secret +handling rules. + +It also flags that **37 of 39 runbooks still instruct `git push origin main`**. Those were +accurate before the PR flow and are stale now. Correcting them is tracked, not done. + +### Also found, not fixed + +- `gate.yml`'s push trigger lists `feature/**`; this repo's branches use `feat/**`, so branch + pushes get no fast-feedback run. PR trigger is unaffected. +- The gitea-actions runner is GitOps-managed (`apps/wave-1/gitea-actions.yaml` + + `gitea-actions-runner-secret.sops.yaml`), so it is deployed but not registering — most likely a + stale registration token in that SOPS secret. Needs in-cluster diagnosis (operator). +- **Plex is still unclaimed and it was never checked** whether it is already claimed. Do that + before spending a token — `runbooks/plex-claim.md` §First commands. +- `runbooks/plex-claim.md` asserts a "~4-minute TTL" for a plex.tv claim token in four places, + **uncited**, and the whole stopwatch procedure is built on it. Upstream `pms-docker`'s README + says nothing about expiry. The number is plausible community folklore, not a measured fact. + The authoritative reading is the countdown rendered on plex.tv/claim at issue time. + +### Next steps + +1. **Operator:** diagnose the gitea-actions runner; get one registered. +2. **Operator:** enable branch protection on `main` — push blocked, approvals required — and only + then add the gate as a required status check. +3. Land `feat/plex-claim-sops-secret` (PR #1) once the gate can actually run. +4. Claim plex, after checking it is not already claimed. +5. Correct the 37 runbooks' `git push origin main` wording. diff --git a/apps/wave-1/gitea-actions-runner-secret.sops.yaml b/apps/wave-1/gitea-actions-runner-secret.sops.yaml index 48e0e83..575e4a4 100644 --- a/apps/wave-1/gitea-actions-runner-secret.sops.yaml +++ b/apps/wave-1/gitea-actions-runner-secret.sops.yaml @@ -14,7 +14,7 @@ metadata: app.kubernetes.io/name: act-runner type: Opaque stringData: - token: ENC[AES256_GCM,data:W48mEtBqGeukX13lMhrhrer+zB21cLVp0acy3m1Xy/oo89DqO8FX/Q==,iv:wfTUU7RiFLdEc1fmEFS9DfE2SmGCFLPK17m0IKspft0=,tag:UhoXA47p7386KHcYNbPzCg==,type:str] + token: ENC[AES256_GCM,data:f574mAUkLvMGq0TYycwCd4tduMuBgQLiTg/140kuRDgFUYYm1ZoSjQ==,iv:Tn7gFWBLm2hHk/BbfVX7qIZegwZp9ah1cMou5i2O91E=,tag:TkdSiMI4sdCNn2JuN++Pig==,type:str] sops: kms: [] gcp_kms: [] @@ -30,8 +30,8 @@ sops: MDZLUE52QzJXbFlrNWxOcVhJbDNnY00KxCpFkbhhCuk4Fa8CC3C4+R9AcsUnaXQR UjAfwKFoXAneXUih7TTXMbR/6+VgQBww5Bmwpn7jVLnzP0N0pIZVTA== -----END AGE ENCRYPTED FILE----- - lastmodified: "2026-08-26T04:48:14Z" - mac: ENC[AES256_GCM,data:b99/zkL332RIT3FYBa1sR3o9+pwlmrUgKvVHawbjWYSJZ7ERUfO+LXuhKJ0HKrp6i1JfkurZlRO27iBcfrBisAXS9KLaqmsCJf4kKfFc4skANrmKCzprrXUmZexr1uiE8eDUzjNCLNxSCZfnFR6d/AkMXJqi6Hsj9/O+wH2lWsc=,iv:3DIPszSdxK81ERO3o/nTs33qQYHaAKLNcGP4qlZViFc=,tag:/B6VKkzMUwh0byNUicS5wQ==,type:str] + lastmodified: "2026-09-29T05:48:35Z" + mac: ENC[AES256_GCM,data:nobqr794nrdRYFrd8Ct8bw9M8NtftJXawRGk9zrL+pTWahwlWrwXVtOAxMGEPVojz3WemsxTJESzmzy0gcT1QcrjpYTsF2MVHcr6+umk8q9gAh2enyxy/0QkYiSt6XjvRVqfqleLanEkeIq1FPf2zhSLSJaiw4/oZTfkCHD1/Zg=,iv:JxaP0htmGcvE4ZU1YAmGF4A6JkejV29NNygyMl1OKF8=,tag:G70NvcPyVuagXhugWoUfTw==,type:str] pgp: [] encrypted_regex: ^(data|stringData)$ version: 3.9.4 diff --git a/infra/gitea/README.md b/infra/gitea/README.md new file mode 100644 index 0000000..d8655e0 --- /dev/null +++ b/infra/gitea/README.md @@ -0,0 +1,59 @@ +# infra/gitea — platform settings that are NOT GitOps-managed + +Gitea's own repo settings (branch protection, required status checks, approvals) live +in Gitea's database, not in a manifest Flux reconciles. They are therefore **drift-prone +and invisible to the gate** — exactly how the state below went unnoticed. + +## The gap this closes + +Verified against the live API on 2026-09-29, against a `DECISIONS-LOG.md` entry +(2026-08-31) asserting the opposite: + +| Claimed | Measured | +|---|---| +| `main` is a protected branch, gate is a required status check | `branch_protections` → `[]`; `protected:false`, `enable_status_check:false`, `required_approvals:0`, `user_can_push:true` | +| Runner `incluster-dind` online | **zero** runners registered at repo, org and site level | + +So `main` accepted direct pushes, and the gate could not run at all. `CLAUDE.md` is the +rule; this directory is how the rule gets enforced by the machine. + +## `branch-protection.sh` + +Blocks direct pushes to `main`. Run it, then run the push probe it prints — **verify, +do not trust the API response**, since Gitea may let site admins bypass protection. + + ./infra/gitea/branch-protection.sh + +It authenticates with the credential `git` already has stored for this host, and never +prints the secret. + +## `runner-token-rotate.sh` + +Mints a fresh instance-level registration token from the Gitea API and writes it into +`apps/wave-1/gitea-actions-runner-secret.sops.yaml` via `sops set`. **The token value is +never printed.** + + ./infra/gitea/runner-token-rotate.sh + +A fresh token is usually **not sufficient on its own**. `act_runner` registers exactly once +and persists its identity in `/data/.runner` on the `act-runner-data` PVC. If Gitea no +longer holds a matching runner row, the pod starts, sees `.runner`, **skips registration**, +and polls with an identity Gitea does not recognise — so it never appears in the runner +list. That is precisely the observed symptom: zero runners, pod running, no error anyone +looks at. Clearing the PVC is what forces a real re-registration; the script prints the +commands. + +## Sequence — order matters + +1. **Get an Actions runner registered** — `runner-token-rotate.sh`, then clear the PVC so + the runner actually re-registers, then confirm it shows `online` in the API. Do not + skip the confirmation: the whole reason this went unnoticed is that a runner was + *recorded* as online without anyone checking. +2. **Watch the gate actually report on a real PR**, and note the context name it produced. +3. **Only then set `enable_status_check: true`** with that exact context. +4. Add the human reviewer as a repo collaborator, then raise `required_approvals` to 1. + +> **Do not do step 3 before step 1.** A required status check with no runner leaves every +> PR permanently pending — including the PR that would fix it. The same applies to +> `gate.yml`'s `conformance` job: it is `if: false` and never runs, yet still publishes a +> `gate / conformance (pull_request)` context. Never list that one as required. diff --git a/infra/gitea/branch-protection.sh b/infra/gitea/branch-protection.sh new file mode 100755 index 0000000..fe0d739 --- /dev/null +++ b/infra/gitea/branch-protection.sh @@ -0,0 +1,57 @@ +#!/usr/bin/env bash +# Enable branch protection on main for home-network-k3s-impl. +# +# Sets enable_push:false -> no direct pushes to main; changes land only by merging a PR. +# +# enable_status_check is deliberately FALSE. There is currently NO Gitea Actions runner +# registered (verified 2026-09-29: zero at repo, org and site level), so requiring the +# gate context would deadlock EVERY PR permanently -- including the PR that fixes it. +# Turn it on only after a runner is live and you have seen the gate actually report on +# a real PR, using the context name CI itself produced. +# +# required_approvals is 0 for the same class of reason: jakman85 is a site admin but NOT +# a repo collaborator, so requiring an approval now could leave nobody able to approve. +# Add the collaborator first, then raise this to 1. +set -euo pipefail +HOST=gitea.home.keithjackson.dev +OWNER=Home-Network +REPO=home-network-k3s-impl + +creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)" +U="$(sed -n 's/^username=//p' <<<"$creds")" +P="$(sed -n 's/^password=//p' <<<"$creds")" +[ -n "$U" ] && [ -n "$P" ] || { echo "no stored credential for $HOST" >&2; exit 1; } +echo "authenticating as '${U}' (secret not shown)" + +echo "==> creating branch protection on 'main'" +jq -n '{ + branch_name: "main", + enable_push: false, + enable_status_check: false, + status_check_contexts: [], + required_approvals: 0, + block_on_rejected_reviews: true, + block_on_outdated_branch: false, + dismiss_stale_approvals: true +}' | curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \ + -X POST -H 'Content-Type: application/json' --data @- \ + "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections" \ +| jq -r 'if .branch_name then "created" else "API said: \(.message // tostring)" end' + +echo "==> verifying (this is the part that matters)" +curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \ + "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \ +| jq -r '{protected, user_can_push, required_approvals, enable_status_check, status_check_contexts}' + +cat <<'NOTE' + +Now prove it actually blocks you -- do not take the API's word for it. From a clean +checkout on main, with nothing to lose: + + git commit --allow-empty -m 'protection probe' + git push origin main # EXPECT: rejected by the remote + git reset --hard HEAD~1 # tidy up either way + +If that push SUCCEEDS, Gitea is letting site admins bypass protection and this is not +yet doing what you want. +NOTE diff --git a/infra/gitea/runner-token-rotate.sh b/infra/gitea/runner-token-rotate.sh new file mode 100755 index 0000000..b1d6dfd --- /dev/null +++ b/infra/gitea/runner-token-rotate.sh @@ -0,0 +1,61 @@ +#!/usr/bin/env bash +# Mint a fresh Gitea Actions registration token and write it into the SOPS secret. +# +# The token value is NEVER printed. It goes from the Gitea API into `sops set` and +# nowhere else. (It is briefly visible in this host's process list as a sops argument; +# acceptable on a single-operator workstation, and far better than echoing it.) +# +# This ALONE may not bring the runner back -- see "IMPORTANT" at the end of the output. +set -euo pipefail +HOST=gitea.home.keithjackson.dev +SECRET=apps/wave-1/gitea-actions-runner-secret.sops.yaml +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +cd "$REPO_ROOT" +[ -f "$SECRET" ] || { echo "not in the repo root, or $SECRET is missing" >&2; exit 1; } + +creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)" +U="$(sed -n 's/^username=//p' <<<"$creds")" +P="$(sed -n 's/^password=//p' <<<"$creds")" +[ -n "$U" ] && [ -n "$P" ] || { echo "no stored git credential for $HOST" >&2; exit 1; } +echo "==> minting an instance-level registration token as '${U}'" + +TOKEN="$(curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \ + "https://${HOST}/api/v1/admin/runners/registration-token" | jq -r '.token // empty')" +[ -n "$TOKEN" ] || { echo "Gitea did not return a token" >&2; exit 1; } +echo "==> got a ${#TOKEN}-character token (value not shown)" + +echo "==> writing it into ${SECRET} via sops" +DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run \ + sops set "$SECRET" '["stringData"]["token"]' "\"${TOKEN}\"" +unset TOKEN + +echo "==> done. Verify the file is still encrypted (no plaintext token):" +grep -q 'ENC\[' "$SECRET" && echo " OK - stringData is still SOPS-encrypted" \ + || { echo " REFUSING TO CONTINUE: $SECRET is not encrypted" >&2; exit 1; } + +cat <<'NOTE' + +Next: commit this on a branch, open a PR, and merge it so Flux applies the new Secret. + +IMPORTANT -- a new token is usually NOT sufficient on its own. +act_runner registers exactly ONCE and then persists its identity in /data/.runner on the +act-runner-data PVC. If Gitea no longer has a matching runner row (DB restore, the row was +deleted, the instance was rebuilt), the pod starts, sees .runner, SKIPS registration, and +polls with an identity Gitea does not recognise -- so it never appears in the runner list, +which is exactly the observed symptom (zero runners, pod running). + +So after the Secret lands, force a genuine re-registration by clearing that state: + + kubectl --context k3s-prod -n gitea-actions scale deploy/act-runner --replicas=0 + kubectl --context k3s-prod -n gitea-actions delete pvc act-runner-data + kubectl --context k3s-prod -n gitea-actions scale deploy/act-runner --replicas=1 + +(The PVC holds only runner registration state -- no build artifacts, nothing durable. +Deleting it is the supported way to re-register. Flux recreates it from the manifest.) + +Then confirm it actually registered -- do not assume: + + curl -s -u '' https://gitea.home.keithjackson.dev/api/v1/admin/actions/runners | jq + +A runner must appear with status "online" and the labels ubuntu-latest / homelab. +NOTE