Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
0c6949ed98 | ||
|
|
20ea70d221 | ||
|
|
8c7a770de2 | ||
|
|
1c5974013e | ||
|
|
4791d4a43a | ||
|
|
dfd608ac32 | ||
|
|
1d2d9d6549 | ||
|
|
4b31f97486 | ||
|
|
50a48e6f07 | ||
|
|
911ead5fbd | ||
|
|
4634b44cc7 | ||
|
|
6586919132 | ||
|
|
e6f6e4febb |
No files matched your search
@@ -110,9 +110,31 @@ jobs:
|
||||
# UNVERIFIED and NAMED in the PASS headline, never faked green. GATE_STRICT_SKIPS + the
|
||||
# tolerated exit 3 are explained in the file header; the tolerance covers exactly one
|
||||
# thing, the unverifiable spec repo, and nothing else can produce a 3 here.
|
||||
# GITEA_SETTINGS_TOKEN feeds ci/gitea-settings.sh, the only check that can see Gitea's
|
||||
# OWN database (branch protection + runner registration) — the blind spot that let `main`
|
||||
# sit unprotected with zero runners for a month while this repo recorded both as working.
|
||||
# Referenced BY NAME only; create it with infra/gitea/set-ci-token.sh.
|
||||
#
|
||||
# WIDENED TOLERANCE, DECLARED: `|| [ "$?" -eq 3 ]` used to cover exactly one skip
|
||||
# (spec-immutability). It now also covers gitea-settings skipping when that secret is
|
||||
# absent. That is a deliberate, temporary weakening: making an unwired credential BLOCK
|
||||
# every PR is the lock-yourself-out failure this repo keeps walking into. Once the secret
|
||||
# exists and the stage has been SEEN green, set REQUIRE_GITEA_SETTINGS=true below and the
|
||||
# tolerance narrows back to spec-immutability alone.
|
||||
- name: Merge gate — ci/gate.sh (what `make -C ci gate` runs)
|
||||
env:
|
||||
GATE_STRICT_SKIPS: "true"
|
||||
SETTINGS_API_TOKEN: ${{ secrets.SETTINGS_API_TOKEN }}
|
||||
# ENABLED 2026-09-29. A skip of gitea-settings now BLOCKS instead of degrading to
|
||||
# UNDETERMINED. Safe because THIS PR's own CI run is the proof: if the stage were
|
||||
# skipping, this line turns that into a failure and the PR goes red instead of
|
||||
# merging. Green here means the token resolved and the live Gitea matched the
|
||||
# declaration in infra/gitea/expected-branch-protection.json.
|
||||
REQUIRE_GITEA_SETTINGS: "true"
|
||||
# ci/approval-policy.sh needs to know WHICH PR it is looking at, so it can read that
|
||||
# PR's changed files and its reviews. Absent on a push event, where the stage is N/A.
|
||||
GITHUB_EVENT_NAME: ${{ github.event_name }}
|
||||
PR_NUMBER: ${{ github.event.pull_request.number }}
|
||||
run: ./ci/gate.sh || [ "$?" -eq 3 ]
|
||||
|
||||
# NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4
|
||||
|
||||
@@ -14,7 +14,28 @@ There is no other path. Specifically:
|
||||
|
||||
- **Do not `git push origin main`.**
|
||||
- **Do not `git push origin HEAD:main`.** Same thing wearing a hat.
|
||||
- **Do not merge your own PR.** A human approves and clicks merge.
|
||||
- **Do not merge your own PR.** See the approval model below — and either way, the merge is
|
||||
someone else's action, never yours.
|
||||
|
||||
### Who approves
|
||||
|
||||
**The CI/CD pipeline is the approver.** A green gate is sufficient for an ordinary change; the
|
||||
branch requires no human approval for one. That is a deliberate transfer of authority, on the
|
||||
operator's stated principle: *if the pipeline is not good enough to be trusted for this, it is not
|
||||
good enough to do its job and needs improving* — so a weak gate is a bug to fix, not a reason to
|
||||
add a human.
|
||||
|
||||
**One exception, enforced by `ci/approval-policy.sh`:** a change to the pipeline itself, the
|
||||
admission policies, the Flux wiring, the SOPS recipients, the RBAC, or this file needs an
|
||||
approving review from someone who is not its author. The reason is narrow and structural — **a
|
||||
change to the gate cannot be gated by the gate.** A PR that weakens a check would be approved by
|
||||
the weakened check. Everything else merges on green.
|
||||
|
||||
The path list is data, with a reason per entry: `infra/gitea/approval-policy.json`. It is itself a
|
||||
sensitive path, so the policy cannot be widened or narrowed without a human.
|
||||
|
||||
That stage is **fail-closed** — if it cannot evaluate, it blocks. Unlike `gitea-settings`, where a
|
||||
missing credential is a lock-out, here a skip would be a bypass.
|
||||
- **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
|
||||
anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
|
||||
|
||||
|
||||
@@ -4964,3 +4964,148 @@ account rubber-stamping the human. That is ceremony, not review. Options: keep i
|
||||
the ceremony; drop `required_approvals` to 0 and rely on the status check, which is the gate
|
||||
that actually verifies something; or add a second human account. **This is a decision for the
|
||||
operator, not a default to be chosen silently.**
|
||||
|
||||
## 2026-09-29 (end of session) — **PM HANDOFF.** Six PRs open; read this before touching anything.
|
||||
|
||||
The session that began "set a SOPS secret" ended having rebuilt the delivery pipeline. Plex is
|
||||
claimed and scanning. `main` is genuinely protected and the gate genuinely enforces. What is
|
||||
open is listed below with the reason each is not merged.
|
||||
|
||||
### MERGE THESE — both gated green
|
||||
|
||||
| PR | What | Risk |
|
||||
|---|---|---|
|
||||
| #12 | `CLUSTER_AVAILABLE=true` would have FAKED A GREEN over 54 skipped tests — now blocks | low; makes a wrong state loud |
|
||||
| #13 | Admission boundary constraining the conformance CI trigger (R-SEC-02) + a fail-open fix in `ci/l0.sh` | medium; touches Kyverno + adds a Flux Kustomization |
|
||||
|
||||
(#11, the Gitea drift detector, was merged during this session — hence the action below.)
|
||||
|
||||
Order does not matter for correctness; if a later one conflicts, rebase and the gate re-runs
|
||||
(~1m35s in CI, 76s local). **Each was gated in the pinned toolbox, not on a host.**
|
||||
|
||||
### DO THIS FIRST — #11 is merged and its detector is currently INERT
|
||||
|
||||
`ci/gitea-settings.sh` SKIPS until its token exists, and **a skipped detector detects nothing**:
|
||||
|
||||
./infra/gitea/set-ci-token.sh # mints + stores GITEA_SETTINGS_TOKEN, never prints it
|
||||
|
||||
Then open any PR and confirm the gate prints `gitea-settings: green`. ONLY THEN uncomment
|
||||
`REQUIRE_GITEA_SETTINGS: "true"` in `.gitea/workflows/gate.yml`, so a future missing token blocks
|
||||
instead of skipping. Doing that before seeing it green locks the repo.
|
||||
|
||||
Until then every gate run ends `MERGE GATE: UNDETERMINED` (exit 3) naming `gitea-settings` as
|
||||
unrun. That is BY DESIGN and `gate.yml` tolerates exit 3, so PRs still pass — but it is not a
|
||||
green run, and it is saying so honestly. The token is what turns it green.
|
||||
|
||||
### A DIFFERENT COMPUTER NEEDS THREE THINGS THAT ARE NOT IN GIT
|
||||
|
||||
This is the part that will silently waste an hour otherwise. None of these are in the repo, by
|
||||
design, and without them the toolbox and the cluster are unreachable:
|
||||
|
||||
1. **`~/.config/sops/age/keys.txt`** — the age PRIVATE key. Without it NO `*.sops.yaml` can be
|
||||
read or edited, and `sops` will fail in a way that looks like a broken file. Copy it from this
|
||||
workstation; the public recipient in `.sops.yaml` is only half the pair.
|
||||
2. **`infra/ansible/.cache/prod.kubeconfig`** — untracked, so it exists only in the main checkout
|
||||
on this machine. Without it `dev.sh` refuses to start. In a git WORKTREE it is absent even
|
||||
here: use `DEV_KUBECONFIG=/dev/null` for anything that needs no cluster (see `CLAUDE.md` §3).
|
||||
3. **A Gitea credential** for `gitea.home.keithjackson.dev` (this session used the stored
|
||||
`~/.git-credentials`). `git push` and every `infra/gitea/*.sh` script needs it.
|
||||
|
||||
### STATE, MEASURED — do not re-derive it from this log, re-measure it
|
||||
|
||||
`CLAUDE.md` §2b exists because this log was confidently wrong for a month. Verify before relying:
|
||||
|
||||
protected:true user_can_push:false enable_status_check:true
|
||||
status_check_contexts:["gate / gate (pull_request)"] required_approvals:1
|
||||
block_admin_merge_override:true runner incluster-dind: online gate ~1m35s
|
||||
|
||||
### NEXT WORK, in priority order
|
||||
|
||||
1. **Finish the L2/L3 wiring** now that the credential model is decided and #13 lands the
|
||||
boundary. Still missing (`docs/plans/l2-l3-conformance-ci-wiring.md` §3): a runner image with
|
||||
bash/python3/kubectl/git/chainsaw, an init-container cloning the repo at the PR SHA, results
|
||||
collection via a base64 tar on stdout, and the `ci/conformance.sh` rewrite to drive the Job.
|
||||
**Tighten the admission policy's image pattern to the literal digest when that image exists.**
|
||||
2. **Do NOT set `vars.CLUSTER_AVAILABLE=true`** until step 1 is done. It does not enable L2/L3; it
|
||||
only changes which way the stage fails (see #12).
|
||||
3. The `gitea-settings` detector covers branch protection + runners. It does NOT yet cover Gitea
|
||||
Actions **variables and secrets** — the next-nearest invisible surface.
|
||||
4. `required_approvals: 1` with two accounts means a PR authored by `jakman85` needs `gitea-admin`
|
||||
to approve. The operator holds ultimate authority and does not require that review; revisit if
|
||||
it becomes friction.
|
||||
|
||||
### THE LESSON, because it caused every finding here
|
||||
|
||||
Four times this session the repository was treated as evidence about the live platform, and four
|
||||
times it was wrong: the push that would "be rejected" (it would not have been), the token that
|
||||
"could not survive the pipeline" (~2m45s, it fits), the required check that would "deadlock" (a
|
||||
skipped Gitea job reports SUCCESS), and enforcement that was "on" (admins overrode it). Two
|
||||
fail-opens in the gate itself were found the same way — `kyverno test` certifying policies that do
|
||||
not load, and `CLUSTER_AVAILABLE` certifying 54 skipped tests as passed.
|
||||
|
||||
**Measure, then assert. `CLAUDE.md` §2b is the rule; this entry is the evidence for it.**
|
||||
|
||||
## 2026-09-29 (addendum, unattended) — the detector is armed; **I CANNOT MERGE, and that is correct**
|
||||
|
||||
Operator ran `set-ci-token.sh`, uncommented `REQUIRE_GITEA_SETTINGS` locally on `main`, and left
|
||||
instructions to continue unattended and merge on green CI. Two things came out of that.
|
||||
|
||||
### 1. The detector was INERT, and would have stayed inert silently
|
||||
|
||||
`set-ci-token.sh` PUT the secret as `GITEA_SETTINGS_TOKEN`. **Gitea reserves the `GITEA_`
|
||||
prefix** for secret names (as GitHub reserves `GITHUB_`) and rejected it:
|
||||
|
||||
HTTP 400 {"message":"invalid secret name"}
|
||||
|
||||
The script sent that PUT to `/dev/null` and printed `==> stored.` unconditionally; `curl -sS`
|
||||
exits 0 on an HTTP error. **A token existed; no secret did.** So `ci/gitea-settings.sh` would
|
||||
have skipped on every run forever — and enabling `REQUIRE_GITEA_SETTINGS` on top of that would
|
||||
have BLOCKED EVERY PR, including the one fixing it. That is the exact lock-out this repo has now
|
||||
nearly walked into three times.
|
||||
|
||||
Fixed in **PR #15**: the secret is `SETTINGS_API_TOKEN`, and `set-ci-token.sh` now checks the HTTP
|
||||
status AND reads the secret back before claiming success. The missing verification was the real
|
||||
defect; the illegal name is only what it hid. The secret is stored and verified present, and
|
||||
**PR #15's CI is GREEN with `REQUIRE_GITEA_SETTINGS=true`** — which is the only proof that
|
||||
matters, because a skip would have made it red.
|
||||
|
||||
The operator's local edit could not be pushed to `main` (protected). It is carried in PR #15.
|
||||
|
||||
### 2. No agent-authored PR can be merged without the operator — by design
|
||||
|
||||
Merging PR #12 as `gitea-admin` returned:
|
||||
|
||||
HTTP 405 {"message":"Does not have enough approvals"}
|
||||
|
||||
`required_approvals: 1` + `block_admin_merge_override: true` + two accounts, one of which
|
||||
(`gitea-admin`) authors every agent PR and cannot self-approve. **So "merge on green CI" is not
|
||||
something an agent can execute here.** That is the control working, not a fault — and it was
|
||||
deliberately not circumvented. Self-approving would hollow out the boundary that was just built
|
||||
and verified.
|
||||
|
||||
**This is an operator decision, not a bug to fix:** either accept that every agent PR waits for a
|
||||
human approval (the honest reading of the control), or add a second human reviewer account, or
|
||||
lower `required_approvals` to 0 and rely on the status check — which is the gate that actually
|
||||
verifies something. Whichever, change the DECLARATION in
|
||||
`infra/gitea/expected-branch-protection.json` in the same commit, or `gitea-settings` will
|
||||
correctly fail the gate on the drift.
|
||||
|
||||
### Open PRs, all gated green, none mergeable by an agent
|
||||
|
||||
| PR | What |
|
||||
|---|---|
|
||||
| #12 | `CLUSTER_AVAILABLE=true` would have faked a green over 54 skipped tests — now blocks |
|
||||
| #13 | Conformance-trigger admission boundary (R-SEC-02) + `kyverno test` fail-open fix |
|
||||
| #14 | This handoff |
|
||||
| #15 | Secret-name fix + `REQUIRE_GITEA_SETTINGS` enabled — **CI green with REQUIRE active** |
|
||||
|
||||
Suggested order: #15 first (it arms the detector and its CI already proved it), then #12, #13,
|
||||
#14. If a later one conflicts, rebase; the gate re-runs in ~1m36s.
|
||||
|
||||
### Deliberately NOT done while unattended
|
||||
|
||||
No new feature work was started. Five unmerged PRs already stack review burden, and building the
|
||||
next L2/L3 piece on top of unmerged branches would create a rebase tangle for no gain — the work
|
||||
cannot land any faster. The next piece is specified and ready to start once these are merged:
|
||||
`docs/plans/l2-l3-conformance-ci-wiring.md` §3, with §4's credential model now decided and its
|
||||
boundary landed in #13.
|
||||
@@ -32,6 +32,12 @@ render-golden: ## render==golden: canonical render reproduces committed apps/ (n
|
||||
l1: ## L1 rendering checks (no cluster; live now)
|
||||
@"$(CI_DIR)l1.sh"
|
||||
|
||||
approval-policy: ## CI is the approver; sensitive paths still need a human (needs PR_NUMBER + token)
|
||||
@./approval-policy.sh
|
||||
|
||||
gitea-settings: ## Gitea branch protection + runners match infra/gitea/expected-branch-protection.json (SKIPS w/o GITEA_SETTINGS_TOKEN)
|
||||
@./gitea-settings.sh
|
||||
|
||||
conformance: ## L2/L3 in-cluster Chainsaw Job (SKIPS w/o CLUSTER_AVAILABLE=true)
|
||||
@"$(CI_DIR)conformance.sh"
|
||||
|
||||
|
||||
Executable
+117
@@ -0,0 +1,117 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/approval-policy.sh — the CI/CD pipeline is the approver, EXCEPT for changes that alter
|
||||
# what the pipeline verifies or the security posture it rests on.
|
||||
#
|
||||
# THE POLICY, and why it is shaped this way.
|
||||
#
|
||||
# Operator intent (2026-09-29): "CI/CD pipeline is the end-all-be-all for approval. If the CI
|
||||
# pipeline is not good enough to be trusted for this, then it is not good enough to do its
|
||||
# intended job and needs to be improved. Manual approvers should only be for architecturally or
|
||||
# security relevant changes -- which is essentially the CI pipeline itself and kyverno policies."
|
||||
#
|
||||
# So `required_approvals` on the branch drops to 0 and the GATE decides. That is a real transfer
|
||||
# of authority, and it has exactly one hole: **a change to the gate cannot be gated by the gate.**
|
||||
# A PR that weakens a check would be approved by the weakened check. The same applies to the
|
||||
# admission policies, the Flux wiring that decides what reconciles, the SOPS recipients, and the
|
||||
# RBAC. Those are the "architecturally or security relevant" set, and this stage is what still
|
||||
# requires a human for them.
|
||||
#
|
||||
# The list is data, not code: infra/gitea/approval-policy.json, with a stated reason per entry so
|
||||
# it stays reviewable. Changing that list is itself a sensitive change (infra/gitea/**), so the
|
||||
# policy cannot be widened without a human.
|
||||
#
|
||||
# FAIL-CLOSED, DELIBERATELY DIFFERENT FROM gitea-settings.sh. That stage skips when its token is
|
||||
# missing, because an unwired credential blocking every PR is a lock-out. Here the reverse is
|
||||
# true: if this cannot evaluate, an unreviewed change to the gate itself could merge. A skip is a
|
||||
# BYPASS, not a gap. So: cannot determine -> BLOCK.
|
||||
#
|
||||
# EXIT CODES
|
||||
# 0 no sensitive paths touched (gate alone is sufficient), OR sensitive paths touched and a
|
||||
# non-author APPROVED review is present
|
||||
# 1 sensitive paths touched without such a review, OR the policy could not be evaluated
|
||||
set -euo pipefail
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "${CI_DIR}/lib.sh"
|
||||
|
||||
log "Stage: approval policy (CI approves; humans approve changes to CI)"
|
||||
|
||||
DECL="${REPO_ROOT}/infra/gitea/approval-policy.json"
|
||||
[ -f "$DECL" ] || die "approval policy declaration missing: ${DECL}"
|
||||
require_tool jq
|
||||
require_tool python3
|
||||
|
||||
# Not a PR -> nothing to gate. A push to main is already merged, and direct pushes are refused
|
||||
# by branch protection, so this is genuinely not-applicable rather than a skipped check.
|
||||
EVENT="${GITHUB_EVENT_NAME:-${EVENT_NAME:-}}"
|
||||
PR="${PR_NUMBER:-}"
|
||||
if [ "$EVENT" != "pull_request" ] || [ -z "$PR" ] || [ "$PR" = "null" ]; then
|
||||
ok "not a pull_request event (event='${EVENT:-none}') — approval policy N/A"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
|
||||
GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
|
||||
TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
|
||||
if [ -z "$TOKEN" ]; then
|
||||
err "no SETTINGS_API_TOKEN — cannot read this PR's changed files or its reviews."
|
||||
err " BLOCKING. A skip here would let an unreviewed change to the gate itself merge."
|
||||
summary "- approval-policy: **BLOCKED — cannot evaluate** (no token; a skip here is a bypass)"
|
||||
exit 1
|
||||
fi
|
||||
api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
|
||||
|
||||
FILES_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/files?limit=1000" || true)"
|
||||
PR_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}" || true)"
|
||||
REVIEWS_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/reviews" || true)"
|
||||
for v in FILES_JSON PR_JSON REVIEWS_JSON; do
|
||||
jq -e . >/dev/null 2>&1 <<<"${!v}" || {
|
||||
err "could not read ${v} from Gitea for PR #${PR} — BLOCKING rather than assuming."
|
||||
summary "- approval-policy: **BLOCKED — Gitea unreadable**"
|
||||
exit 1; }
|
||||
done
|
||||
|
||||
AUTHOR="$(jq -r '.user.login // empty' <<<"$PR_JSON")"
|
||||
[ -n "$AUTHOR" ] || { err "could not determine PR author — BLOCKING"; exit 1; }
|
||||
|
||||
# NOTE: the PR file list goes in via an ENV VAR, not a second stdin redirect. `python3 - <<EOF`
|
||||
# already consumes stdin for the script; adding `<<<"$json"` silently wins and the script body is
|
||||
# never read. Caught before shipping, by running it.
|
||||
SENSITIVE="$(FILES_JSON="$FILES_JSON" DECL="$DECL" python3 -c '
|
||||
import json, os, fnmatch
|
||||
decl = json.load(open(os.environ["DECL"]))
|
||||
files = [f.get("filename", "") for f in json.loads(os.environ["FILES_JSON"])]
|
||||
pats = decl["sensitive_paths"]
|
||||
hits = sorted({f for f in files for p in pats
|
||||
if fnmatch.fnmatch(f, p) or (p.endswith("/**") and f.startswith(p[:-2]))})
|
||||
print("\n".join(hits))
|
||||
')"
|
||||
|
||||
if [ -z "$SENSITIVE" ]; then
|
||||
ok "no architecturally/security-relevant paths touched — the GATE is the approver."
|
||||
summary "- approval-policy: **green** (CI approves; no sensitive paths touched)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo " sensitive paths touched:" >&2
|
||||
printf ' %s\n' $SENSITIVE >&2
|
||||
|
||||
APPROVER="$(jq -r --arg a "$AUTHOR" \
|
||||
'[.[] | select(.state == "APPROVED") | select(.user.login != $a) | .user.login] | first // empty' \
|
||||
<<<"$REVIEWS_JSON")"
|
||||
|
||||
if [ -n "$APPROVER" ]; then
|
||||
ok "approved by '${APPROVER}' (author '${AUTHOR}') — sensitive change is human-reviewed."
|
||||
summary "- approval-policy: **green** (sensitive paths approved by \`${APPROVER}\`)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
err "This PR changes the pipeline or the security posture it rests on, and has NO approving"
|
||||
err " review from anyone other than its author ('${AUTHOR}')."
|
||||
err " A change to the gate cannot be gated by the gate — that is the one thing CI cannot"
|
||||
err " honestly approve for itself. Get a human review, or move the change out of:"
|
||||
printf ' %s\n' $SENSITIVE >&2
|
||||
err " The list lives in infra/gitea/approval-policy.json, with a reason per entry."
|
||||
summary "- approval-policy: **BLOCKED — sensitive paths need a non-author approval**"
|
||||
exit 1
|
||||
@@ -61,5 +61,45 @@ if [ -d "${CONF_ART}/coverage" ]; then
|
||||
cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# DID IT ACTUALLY RUN? -- the guard without which this stage FAKES A GREEN.
|
||||
#
|
||||
# run.sh exits 0 in "scaffold mode" when it cannot reach a cluster: it emits a JUnit file
|
||||
# whose every testcase is <skipped> and returns success. That is correct FOR run.sh -- an
|
||||
# honest skip is its documented no-cluster behaviour. It is catastrophic HERE, because
|
||||
# reaching this point means the environment DECLARED CLUSTER_AVAILABLE=true, and this stage
|
||||
# would otherwise print "L2/L3 conformance passed" over 54 tests that never executed.
|
||||
#
|
||||
# MEASURED 2026-09-29, before this guard existed:
|
||||
# CLUSTER_AVAILABLE=true ./ci/conformance.sh -> rc=0
|
||||
# "[junit] wrote skipped JUnit for 54 scaffolded test(s)"
|
||||
# "[ ok ] L2/L3 conformance passed for touched subset"
|
||||
# One Gitea variable away from a fake green on an enforcing merge gate. That is exactly the
|
||||
# HARD RULE 4 violation this repo's whole gate design exists to prevent, and nothing caught
|
||||
# it because nobody had ever set the variable.
|
||||
#
|
||||
# A skip while a cluster is DECLARED is a contradiction: either the declaration is wrong or
|
||||
# the cluster is unreachable. The merge-gate decision matrix already says this case is BLOCK
|
||||
# ("conformance skipped while CLUSTER_AVAILABLE=true -> BLOCK (it should have run)"). This
|
||||
# makes the stage itself honour that instead of reporting a pass.
|
||||
JUNIT="${CONF_ART}/report.junit.xml"
|
||||
if [ -f "$JUNIT" ]; then
|
||||
total="$(grep -o '<testcase' "$JUNIT" | wc -l | tr -d ' ')"
|
||||
skipped="$(grep -o '<skipped' "$JUNIT" | wc -l | tr -d ' ')"
|
||||
if [ "${total:-0}" -gt 0 ] && [ "${skipped:-0}" -ge "${total:-0}" ]; then
|
||||
err "CLUSTER_AVAILABLE=true, but the L2/L3 suite SKIPPED every one of its ${total} tests"
|
||||
err " (scaffold mode -- run.sh could not reach a cluster and exited 0)."
|
||||
err " This stage will NOT report that as a pass. Either the cluster is unreachable from"
|
||||
err " this runner, or CLUSTER_AVAILABLE is set when it should not be."
|
||||
err " NOTE: the in-cluster trigger path is still UNIMPLEMENTED -- see"
|
||||
err " docs/plans/l2-l3-conformance-ci-wiring.md. Setting CLUSTER_AVAILABLE=true does not"
|
||||
err " by itself make L2/L3 run; it only makes this stage attempt a LOCAL run that the CI"
|
||||
err " runner has no credential for (automountServiceAccountToken: false, by design)."
|
||||
summary "- conformance-l2-l3: **BLOCKED — declared available but every test skipped** (not a pass)"
|
||||
exit 1
|
||||
fi
|
||||
log "L2/L3 executed: ${total} testcase(s), ${skipped} skipped"
|
||||
fi
|
||||
|
||||
ok "L2/L3 conformance passed for touched subset"
|
||||
summary "- conformance-l2-l3: **passed**"
|
||||
+41
@@ -112,6 +112,8 @@ run_stage lint "${CI_DIR}/lint.sh"
|
||||
run_stage l0 "${CI_DIR}/l0.sh"
|
||||
run_stage render-golden "${CI_DIR}/render-golden.sh"
|
||||
run_stage l1 "${CI_DIR}/l1.sh"
|
||||
run_stage approval-policy "${CI_DIR}/approval-policy.sh"
|
||||
run_stage gitea-settings "${CI_DIR}/gitea-settings.sh"
|
||||
run_stage conformance "${CI_DIR}/conformance.sh"
|
||||
|
||||
echo >&2
|
||||
@@ -188,6 +190,45 @@ case "${OUTCOME[spec-immutability]}" in
|
||||
pass) ok "spec-immutability: green" ;;
|
||||
esac
|
||||
|
||||
# approval-policy: the branch requires NO human approvals (required_approvals: 0) -- the GATE
|
||||
# is the approver, per operator intent. This stage is the single exception: a change to the
|
||||
# pipeline itself, the admission policies, the Flux wiring, the SOPS recipients or the RBAC
|
||||
# cannot honestly be approved BY the thing it changes, so those paths still need a human.
|
||||
#
|
||||
# It never SKIPS -- it exits 0 or 1 only, and blocks when it cannot evaluate. That is the
|
||||
# opposite of gitea-settings' tolerance and it is deliberate: there, a missing credential is a
|
||||
# lock-out; here, a missing credential would let an unreviewed change to the gate merge.
|
||||
case "${OUTCOME[approval-policy]}" in
|
||||
fail) err "approval-policy: a sensitive change lacks a non-author approval -> blocks merge"; BLOCK=1 ;;
|
||||
skip) err "approval-policy: did not run -- it is fail-closed by design, so treat as a block"; BLOCK=1 ;;
|
||||
pass) ok "approval-policy: green" ;;
|
||||
esac
|
||||
|
||||
# gitea-settings: the ONLY check that can see Gitea's own database (branch protection +
|
||||
# runner registration). A FAIL is real drift and always blocks -- that is the whole point.
|
||||
# A SKIP means no token or unreachable Gitea, which is an UNWIRED CREDENTIAL, not a defect
|
||||
# in the change under test. It is deliberately NOT in the strict-skips loop below: making an
|
||||
# absent secret block every PR is precisely the lock-yourself-out failure this repo keeps
|
||||
# walking into (a required check with nothing able to satisfy it). So a skip denies a PASS
|
||||
# (exit 3) and says loudly what is unverified. Set REQUIRE_GITEA_SETTINGS=true once the
|
||||
# token is wired, and the skip becomes a block as it should.
|
||||
case "${OUTCOME[gitea-settings]}" in
|
||||
fail) err "gitea-settings: Gitea DRIFTED from infra/gitea/expected-branch-protection.json -> blocks merge"; BLOCK=1 ;;
|
||||
skip)
|
||||
if truthy "${REQUIRE_GITEA_SETTINGS:-false}"; then
|
||||
err "gitea-settings: skipped and REQUIRE_GITEA_SETTINGS=true -> blocks merge"
|
||||
summary "- gitea-settings: **SKIPPED** (no token) — blocks merge (REQUIRE_GITEA_SETTINGS=true)"
|
||||
BLOCK=1
|
||||
else
|
||||
warn "gitea-settings: skipped (no GITEA_SETTINGS_TOKEN) — branch protection and runner"
|
||||
warn " registration are UNVERIFIED. Both were silently wrong for a month. Wire the token"
|
||||
warn " (infra/gitea/set-ci-token.sh), then set REQUIRE_GITEA_SETTINGS=true."
|
||||
summary "- gitea-settings: **SKIPPED** (no token — protection/runners unverified; not a pass)"
|
||||
NOT_ESTABLISHED+=("gitea-settings")
|
||||
fi ;;
|
||||
pass) ok "gitea-settings: green" ;;
|
||||
esac
|
||||
|
||||
# L0/L1 stages: any failure blocks. A SKIP here means a pinned tool is missing or a seam is
|
||||
# unwired — NEVER green. In CI (strict) that is a defect and blocks; on a workstation it
|
||||
# degrades to UNDETERMINED, which is still non-zero and still not a pass. render-golden is a
|
||||
|
||||
Executable
+126
@@ -0,0 +1,126 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/gitea-settings.sh — Stage: the Gitea platform settings that gate this repo still
|
||||
# match what git declares.
|
||||
#
|
||||
# WHY THIS STAGE EXISTS AT ALL
|
||||
# ----------------------------
|
||||
# Every other check in this repo reads the repository, the rendered manifests, or the
|
||||
# cluster. NONE of them can see Gitea's own database — and that is where branch
|
||||
# protection and runner registration live. On 2026-09-29 that blind spot was measured:
|
||||
#
|
||||
# * `main` had NO branch protection, and
|
||||
# * ZERO Actions runners were registered,
|
||||
#
|
||||
# for roughly a month, while DECISIONS-LOG.md recorded both as working and every agent
|
||||
# and human in the loop repeated that claim because the repository said so. PR #1 — the
|
||||
# first pull request this repo ever had — sat `pending` for 30.6 minutes with nothing to
|
||||
# execute it. Later the same day `enable_status_check` and `required_approvals` were turned
|
||||
# on and STILL did not bite, because `block_admin_merge_override` was false; PRs #5-#9
|
||||
# merged with zero approvals recorded against `required_approvals: 1`.
|
||||
#
|
||||
# Three separate failures, one shape: a setting that READ as enforced while not being, with
|
||||
# no mechanism anywhere that could notice. Scripts that ASSERT the desired state
|
||||
# (infra/gitea/*.sh) do not detect drift from it. This stage is the detector.
|
||||
#
|
||||
# WHAT IT ASSERTS
|
||||
# 1. every key in infra/gitea/expected-branch-protection.json `.protection` matches the
|
||||
# LIVE rule at /repos/{repo}/branch_protections/{branch}, exactly;
|
||||
# 2. at least `.runners_online_min` Actions runners report status "online".
|
||||
#
|
||||
# (2) is not decoration. A required status check with no runner is the deadlock this repo
|
||||
# nearly walked into: every PR pends forever, including the one that would fix it.
|
||||
#
|
||||
# READ THE RULE OBJECT, NOT THE BRANCH VIEW. /repos/{repo}/branches/{branch} reports
|
||||
# `protected`, `enable_status_check` and `required_approvals` but NOT
|
||||
# `block_admin_merge_override` — it showed all-green while the rule was overridable by the
|
||||
# only account that merges. This stage queries /branch_protections/ deliberately.
|
||||
#
|
||||
# EXIT CODES (the gate contract)
|
||||
# 0 live Gitea matches the declaration
|
||||
# 1 DIVERGENCE — something drifted, or was never applied. Blocks the merge.
|
||||
# 3 SKIP — no token, or Gitea unreachable. Never a pass; see gate.sh, which treats this
|
||||
# skip like spec-immutability's (UNDETERMINED, exit 3) rather than a hard block, so
|
||||
# that an unwired credential cannot lock the repository.
|
||||
#
|
||||
# TO WIRE IT: create a Gitea repo secret GITEA_SETTINGS_TOKEN (a token that can read repo
|
||||
# settings) — infra/gitea/set-ci-token.sh does it. Until then this stage skips LOUDLY and
|
||||
# the drift it exists to catch is still undetected.
|
||||
set -euo pipefail
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "${CI_DIR}/lib.sh"
|
||||
|
||||
log "Stage: Gitea platform settings match the declaration"
|
||||
|
||||
DECL="${REPO_ROOT}/infra/gitea/expected-branch-protection.json"
|
||||
GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
|
||||
GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
|
||||
# SETTINGS_API_TOKEN, not GITEA_SETTINGS_TOKEN. Gitea RESERVES the `GITEA_` prefix for
|
||||
# secret names and rejects a PUT of one with HTTP 400 "invalid secret name" — the same way
|
||||
# GitHub reserves `GITHUB_`. The original name could therefore never be stored, and because
|
||||
# infra/gitea/set-ci-token.sh sent that PUT to /dev/null and printed "stored" unconditionally,
|
||||
# it looked like it had worked. A token was minted, no secret existed, and this stage would
|
||||
# have skipped forever. GITEA_SETTINGS_TOKEN is still honoured for a local run.
|
||||
TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
|
||||
|
||||
[ -f "$DECL" ] || die "declaration missing: ${DECL}"
|
||||
require_tool jq
|
||||
require_tool curl
|
||||
|
||||
if [ -z "$TOKEN" ]; then
|
||||
skip "SETTINGS_API_TOKEN is not set — cannot read the live Gitea rule."
|
||||
skip " Branch protection and runner registration are therefore UNVERIFIED. They were"
|
||||
skip " wrong, unnoticed, for a month (2026-09-29). Wire it: infra/gitea/set-ci-token.sh"
|
||||
summary "- gitea-settings: **SKIPPED — no token** (protection + runners UNVERIFIED; not a pass)"
|
||||
exit 3
|
||||
fi
|
||||
|
||||
api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
|
||||
|
||||
BRANCH="$(jq -r '.branch' "$DECL")"
|
||||
LIVE="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/branch_protections/${BRANCH}" || true)"
|
||||
if [ -z "$LIVE" ] || ! jq -e 'type == "object" and has("branch_name")' >/dev/null 2>&1 <<<"$LIVE"; then
|
||||
skip "could not read the branch-protection rule from ${GITEA_URL} (unreachable, or the token"
|
||||
skip " lacks permission). Not treating this as a pass."
|
||||
summary "- gitea-settings: **SKIPPED — Gitea unreadable** (not a pass)"
|
||||
exit 3
|
||||
fi
|
||||
|
||||
RC=0
|
||||
|
||||
# --- 1. every declared protection key must match the live rule, exactly ---------------
|
||||
while IFS= read -r key; do
|
||||
want="$(jq -Sc --arg k "$key" '.protection[$k]' "$DECL")"
|
||||
got="$(jq -Sc --arg k "$key" 'if has($k) then .[$k] else "<<ABSENT>>" end' <<<"$LIVE")"
|
||||
if [ "$want" = "$got" ]; then
|
||||
ok " ${key} = ${got}"
|
||||
else
|
||||
err " ${key}: DECLARED ${want} but LIVE is ${got}"
|
||||
RC=1
|
||||
fi
|
||||
done < <(jq -r '.protection | keys[]' "$DECL")
|
||||
|
||||
# --- 2. a runner must actually be online ----------------------------------------------
|
||||
MIN="$(jq -r '.runners_online_min // 1' "$DECL")"
|
||||
RUNNERS="$(api "${GITEA_URL}/api/v1/admin/actions/runners" || true)"
|
||||
N="$(jq -r '[(.runners//[])[]|select(.status=="online")]|length' <<<"${RUNNERS:-{\}}" 2>/dev/null || echo 0)"
|
||||
if [ "${N:-0}" -ge "$MIN" ]; then
|
||||
ok " runners online: ${N} (>= ${MIN})"
|
||||
else
|
||||
err " runners online: ${N}, DECLARED minimum ${MIN}."
|
||||
err " A required status check with no runner leaves every PR pending forever —"
|
||||
err " including the PR that would fix it. This is the 2026-09-29 outage condition."
|
||||
RC=1
|
||||
fi
|
||||
|
||||
if [ "$RC" -eq 0 ]; then
|
||||
ok "Gitea settings match the declaration."
|
||||
summary "- gitea-settings: **green** (branch protection + runners match \`infra/gitea/expected-branch-protection.json\`)"
|
||||
else
|
||||
err "Gitea has DRIFTED from infra/gitea/expected-branch-protection.json."
|
||||
err " Re-assert it (infra/gitea/require-gate.sh, block-admin-override.sh), or if the live"
|
||||
err " state is what you now want, change the DECLARATION in the same commit — deliberately."
|
||||
summary "- gitea-settings: **DRIFT** — live Gitea does not match the declaration; blocks merge"
|
||||
fi
|
||||
exit "$RC"
|
||||
@@ -0,0 +1,42 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — every authored admission policy is actually APPLIED (R-SEC-01, R-SEC-02)
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# WHY THIS EXISTS. `kyverno test` proves a policy is CORRECT. Nothing proved it was DEPLOYED.
|
||||
# platform/kyverno/ holds three lists and only platform-config/kustomization.yaml applies
|
||||
# anything; the other two are a reference copy and a namespace+HelmRelease render root. On
|
||||
# 2026-09-29 the R-SEC-02 boundary policy was added to the reference copy: it parsed, its
|
||||
# kyverno test passed, the gate passed, Flux reported the tier Ready at the new revision — and
|
||||
# the cluster ran the conformance-trigger ServiceAccount with nothing bounding it.
|
||||
#
|
||||
# WHOLE-REPO SCOPE, DELIBERATELY. The failure mode is a policy that is present but wired
|
||||
# nowhere, which by definition touches no requirement the diff attributes. A path-scoped run
|
||||
# would have skipped this check on the very commit that introduced the defect.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: admission-policies-actually-applied
|
||||
labels:
|
||||
conformance.home.k3s/R-SEC-01: "true"
|
||||
conformance.home.k3s/R-SEC-02: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-SEC-01,R-SEC-02"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Every ClusterPolicy authored under platform/kyverno/ is either referenced by
|
||||
platform-config/kustomization.yaml (so Flux applies it) or is a documented CI-only
|
||||
exemption that is genuinely not applied — so a policy cannot be authored, tested,
|
||||
green, and absent from the cluster.
|
||||
spec:
|
||||
steps:
|
||||
- name: every-authored-policy-is-applied-or-documented-ci-only
|
||||
description: >-
|
||||
Enumerate the ClusterPolicy files under platform/kyverno/ and assert each is wired
|
||||
into the Kustomization that actually applies Kyverno CRs, or exempted with a reason.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -0,0 +1,120 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — admission-policies-actually-applied
|
||||
# Serves: R-SEC-01, R-SEC-02 (admission enforcement is only real if it is APPLIED)
|
||||
#
|
||||
# WHY THIS EXISTS — a policy that ships green and never reaches the cluster.
|
||||
#
|
||||
# platform/kyverno/ holds THREE lists, and only one of them applies anything:
|
||||
# 1. platform/kyverno/admission/kustomization.yaml — reference only. Its own sibling says
|
||||
# "kept as reference; THIS file is the buildable render root".
|
||||
# 2. platform/kyverno/kustomization.yaml — that render root: namespace +
|
||||
# HelmRelease ONLY. It explicitly notes the ClusterPolicies are applied elsewhere.
|
||||
# 3. platform-config/kustomization.yaml — THE ONE. Kyverno CRs go here because
|
||||
# they need the CRDs from the HelmRelease in (2), so they reconcile a tier later.
|
||||
#
|
||||
# On 2026-09-29 a ClusterPolicy (conformance-trigger-job-shape, the R-SEC-02 boundary bounding
|
||||
# what the conformance CI trigger may create) was added to (1). Everything went green: the file
|
||||
# parsed, `kyverno test` exercised it BY PATH and passed, the gate passed, Flux reported
|
||||
# `platform` Ready at the new revision — and the policy was never in any applied set. The
|
||||
# cluster ended up running the conformance-trigger ServiceAccount with NOTHING bounding it,
|
||||
# which is precisely the state that identity's own RBAC header says must never exist.
|
||||
#
|
||||
# Nothing could have caught it. `kyverno test` proves a policy is CORRECT, not that it is
|
||||
# DEPLOYED. This check asserts the second thing.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN every file under platform/kyverno/ that declares a ClusterPolicy is enumerated,
|
||||
# THEN each one is either referenced by platform-config/kustomization.yaml (so Flux applies
|
||||
# it), or named in CI_ONLY below with a reason — and CI_ONLY itself may not name a
|
||||
# policy that IS applied, so the exemption list cannot rot into a lie either way.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
exec python3 - "$ROOT" <<'PY'
|
||||
import sys, subprocess
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES, CHECKS = [], 0
|
||||
|
||||
def check(cond, desc):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(desc)
|
||||
|
||||
# Policies authored for the CI L0 gate that are deliberately NOT applied in-cluster. Each needs
|
||||
# a reason, and each is asserted BELOW to really be absent from the applied set — an exemption
|
||||
# that silently starts being applied is as much a lie as one that silently stops.
|
||||
CI_ONLY = {
|
||||
"disallow-plaintext-secrets":
|
||||
"CI-only by design: it requires a top-level `sops` block, which is IMPOSSIBLE at "
|
||||
"admission because Flux decrypts before the Secret reaches the API server. Its "
|
||||
"in-cluster counterpart is admission/secret-provenance.yaml.",
|
||||
}
|
||||
|
||||
tracked = subprocess.run(["git", "ls-files", "-z", "--", "platform/kyverno/**/*.yaml",
|
||||
"platform/kyverno/*.yaml"],
|
||||
cwd=ROOT, capture_output=True, text=True, check=True)
|
||||
files = [ROOT / p for p in tracked.stdout.split("\0") if p]
|
||||
|
||||
# A ClusterPolicy file, identified without a YAML parse of every doc: cheap and sufficient.
|
||||
policies = {}
|
||||
for f in files:
|
||||
try:
|
||||
txt = f.read_text()
|
||||
except Exception:
|
||||
continue
|
||||
if "kind: ClusterPolicy" not in txt:
|
||||
continue
|
||||
for line in txt.splitlines():
|
||||
st = line.strip()
|
||||
if st.startswith("name:") and " name:" in line:
|
||||
policies[f.relative_to(ROOT).as_posix()] = st.split("name:", 1)[1].strip()
|
||||
break
|
||||
|
||||
check(len(policies) >= 6,
|
||||
f"(0) the scan found {len(policies)} ClusterPolicy file(s) under platform/kyverno/ — "
|
||||
f"too few means the scan broke, and every assertion below would be vacuous")
|
||||
|
||||
applied_txt = (ROOT / "platform-config" / "kustomization.yaml").read_text()
|
||||
|
||||
unapplied, wrongly_exempt = [], []
|
||||
for rel, name in sorted(policies.items()):
|
||||
# platform-config references these as ../platform/kyverno/...
|
||||
referenced = rel in applied_txt or rel.replace("platform/", "../platform/", 1) in applied_txt
|
||||
if name in CI_ONLY:
|
||||
if referenced:
|
||||
wrongly_exempt.append(f"{name} ({rel}) is listed CI_ONLY but IS applied by platform-config")
|
||||
elif not referenced:
|
||||
unapplied.append(f"{name} ({rel})")
|
||||
|
||||
check(not unapplied,
|
||||
f"(1) every in-cluster ClusterPolicy is referenced by platform-config/kustomization.yaml "
|
||||
f"— an unreferenced policy is authored, tested, green, and NEVER APPLIED {unapplied if unapplied else ''}")
|
||||
|
||||
check(not wrongly_exempt,
|
||||
f"(2) no CI_ONLY exemption names a policy that IS applied — a stale exemption hides a "
|
||||
f"real deployment {wrongly_exempt if wrongly_exempt else ''}")
|
||||
|
||||
for name, reason in CI_ONLY.items():
|
||||
check(bool(reason.strip()),
|
||||
f"(3) CI_ONLY['{name}'] carries a reason — an unexplained exemption is how this "
|
||||
f"list stops being reviewable")
|
||||
|
||||
check(CHECKS >= 4,
|
||||
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op")
|
||||
|
||||
if FAILURES:
|
||||
print(f"CHECK FAIL (admission-policies-actually-applied): {len(FAILURES)} of {CHECKS} assertions failed",
|
||||
file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
print(f"check OK: admission-policies-actually-applied — {CHECKS} assertions; "
|
||||
f"{len(policies)} ClusterPolicy file(s), {len(policies) - len(CI_ONLY)} applied by "
|
||||
f"platform-config, {len(CI_ONLY)} CI-only with reasons. Proves policies are DEPLOYED, "
|
||||
f"not that they are correct — `kyverno test` is what proves correctness.")
|
||||
PY
|
||||
@@ -38,8 +38,21 @@ agents. So the credential model must be decided by the operator before anything
|
||||
| Runner deployment | `apps/wave-1/gitea-actions.yaml` | act_runner + DinD (Docker executor). **`automountServiceAccountToken: false`** — the runner and its job containers hold NO SA token. Broad egress allowed (`- {}`), so it can *reach* the kube-API VIP `192.168.1.49:6443`, but has no credential. |
|
||||
| Secrets encryption | `.sops.yaml` (repo root) | age recipient `age175l8m…mfzg`; encrypts `data`/`stringData` of `*.sops.yaml` under `platform|apps|catalog/**` and `secrets/**`. Private key is the in-cluster `sops-age` Secret Flux uses. |
|
||||
|
||||
**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the dev cluster
|
||||
is LIVE (Phase 6). So `CLUSTER_AVAILABLE` is the only remaining gate, exactly as the blueprint says.
|
||||
**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the cluster is
|
||||
LIVE (and it is now PROD; dev was decommissioned 2026-09-12).
|
||||
|
||||
> **CORRECTION (2026-09-29): `CLUSTER_AVAILABLE` is NOT "the only remaining gate", and setting it
|
||||
> today is actively DANGEROUS.** Measured: with `CLUSTER_AVAILABLE=true` and no reachable cluster,
|
||||
> `conformance/run.sh` exits 0 in scaffold mode having marked all 54 tests `<skipped>`, and
|
||||
> `ci/conformance.sh` then printed `[ ok ] L2/L3 conformance passed for touched subset` — **rc 0, a
|
||||
> fake green**, on a merge gate that is now an enforcing required status check. The CI runner has
|
||||
> no cluster credential at all (`automountServiceAccountToken: false`, by design), so the "local
|
||||
> run" this stage attempts can never succeed; it can only skip and, before the guard added on
|
||||
> 2026-09-29, lie about it.
|
||||
>
|
||||
> `ci/conformance.sh` now BLOCKS (exit 1) when a cluster is declared but every test skipped. **Do
|
||||
> not set `CLUSTER_AVAILABLE=true` until §3's trigger path actually exists** — the variable does
|
||||
> not enable L2/L3, it only changes which way this stage fails.
|
||||
|
||||
---
|
||||
|
||||
|
||||
+32
-4
@@ -86,6 +86,32 @@ enabling enforcement and probing the push. `require-gate.sh` is the enforcement
|
||||
|
||||
Re-run `verify-then-enforce.sh` after any future change to the act-runner Deployment.
|
||||
|
||||
### `expected-branch-protection.json` + `ci/gitea-settings.sh` — the detector
|
||||
|
||||
`expected-branch-protection.json` **declares** the desired rule. `ci/gitea-settings.sh` runs as
|
||||
a merge-gate stage, reads the LIVE rule, and **fails the gate on any divergence** — plus asserts
|
||||
at least one runner is online. It is the only thing in this repository that can see Gitea's own
|
||||
database.
|
||||
|
||||
It reads `/branch_protections/main`, not `/branches/main`: the branch view does **not** expose
|
||||
`block_admin_merge_override`, and it showed all-green for hours while the rule was overridable.
|
||||
|
||||
Behaviour, proven by test rather than assumed:
|
||||
|
||||
| Situation | Exit | Effect |
|
||||
|---|---|---|
|
||||
| live matches the declaration | 0 | green |
|
||||
| any key drifted, or no runner online | 1 | **blocks the merge** |
|
||||
| no `GITEA_SETTINGS_TOKEN`, or Gitea unreachable | 3 | skips **loudly**; never a pass, never a block |
|
||||
|
||||
The skip is deliberately *not* a block: an unwired credential blocking every PR is the
|
||||
lock-yourself-out failure this repo keeps hitting. Wire the token with `set-ci-token.sh`, confirm
|
||||
the stage goes **green on a real PR**, and only then set `REQUIRE_GITEA_SETTINGS=true` in
|
||||
`gate.yml` so a future missing token blocks instead of skipping.
|
||||
|
||||
**Changing the declaration is how you change policy.** If the live state is what you now want,
|
||||
edit this JSON in the same commit — deliberately, in review — rather than letting the two drift.
|
||||
|
||||
### Why these scripts exist at all, and what is still missing
|
||||
|
||||
Gitea's repo settings live in **its own database**, not in anything Flux reconciles or the
|
||||
@@ -94,10 +120,12 @@ the root cause of the entire 2026-09-29 session: branch protection was off and t
|
||||
gone for a month while the log recorded both as working, and **no check existed that could
|
||||
have caught it.**
|
||||
|
||||
These scripts encode the desired state and can re-assert it, but they are not a detector. The
|
||||
durable fix is a conformance check that reads the live Gitea API and fails when it diverges
|
||||
from what is declared here — turning a class of silent, invisible failure into a loud one.
|
||||
**That check does not exist yet and is the highest-value remaining work in this directory.**
|
||||
These scripts encode the desired state and can re-assert it; `ci/gitea-settings.sh` above is the
|
||||
**detector** that turns divergence from a silent, invisible failure into a loud one. It closes the
|
||||
root cause of the 2026-09-29 session.
|
||||
|
||||
It is only as good as its wiring: **until `GITEA_SETTINGS_TOKEN` exists it skips**, and a skipped
|
||||
detector detects nothing. That is the remaining step.
|
||||
|
||||
## Historical sequence (kept for context)
|
||||
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"_intent": "The CI/CD pipeline is the approver. A human approval is required ONLY for changes that alter what the pipeline itself verifies, or the security posture it rests on -- because those are the changes CI cannot honestly gate: a PR that weakens the gate would be approved by the weakened gate.",
|
||||
"_rule": "If a PR touches any sensitive_paths glob, it needs an APPROVED review from someone who is not its author. Everything else merges on a green gate alone.",
|
||||
"_on_failure_to_evaluate": "BLOCK. Unlike gitea-settings, a skip here is a bypass, not a gap.",
|
||||
"sensitive_paths": [
|
||||
"ci/**",
|
||||
".gitea/**",
|
||||
"conformance/**",
|
||||
"clusters/**",
|
||||
"bootstrap/**",
|
||||
"infra/gitea/**",
|
||||
"platform/kyverno/**",
|
||||
"platform-config/kustomization.yaml",
|
||||
".sops.yaml",
|
||||
"**/*.sops.yaml",
|
||||
"**/rbac.yaml",
|
||||
"CLAUDE.md"
|
||||
],
|
||||
"_why_each": {
|
||||
"ci/**, .gitea/**": "the pipeline itself -- the thing doing the approving",
|
||||
"conformance/**": "the tests and the harness; changing them changes what green MEANS",
|
||||
"clusters/**, bootstrap/**": "Flux wiring -- what reconciles, in what order, with what substitution",
|
||||
"infra/gitea/**, platform-config/kustomization.yaml": "branch protection, the drift detector, and the list that decides which admission policies are actually applied",
|
||||
"platform/kyverno/**": "the admission policy engine",
|
||||
".sops.yaml, **/*.sops.yaml, **/rbac.yaml": "encryption recipients, secrets, and cluster authority",
|
||||
"CLAUDE.md": "the delivery contract agents follow"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"_comment": "DECLARED desired state of Gitea branch protection on main. ci/gitea-settings.sh fails the merge gate when the LIVE Gitea diverges from this. Gitea settings live in Gitea database - invisible to Flux, unreadable by every other check - so this file plus that check are the only drift detection that exists for them.",
|
||||
"_generated": "2026-09-29 from the live rule, after the session that found protection OFF and zero runners for a month",
|
||||
"branch": "main",
|
||||
"runners_online_min": 1,
|
||||
"protection": {
|
||||
"enable_push": false,
|
||||
"enable_push_whitelist": false,
|
||||
"enable_merge_whitelist": false,
|
||||
"enable_status_check": true,
|
||||
"status_check_contexts": [
|
||||
"gate / gate (pull_request)"
|
||||
],
|
||||
"required_approvals": 1,
|
||||
"enable_approvals_whitelist": false,
|
||||
"block_admin_merge_override": true,
|
||||
"block_on_rejected_reviews": true,
|
||||
"dismiss_stale_approvals": true
|
||||
}
|
||||
}
|
||||
Executable
+56
@@ -0,0 +1,56 @@
|
||||
#!/usr/bin/env bash
|
||||
# Mint a Gitea API token and store it as the repo Actions secret GITEA_SETTINGS_TOKEN,
|
||||
# so ci/gitea-settings.sh can read the live branch-protection rule during the gate.
|
||||
#
|
||||
# The token value is NEVER printed. It goes from the token API into the secrets API and
|
||||
# nowhere else -- not to stdout, not to a file, not into an agent transcript.
|
||||
#
|
||||
# Scope: `read:repository` + `read:admin` are what the check needs (the rule object, and the
|
||||
# runner list). Gitea token scopes are coarse; grant the least that works on your version and
|
||||
# confirm the check goes green rather than assuming the scope name.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
|
||||
# NOT GITEA_* — Gitea reserves that prefix for secret names and returns HTTP 400
|
||||
# "invalid secret name". The first version of this script used GITEA_SETTINGS_TOKEN, sent the
|
||||
# PUT to /dev/null, and printed "stored" regardless; the secret never existed and nobody knew.
|
||||
SECRET=SETTINGS_API_TOKEN
|
||||
NAME="ci-gitea-settings-$(date +%Y%m%d%H%M%S)"
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
basic() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
||||
|
||||
echo "==> minting token '${NAME}' as ${U}"
|
||||
TOKEN="$(basic -X POST -H 'Content-Type: application/json' \
|
||||
-d "{\"name\":\"${NAME}\",\"scopes\":[\"read:repository\",\"read:admin\"]}" \
|
||||
"https://${HOST}/api/v1/users/${U}/tokens" | jq -r '.sha1 // empty')"
|
||||
[ -n "$TOKEN" ] || { echo "token creation failed (check scopes for your Gitea version)" >&2; exit 1; }
|
||||
echo "==> got a ${#TOKEN}-character token (value not shown)"
|
||||
|
||||
echo "==> storing it as repo Actions secret ${SECRET}"
|
||||
code="$(basic -o /tmp/set-ci-token.out -w '%{http_code}' -X PUT -H 'Content-Type: application/json' \
|
||||
-d "$(jq -n --arg d "$TOKEN" '{data:$d}')" \
|
||||
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets/${SECRET}")"
|
||||
unset TOKEN
|
||||
if [ "$code" != 201 ] && [ "$code" != 204 ]; then
|
||||
echo " PUT failed: HTTP ${code} -- $(head -c 200 /tmp/set-ci-token.out)" >&2
|
||||
rm -f /tmp/set-ci-token.out; exit 1
|
||||
fi
|
||||
rm -f /tmp/set-ci-token.out
|
||||
|
||||
# VERIFY BY READING IT BACK. curl -sS exits 0 on an HTTP error, so without this the script
|
||||
# reports success for a store that never happened -- which is exactly what it did.
|
||||
names="$(basic "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets" \
|
||||
| jq -rc '[.. | objects | select(has("name")) | .name] | unique')"
|
||||
grep -q "$SECRET" <<<"$names" \
|
||||
|| { echo " stored HTTP ${code} but ${SECRET} is NOT in ${names} -- refusing to claim success" >&2; exit 1; }
|
||||
echo "==> stored and VERIFIED present: ${names}"
|
||||
|
||||
cat <<'NOTE'
|
||||
|
||||
Next:
|
||||
1. .gitea/workflows/gate.yml already passes this secret to the gate step.
|
||||
2. Open any PR and confirm the gate prints "gitea-settings: green" -- it SKIPS silently
|
||||
without the token, so a green run is the only proof the wiring works.
|
||||
3. THEN set REQUIRE_GITEA_SETTINGS=true in gate.yml, so a future missing token BLOCKS
|
||||
instead of skipping. Do that only after you have seen it green, or you lock the repo.
|
||||
NOTE
|
||||
@@ -34,6 +34,14 @@ resources:
|
||||
# policies, the next import is blocked in the gap.
|
||||
- ../platform/kyverno/policies/label-cdi-worker-pods.yaml
|
||||
- ../platform/kyverno/admission/secret-provenance.yaml
|
||||
# R-SEC-02 — bounds what the conformance CI trigger may create. MUST be listed HERE: this
|
||||
# Kustomization is what actually applies ClusterPolicies. Adding a policy only to
|
||||
# platform/kyverno/admission/kustomization.yaml is a SILENT NO-OP — that file is reference
|
||||
# only ("kept as reference; THIS file is the buildable render root" in
|
||||
# platform/kyverno/kustomization.yaml, which itself ships only the namespace + HelmRelease).
|
||||
# That is exactly how this policy shipped green and never reached the cluster on 2026-09-29,
|
||||
# leaving the conformance-trigger identity live with nothing bounding it.
|
||||
- ../platform/kyverno/admission/conformance-trigger-job-shape.yaml
|
||||
# cilium: L2 LB IP pool + announcement policy (cilium.io CRDs already registered by Cilium)
|
||||
- cilium/l2-ip-pool.yaml
|
||||
- cilium/l2-announcement-policy.yaml
|
||||
|
||||
Reference in new issue
Block a user