Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 2 Actions 1 Packages Projects Releases Wiki Activity

Merge pull request 'feat(sec): provision the narrow conformance CI trigger credential -- and refuse to arm a fake green' (#19) from feat/conformance-trigger-kubeconfig into main
gate / conformance (push) Has been cancelled
Details
gate / gate (push) Has been cancelled
Details

Browse Source 
Reviewed-on: #19
Reviewed-by: Keith Jackson <[email protected]>
This commit was merged in pull request #19.
This commit is contained in:
jakman85 committed 2026-09-30 03:58:47 +00:00
2 parents 6dd0914df9 ad5de2dba9
commit 762c3833fc
3 files changed
+769 -3
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

decisions/2026-09-29-conformance-trigger-kubeconfig.md
+125
View File
@@ -0,0 +1,125 @@
---
date: 2026-09-29
branch: feat/conformance-trigger-kubeconfig
title: The trigger credential script — it REFUSES, on purpose, and it is UNEXECUTED
---
## What changed
Delivered `infra/gitea/conformance-trigger-kubeconfig.sh` —
`docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 6 — plus its section in
`infra/gitea/README.md`. **The script is UNEXECUTED.** Minting a cluster credential and
changing Gitea variables are operator actions; nothing was minted, no Gitea object was
written, no `kubectl` mutation was made. Everything below is a claim about the *script*,
not about the platform.
*What it does.* Mints a token for `conformance/conformance-trigger` (TokenRequest), wraps
**only that token** with the cluster CA and server into a kubeconfig, stores it as the
Actions secret `CONFORMANCE_TRIGGER_KUBECONFIG` (legal name — **not** the reserved `GITEA_`
prefix), and sets `vars.CLUSTER_AVAILABLE` only as a separately-requested,
precondition-gated, TTY-confirmed final step. The token is never printed and never reaches
argv: it goes into a 0600 file and into the PUT **body file**. The script refuses to run
under `set -x`.
*The design decision worth reviewing: TWO precondition classes.*
- **mint-blocking (cluster-side):** SA/Role/RoleBinding present, and ClusterPolicy
`conformance-trigger-job-shape` present + `Enforce` + `failurePolicy: Fail` + Ready. If any
fails, **nothing is minted**. Rationale: Kubernetes puts no `use` check on a Job's
`serviceAccountName`, so a `create jobs` token in that namespace IS the privileged
`conformance-runner` identity unless the admission policy bounds it. RBAC cannot fix that
(plan §4); minting before the boundary is live would hand CI the escalation the boundary
exists to close.
- **arm-blocking (repo-side):** digest-pinned `CONFORMANCE_RUNNER_IMAGE`, an init-container
plus the admitted image in `chainsaw-job.yaml`, and `CONFORMANCE_TRIGGER_KUBECONFIG`
referenced in `ci/conformance.sh` **code rather than a comment**. These block only the
variable, because storing the secret early is harmless and is the plan's own order.
**Measured in the tree at the time of writing: all three arm-blocking checks FAIL** (no
`CONFORMANCE_RUNNER_IMAGE` in `conformance/versions.env`; `chainsaw-job.yaml` is still the
skeleton with an empty `workspace` emptyDir and the stock chainsaw image; `ci/conformance.sh`
names the trigger kubeconfig only in its TODO comment). So on a live run today the script
would provision the secret and then **refuse to arm `CLUSTER_AVAILABLE`**, naming each
missing item — which is the intended behaviour, not a defect. §3 items 1, 2 and 4 are the
work that unblocks it.
*Both writes are verified the way `set-ci-token.sh` learned to.* `curl -sS` exits 0 on an
HTTP error, so status alone proves nothing: each write checks `%{http_code}` **and** reads
the object back. For the secret that is presence-by-name only (Gitea never returns secret
values) and the script says so rather than implying the content was checked; for the variable
it asserts the value reads back as exactly `true`.
*Noted, not fixed at the time (since fixed on this branch).* `infra/gitea/README.md`'s
detector table still named `GITEA_SETTINGS_TOKEN` — the illegal name PR #15 replaced with
`SETTINGS_API_TOKEN`. The second commit on this branch corrects the README's table and its
closing "remaining step" line, and records what `ci/gitea-settings.sh` actually does
(`TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"` — the old name survives only as a
local-run environment variable, never as CI wiring, because the secret that would feed it
cannot be created at all). Documentation only, no behaviour change. Still NOT fixed, because
it is outside this PR's diff: `ci/gitea-settings.sh:45` instructs the reader to create a repo
secret `GITEA_SETTINGS_TOKEN`, which is impossible.
*Rework, 2026-09-29.* This branch originally appended ~99 lines to the tail of
`DECISIONS-LOG.md`. That is the hunk every concurrent branch collided on and the sole reason
PR #19 was `mergeable: false`. The branch was rebuilt on today's `main` with the
`DECISIONS-LOG.md` change dropped entirely, and this file is that handoff moved into
`decisions/`. The changes to `infra/gitea/**` are byte-identical to the original three
commits; the commit that contained only the log append was dropped and its substance folded
in below.
## What I verified
*Of the script, before it stores anything:* it PROVES the credential narrow — `auth whoami`
must return `system:serviceaccount:conformance:conformance-trigger`; a real `get jobs`
round-trip must succeed; then an `auth can-i` matrix asserts BOTH halves — granted
(`create/get jobs`, `get pods`, `get pods/log`) and **denied** (`delete jobs`, `create
pods/exec`, `get secrets`, `create jobs` in `default`, `get nodes`,
`'*' '*' --all-namespaces`). A mismatch aborts before anything is stored, and the message
says explicitly **not** to widen `conformance/incluster/rbac.yaml` to make it pass. That is a
property of the code as read, not of an execution.
*Of PR #19's CI, measured rather than assumed — and it contradicts the brief.* The brief for
this work stated this PR's CI would FAIL `approval-policy` because `infra/gitea/**` is
security-relevant. Measured on the PR head commit's `/statuses`:
gate / gate (pull_request) success Successful in 1m34s
gate / conformance (pull_request) pending Blocked by required conditions
`.gitea/workflows/` on `main` contains only `gate.yml` and the dormant `merge-gate.yaml`.
**No path-scoped approval policy exists**, so nothing machine-enforces "security-relevant
paths need a human"; that rule rests entirely on `required_approvals: 1` plus a reviewer
actually reading the diff. Another instance of the CLAUDE.md §2b failure mode — a control
believed to exist because a brief said so. If the policy is intended, it is missing and needs
its own change. And per this repo's own rule, that green `gate` tick is **not** evidence this
script works: the gate never executes `infra/gitea/**`, and the L2/L3 stage skipped.
*Gate on the reworked branch (pinned toolbox, from the agent worktree, `DEV_KUBECONFIG=/dev/null`
plus the `ENGINE` bind-mount shim per CLAUDE.md §3):* `MERGE GATE: UNDETERMINED`, **rc 3**.
Diff base `origin/main @ c59fe9f923c4` ... this branch's head; touched requirement resolved as
`R-ENV-01`. `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden` and `l1` all
green; the gate decision names exactly one stage that did not run — `gitea-settings`
(`SETTINGS_API_TOKEN` is a credential this agent does not hold, unwired repo-wide). The
`conformance` stage also reported SKIPPED (`CLUSTER_AVAILABLE!=true`, Phase 0). The new
`decisions-log` stage — merged to `main` as PR #24 after this rework began — validated this very
file: "archive frozen, 2 entries well-formed", with `DECISIONS-LOG.md` unchanged and matching
`decisions/ARCHIVE.sha256`. rc 3 is UNDETERMINED, not green, and is not reported as green here.
## What I did NOT verify
**The script has never executed.** Not one line of it has run against Gitea or against the
cluster. Only `bash -n` was run (clean); `shellcheck` is not on the host and there is no
shellcheck stage in the gate.
- The Gitea variables API shape (POST-creates / PUT-updates, body field `value` not `data`)
is from **documentation, not measurement**. The script tries POST then falls back to PUT,
prints the response body on failure, and treats the read-back as the authority.
- The Kyverno readiness field (`.status.ready` vs. a Ready condition) is handled both ways
and treats "neither present" as a block rather than assuming ready — but which form this
cluster actually reports is **unmeasured**.
- Whether the kubeconfig's embedded server URL is reachable from the runner pod **cannot be
known from here** — hence `--server`, and a hard refusal on a loopback address.
- No credential was minted, no Actions secret was created, `vars.CLUSTER_AVAILABLE` was not
set, and no `can-i` matrix was ever run against a real token. The narrowness argument above
is read from the RBAC and the script, not observed.
- The gate proves only its L0/L1 offline subset. L2/L3 conformance did not run, so nothing
here is evidence about runtime behaviour.
infra/gitea/README.md
+69 -3
View File
@@ -102,7 +102,19 @@ Behaviour, proven by test rather than assumed:
|---|---|---|
| live matches the declaration | 0 | green |
| any key drifted, or no runner online | 1 | **blocks the merge** |
| no `GITEA_SETTINGS_TOKEN`, or Gitea unreachable | 3 | skips **loudly**; never a pass, never a block |
| no `SETTINGS_API_TOKEN`, or Gitea unreachable | 3 | skips **loudly**; never a pass, never a block |
> **The secret is `SETTINGS_API_TOKEN`.** An earlier draft of this table named
> `GITEA_SETTINGS_TOKEN`, which **cannot exist as a Gitea secret at all** — Gitea reserves the
> `GITEA_` prefix and returns HTTP 400 `invalid secret name`. That is the whole 2026-09-29
> inert-detector story, so the wrong name surviving here is worth correcting rather than
> reading past. Measured in `ci/gitea-settings.sh`:
>
> TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
>
> so `GITEA_SETTINGS_TOKEN` is still honoured — but **only as an environment variable for a
> LOCAL run**. It can never be the CI wiring, because the secret that would feed it cannot be
> created. In CI, only `SETTINGS_API_TOKEN` exists.
The skip is deliberately *not* a block: an unwired credential blocking every PR is the
lock-yourself-out failure this repo keeps hitting. Wire the token with `set-ci-token.sh`, confirm
@@ -112,6 +124,59 @@ the stage goes **green on a real PR**, and only then set `REQUIRE_GITEA_SETTINGS
**Changing the declaration is how you change policy.** If the live state is what you now want,
edit this JSON in the same commit — deliberately, in review — rather than letting the two drift.
## `conformance-trigger-kubeconfig.sh` — the narrow L2/L3 trigger credential
Implements `docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 6 (= §5 step 7): mint a
ServiceAccount token for `conformance/conformance-trigger`, wrap **only that token** in a
kubeconfig, store it as the Actions secret `CONFORMANCE_TRIGGER_KUBECONFIG`, and — separately,
and only on request — set `vars.CLUSTER_AVAILABLE=true`.
./infra/gitea/conformance-trigger-kubeconfig.sh --dry-run # measure only; writes nothing
./infra/gitea/conformance-trigger-kubeconfig.sh # mint + store the secret
./infra/gitea/conformance-trigger-kubeconfig.sh --arm-cluster-available # + the variable
Never a cluster-admin kubeconfig, never a copy of the operator's own. The token is never
printed: it goes from the TokenRequest API into a 0600 file, into the request **body file**
(`--data @file`, so not even into `ps`), and nowhere else. The script refuses to run under
`set -x`.
**Its preconditions are split in two, and the split is the point.**
| Class | Checks | Blocks |
|---|---|---|
| **mint-blocking** (cluster) | SA/Role/RoleBinding exist; ClusterPolicy `conformance-trigger-job-shape` is present, `Enforce`, `failurePolicy: Fail`, Ready | the **mint** — nothing is created |
| **arm-blocking** (repo) | digest-pinned `CONFORMANCE_RUNNER_IMAGE`; `chainsaw-job.yaml` has an init-container and the admitted image; `ci/conformance.sh` names the trigger kubeconfig in **code, not a comment** | only `CLUSTER_AVAILABLE` |
Why mint-blocking: Kubernetes puts **no `use` check on a Job's `serviceAccountName`**, so a
token that can `create jobs` in `conformance` can run arbitrary code as the privileged
`conformance-runner` SA. RBAC cannot close that; the Kyverno policy is the boundary. Minting
while it is absent or in `Audit` hands CI the privileged identity, so the script will not do it.
Why arm-blocking is separate: storing the secret before the runner image / Job template / CI
trigger flow exist is harmless (the job is gated off), and is the plan's intended order. So
running this today provisions the secret and then **refuses to arm the variable**, naming the
missing item.
**`CLUSTER_AVAILABLE` is not a "turn conformance on" switch.** Measured 2026-09-29: set with no
trigger path, `run.sh` scaffold-skipped all 54 tests at rc 0 and `ci/conformance.sh` printed
"L2/L3 conformance passed" — a fake green on an enforcing required check. `ci/conformance.sh`
now blocks that case, so arming early does not enable L2/L3; it only breaks every PR. Hence:
`--arm-cluster-available` plus all preconditions green plus a typed confirmation on a TTY, or
nothing happens.
Both writes are verified the way `set-ci-token.sh` learned to: check `%{http_code}` **and**
read the object back. For the secret that proves presence by name only (Gitea never returns
secret values); for the variable it proves the value is exactly `true`, which is what
`gate.yml` compares against.
The token **expires** — it is a TokenRequest token, and the script decodes the JWT `exp` claim
to report the real expiry rather than trusting `--duration` (the API server caps it). Re-run to
rotate. After expiry the trigger gets 401 and the stage fails loudly rather than skipping.
`--server` exists because the embedded API-server URL must be reachable **from the CI runner
pod** — the VIP — which is not necessarily the address your workstation uses. The script
refuses a loopback server outright.
### Why these scripts exist at all, and what is still missing
Gitea's repo settings live in **its own database**, not in anything Flux reconciles or the
@@ -124,8 +189,9 @@ These scripts encode the desired state and can re-assert it; `ci/gitea-settings.
**detector** that turns divergence from a silent, invisible failure into a loud one. It closes the
root cause of the 2026-09-29 session.
It is only as good as its wiring: **until `GITEA_SETTINGS_TOKEN` exists it skips**, and a skipped
detector detects nothing. That is the remaining step.
It is only as good as its wiring: **until `SETTINGS_API_TOKEN` exists it skips**, and a skipped
detector detects nothing. (That secret was stored and verified in PR #15; this sentence predates
it. The name matters — see the note under the behaviour table above.)
## Historical sequence (kept for context)
infra/gitea/conformance-trigger-kubeconfig.sh
Executable
+575
View File
@@ -0,0 +1,575 @@
#!/usr/bin/env bash
# infra/gitea/conformance-trigger-kubeconfig.sh
#
# OPERATOR script for docs/plans/l2-l3-conformance-ci-wiring.md §3 item 6 (= §5 step 7):
# provision the NARROW CI trigger credential for L2/L3 conformance, and — only when the
# rest of the trigger path actually exists — arm `vars.CLUSTER_AVAILABLE`.
#
# What it does, in order:
# 1. measures the preconditions (cluster-side AND repo-side) instead of assuming them
# 2. mints a bounded-lifetime ServiceAccount token for `conformance-trigger` in ns
# `conformance` (kubectl create token -> TokenRequest API)
# 3. assembles a kubeconfig containing ONLY that token + the cluster CA + the API server
# 4. PROVES the kubeconfig is the narrow identity (auth whoami + an allow/deny matrix)
# 5. stores it as the Gitea Actions repo secret CONFORMANCE_TRIGGER_KUBECONFIG, and
# VERIFIES the store by reading the secret list back
# 6. sets vars.CLUSTER_AVAILABLE=true ONLY under --arm-cluster-available, only with every
# precondition green, and only after a typed confirmation. Otherwise it REFUSES.
#
# NOTHING SECRET IS EVER PRINTED. The token exists in a shell variable and in a 0600 file
# under a 0700 temp dir that is removed on exit; it reaches the Gitea API in a request-body
# FILE (`--data @file`), never in argv, never on stdout. This script prints names, byte
# lengths, HTTP status codes, the token's expiry timestamp, and yes/no authorization answers.
#
# ----------------------------------------------------------------------------------------
# THREE HARD-WON LESSONS THIS SCRIPT IS BUILT AROUND
# ----------------------------------------------------------------------------------------
# 1. GITEA RESERVES THE `GITEA_` PREFIX FOR SECRET NAMES.
# `PUT .../actions/secrets/GITEA_ANYTHING` returns HTTP 400 {"message":"invalid secret
# name"}. That is why set-ci-token.sh stores SETTINGS_API_TOKEN and not
# GITEA_SETTINGS_TOKEN. `CONFORMANCE_TRIGGER_KUBECONFIG` is legal, and is the exact name
# .gitea/workflows/{gate.yml,merge-gate.yaml} already reference — do not "tidy" it.
#
# 2. `curl -sS` EXITS 0 ON AN HTTP ERROR. The first version of set-ci-token.sh sent its PUT
# to /dev/null and printed "stored" unconditionally; the secret never existed and nobody
# knew for hours. So every write here captures `-w '%{http_code}'`, checks it, AND reads
# the object back afterwards. A store is not claimed unless the read-back shows it.
#
# 3. `CLUSTER_AVAILABLE` IS A LOADED GUN — SEE THE 2026-09-29 CORRECTION IN THE PLAN.
# `conformance/run.sh` exits 0 in scaffold mode with all 54 tests `<skipped>` when it
# cannot reach a cluster. Measured on 2026-09-29: with CLUSTER_AVAILABLE=true and no
# trigger path, ci/conformance.sh printed "[ ok ] L2/L3 conformance passed" at rc 0 — a
# FAKE GREEN on an enforcing required status check. ci/conformance.sh now BLOCKS that
# case (exit 1), so today the variable does not enable L2/L3: it only changes which way
# the stage fails. Arming it before §3 items 1-4 land therefore buys nothing and breaks
# every PR. This script refuses to do it.
#
# ----------------------------------------------------------------------------------------
# WHY THE PRECONDITIONS ARE SPLIT INTO TWO CLASSES (deliberate, and load-bearing)
# ----------------------------------------------------------------------------------------
# MINT-BLOCKING (cluster-side). Kubernetes places NO `use` check on a Job's
# `serviceAccountName`, so a token with plain `create jobs` in ns `conformance` can create a
# Job running as the PRIVILEGED `conformance-runner` SA with an arbitrary command. The
# trigger Role is only the necessary half; the sufficient half is the Kyverno ClusterPolicy
# `conformance-trigger-job-shape` (platform/kyverno/admission/), which must be present AND
# Enforce AND Ready. Minting this token while that policy is absent or in Audit hands CI a
# privileged identity. So those checks block the MINT itself — nothing is created if they
# fail. (See conformance/incluster/rbac.yaml's trigger header and plan §4.)
#
# ARM-BLOCKING (repo-side). The runner image, the Job template's init-container, and
# ci/conformance.sh's in-cluster trigger flow (plan §3 items 1, 2, 4) are what make a real
# L2/L3 run possible. Storing the kubeconfig secret without them is harmless — the
# conformance job is gated on the variable and never starts — and the plan explicitly lands
# the secret before the switch. So these checks block ONLY the variable.
#
# Net effect: you can safely run this script today to provision the secret. It will then
# REFUSE to arm the variable, and tell you exactly which repo-side item is missing.
#
# ----------------------------------------------------------------------------------------
# USAGE
# ----------------------------------------------------------------------------------------
# ./infra/gitea/conformance-trigger-kubeconfig.sh --dry-run
# Read-only. Measures every precondition, mints nothing, writes nothing.
# ./infra/gitea/conformance-trigger-kubeconfig.sh [--server URL]
# Mint + store the secret. Does NOT touch CLUSTER_AVAILABLE.
# ./infra/gitea/conformance-trigger-kubeconfig.sh --arm-cluster-available
# The above, then set vars.CLUSTER_AVAILABLE=true after a typed confirmation —
# refused unless every precondition is green.
#
# Options:
# --dry-run measure only; no TokenRequest, no PUT/POST
# --arm-cluster-available additionally set vars.CLUSTER_AVAILABLE=true (guarded)
# --server URL API server URL to embed (default: the operator context's own)
# --context NAME kube context to read the CA/server from (default: k3s-prod)
# --duration D requested token lifetime (default: 720h = 30d)
# -h | --help this text
#
# ON --server: the embedded server URL must be reachable FROM THE CI RUNNER POD, which is
# not necessarily the address your workstation uses. The runner has broad egress and reaches
# the kube-API VIP (per the plan: 192.168.1.49:6443). If your operator kubeconfig points at
# a workstation-only name or a localhost tunnel, pass --server explicitly; the script warns
# on a loopback server but cannot know your topology for you.
#
# ON TOKEN LIFETIME: this is a TokenRequest token and it EXPIRES. The API server may cap the
# requested duration, so the script decodes the JWT `exp` claim and reports the ACTUAL
# expiry rather than trusting --duration. Re-run this script to rotate. After expiry the
# trigger gets 401 and the conformance stage FAILS (loudly) rather than skipping — which is
# the correct direction, but put the expiry date in your calendar.
#
# DELIBERATELY NOT DONE: SOPS-sealing the token into secrets/conformance-trigger.sops.yaml
# (plan §3 item 6's "for reproducibility/rotation"). A bearer token with a 30-day life
# cannot usefully survive the branch -> PR -> review -> merge flow (CLAUDE.md §6), and
# committing one buys re-issuance we get for free by re-running this script against the
# live SA. If the operator still wants it sealed, that is a separate reviewed change.
#
# FORBIDDEN HERE, ON PURPOSE: no cluster-admin kubeconfig, no copy of the operator's own
# kubeconfig, no widening of conformance/incluster/rbac.yaml, no mutating kubectl beyond the
# TokenRequest itself, no `kubectl apply`.
set -euo pipefail
# xtrace would print the token. Refuse rather than leak.
case "$-" in
*x*) echo "FATAL: refusing to run under 'set -x' — xtrace would print the token" >&2; exit 1 ;;
esac
umask 077
HOST=gitea.home.keithjackson.dev
OWNER=Home-Network
REPO=home-network-k3s-impl
# Legal secret name: NOT prefixed GITEA_ (lesson 1), and the name the workflows already use.
SECRET=CONFORMANCE_TRIGGER_KUBECONFIG
VARIABLE=CLUSTER_AVAILABLE
NS=conformance
SA=conformance-trigger
RUNNER_SA=conformance-runner
POLICY=conformance-trigger-job-shape
# The image prefix the admission policy admits. Kept in sync with
# platform/kyverno/admission/conformance-trigger-job-shape.yaml by the P2 check below.
RUNNER_IMAGE_PREFIX='gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:'
KCTX=k3s-prod
DURATION=720h
DRY_RUN=false
ARM=false
SERVER_OVERRIDE=""
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
step() { printf '\n==> %s\n' "$*"; }
info() { printf ' %s\n' "$*"; }
warn() { printf ' WARN: %s\n' "$*" >&2; }
die() { printf 'FATAL: %s\n' "$*" >&2; exit 1; }
usage() {
cat <<'USAGE'
conformance-trigger-kubeconfig.sh — provision the narrow L2/L3 conformance CI trigger credential.
--dry-run measure every precondition; mint nothing, write nothing (read-only)
--arm-cluster-available additionally set vars.CLUSTER_AVAILABLE=true (guarded + confirmed)
--server URL API server URL to embed (default: the operator context's own;
must be reachable FROM THE CI RUNNER POD — the kube-API VIP)
--context NAME kube context to read the CA/server from (default: k3s-prod)
--duration D requested token lifetime (default: 720h)
-h | --help this text
Read the header of this file before running it: the CLUSTER_AVAILABLE guard, the split between
mint-blocking and arm-blocking preconditions, and why nothing secret is ever printed.
USAGE
}
while [ $# -gt 0 ]; do
case "$1" in
--dry-run) DRY_RUN=true ;;
--arm-cluster-available) ARM=true ;;
--server) SERVER_OVERRIDE="${2:?--server needs a URL}"; shift ;;
--context) KCTX="${2:?--context needs a name}"; shift ;;
--duration) DURATION="${2:?--duration needs a value}"; shift ;;
-h|--help) usage; exit 0 ;;
*) die "unknown argument: $1 (try --help)" ;;
esac
shift
done
if $DRY_RUN && $ARM; then die "--dry-run and --arm-cluster-available are contradictory"; fi
for t in kubectl jq curl git base64; do
command -v "$t" >/dev/null || die "missing required tool: $t"
done
WORK="$(mktemp -d)"
KC="${WORK}/conformance-trigger.kubeconfig"
cleanup() { rm -rf "$WORK"; }
trap cleanup EXIT
trap 'cleanup; exit 130' INT TERM
# ========================================================================================
# 0. Gitea credentials — from the credential helper git already has for this host, exactly
# as branch-protection.sh / set-ci-token.sh / require-gate.sh do. Never in argv: curl
# reads them from a --config file descriptor.
# ========================================================================================
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"
P="$(sed -n 's/^password=//p' <<<"$creds")"
unset creds
[ -n "$U" ] && [ -n "$P" ] || die "no stored credential for ${HOST} (git credential fill returned nothing)"
basic() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
API="https://${HOST}/api/v1/repos/${OWNER}/${REPO}"
# ========================================================================================
# 1. PRECONDITIONS — measured, never assumed (CLAUDE.md §2b: query the thing itself).
# Two independent tallies, per the split documented in the header.
# ========================================================================================
MINT_FAIL=0
ARM_FAIL=0
pass() { printf ' [ ok ] %s\n' "$*"; }
fail_mint() { printf ' [MINT-BLOCK] %s\n' "$*" >&2; MINT_FAIL=$((MINT_FAIL + 1)); }
fail_arm() { printf ' [ARM-BLOCK] %s\n' "$*" >&2; ARM_FAIL=$((ARM_FAIL + 1)); }
step "preconditions — cluster side (these block the MINT: without them the token IS privileged)"
kubectl --context "$KCTX" get --raw /version >/dev/null 2>&1 \
|| die "cannot reach the API server via context '${KCTX}' — nothing below can be measured"
pass "API server reachable via context ${KCTX}"
# C1: the trigger identity exists (created by Flux from conformance/incluster/rbac.yaml via
# clusters/base/conformance.yaml). We print object names only, never contents.
if kubectl --context "$KCTX" -n "$NS" get serviceaccount "$SA" -o name >/dev/null 2>&1; then
pass "ServiceAccount ${NS}/${SA} exists"
else
fail_mint "ServiceAccount ${NS}/${SA} is ABSENT — is the 'conformance' Flux Kustomization reconciled?"
fi
for kind in role rolebinding; do
if kubectl --context "$KCTX" -n "$NS" get "$kind" "$SA" -o name >/dev/null 2>&1; then
pass "${kind} ${NS}/${SA} exists"
else
fail_mint "${kind} ${NS}/${SA} is ABSENT — the token would authenticate but authorize nothing"
fi
done
# C2: THE SECURITY BOUNDARY. Present, Enforce, and Ready. Anything else and we do not mint.
# `validationFailureAction: Audit` is not a boundary; an unready policy is not a boundary.
if pol="$(kubectl --context "$KCTX" get clusterpolicy "$POLICY" -o json 2>/dev/null)"; then
action="$(jq -r '.spec.validationFailureAction // .spec.rules[0].validate.failureAction // "UNSET"' <<<"$pol")"
failpol="$(jq -r '.spec.failurePolicy // "UNSET"' <<<"$pol")"
# Kyverno moved readiness from .status.ready to a Ready condition; accept either, and
# treat "neither key present" as UNMEASURABLE -> block (fail closed, do not assume ready).
ready="$(jq -r '
if (.status.ready != null) then (.status.ready | tostring)
elif (.status.conditions != null) then
((.status.conditions[] | select(.type == "Ready") | .status) // "UNKNOWN")
else "UNKNOWN" end' <<<"$pol")"
info "ClusterPolicy ${POLICY}: validationFailureAction=${action} failurePolicy=${failpol} ready=${ready}"
[ "$action" = "Enforce" ] || fail_mint "ClusterPolicy ${POLICY} is '${action}', not Enforce — the trigger would be unbounded"
[ "$failpol" = "Fail" ] || fail_mint "ClusterPolicy ${POLICY} failurePolicy='${failpol}', not Fail — a Kyverno outage would admit arbitrary Jobs"
case "$ready" in
true|True) pass "ClusterPolicy ${POLICY} reports Ready" ;;
*) fail_mint "ClusterPolicy ${POLICY} does not report Ready (got '${ready}') — refusing to assume it is loaded" ;;
esac
else
fail_mint "ClusterPolicy ${POLICY} is ABSENT — plan §4's boundary does not exist; a token with 'create jobs' here can run arbitrary code as ${RUNNER_SA}"
fi
step "preconditions — repo side (these block ONLY arming ${VARIABLE}; the secret may be stored first)"
# R1: the runner image is built and digest-pinned. Without it the Job cannot run run.sh at
# all (the stock chainsaw image has no bash/python3/kubectl/git) — plan §3 item 1.
VERSIONS="${REPO_ROOT}/conformance/versions.env"
if grep -Eq '^[[:space:]]*CONFORMANCE_RUNNER_IMAGE=.*@sha256:[0-9a-f]{64}' "$VERSIONS" 2>/dev/null; then
pass "CONFORMANCE_RUNNER_IMAGE is digest-pinned in conformance/versions.env"
else
fail_arm "conformance/versions.env has no digest-pinned CONFORMANCE_RUNNER_IMAGE (plan §3 item 1) — the Job cannot run ./run.sh"
fi
# R2: the Job template is the real thing, not the skeleton: the admitted image AND an
# init-container that populates the empty `workspace` volume with the suite at the PR SHA
# (plan §3 item 2). A template whose image the policy will not admit is a guaranteed denial.
JOBTPL="${REPO_ROOT}/conformance/incluster/chainsaw-job.yaml"
if grep -q 'initContainers' "$JOBTPL" 2>/dev/null; then
pass "chainsaw-job.yaml declares an initContainers block (workspace population)"
else
fail_arm "chainsaw-job.yaml has no initContainers — 'workspace' is an empty emptyDir and the suite is never delivered (plan §3 item 2)"
fi
if grep -q "$RUNNER_IMAGE_PREFIX" "$JOBTPL" 2>/dev/null; then
pass "chainsaw-job.yaml uses the digest-pinned image the admission policy admits"
else
fail_arm "chainsaw-job.yaml does not use ${RUNNER_IMAGE_PREFIX}… — every triggered Job would be DENIED by ${POLICY}"
fi
# R3: ci/conformance.sh actually drives the in-cluster Job. Today it mentions
# CONFORMANCE_TRIGGER_KUBECONFIG only in a TODO comment and otherwise runs run.sh LOCALLY in
# the CI job — where there is no credential at all (the runner sets
# automountServiceAccountToken: false by design), so it can only scaffold-skip. Hence the
# check must ignore comment lines: a mention in prose is not an implementation.
#
# ONE awk, NOT `grep -v … | grep -q …`: under `set -o pipefail` a successful `grep -q` exits
# early, the upstream grep takes EPIPE, and the pipeline reports non-zero — which would read
# here as "unwired" exactly when the seam IS wired. A single process cannot have that bug.
CONF_STAGE="${REPO_ROOT}/ci/conformance.sh"
if awk '!/^[[:space:]]*#/ && /CONFORMANCE_TRIGGER_KUBECONFIG/ { found = 1 } END { exit !found }' \
"$CONF_STAGE" 2>/dev/null; then
pass "ci/conformance.sh references CONFORMANCE_TRIGGER_KUBECONFIG in code (in-cluster trigger path wired)"
else
fail_arm "ci/conformance.sh mentions CONFORMANCE_TRIGGER_KUBECONFIG only in comments — the trigger path is UNIMPLEMENTED (plan §3 item 4); arming ${VARIABLE} now only changes which way the stage fails"
fi
info ""
info "precondition tally: mint-blocking failures=${MINT_FAIL} arm-blocking failures=${ARM_FAIL}"
if [ "$MINT_FAIL" -gt 0 ]; then
cat >&2 <<EOF
REFUSING TO MINT. ${MINT_FAIL} cluster-side precondition(s) failed.
Minting a token for ${NS}/${SA} while the admission boundary is absent/Audit/unready hands CI
an identity that can create a Job running as ${RUNNER_SA} (nodes/patch, pods/exec,
namespaces/patch) with an arbitrary command. That is precisely the escalation
docs/plans/l2-l3-conformance-ci-wiring.md §4 exists to close, and RBAC cannot close it.
Fix the cluster side first (reconcile platform/kyverno/admission/ and conformance/incluster/,
confirm the ClusterPolicy is Enforce + Ready), then re-run. Nothing was created.
EOF
exit 1
fi
if $ARM && [ "$ARM_FAIL" -gt 0 ]; then
cat >&2 <<EOF
REFUSING TO ARM ${VARIABLE}. ${ARM_FAIL} repo-side precondition(s) failed.
Measured 2026-09-29: with ${VARIABLE}=true and no working trigger path, conformance/run.sh
scaffold-skips all 54 tests at rc 0 and ci/conformance.sh used to print
"[ ok ] L2/L3 conformance passed" — a fake green on an enforcing required status check. The
guard now in ci/conformance.sh turns that into exit 1 instead, so arming today does not
enable L2/L3: it breaks every PR while proving nothing.
Land plan §3 items 1, 2 and 4 first. Re-run WITHOUT --arm-cluster-available to provision the
secret in the meantime — that part is safe and is step 6's intended order.
EOF
exit 1
fi
if $DRY_RUN; then
step "DRY RUN — stopping here. Nothing was minted, nothing was written."
info "secret that WOULD be written: ${SECRET} (repo ${OWNER}/${REPO})"
info "variable that WOULD be set: none — ${VARIABLE} is never touched in --dry-run"
# Read-only visibility into what is already there. Gitea never returns secret VALUES.
names="$(basic "${API}/actions/secrets" | jq -rc '[.. | objects | select(has("name")) | .name] | unique')" || true
info "existing Actions secret names: ${names:-<unreadable>}"
cur="$(basic -o "${WORK}/var.json" -w '%{http_code}' "${API}/actions/variables/${VARIABLE}")" || true
if [ "$cur" = 200 ]; then
info "existing ${VARIABLE} = $(jq -r '.data // "<no data field>"' "${WORK}/var.json")"
else
info "existing ${VARIABLE}: not set (HTTP ${cur})"
fi
exit 0
fi
# ========================================================================================
# 2. MINT the ServiceAccount token (TokenRequest). This is the only write this script makes
# against the cluster, and it creates no durable object.
# ========================================================================================
step "minting a ServiceAccount token for ${NS}/${SA} (requested duration ${DURATION})"
TOKEN="$(kubectl --context "$KCTX" -n "$NS" create token "$SA" --duration="$DURATION")" \
|| die "kubectl create token failed for ${NS}/${SA}"
[ -n "$TOKEN" ] || die "kubectl create token returned an empty token"
info "got a ${#TOKEN}-character token (value NOT shown)"
# Report the ACTUAL expiry from the JWT rather than trusting --duration: the API server caps
# TokenRequest lifetimes (--service-account-max-token-expiration) and silently shortens.
jwt_exp() {
local payload pad b64
payload="$(cut -d. -f2 <<<"$1")"
pad=$(( (4 - ${#payload} % 4) % 4 ))
b64="$(tr '_-' '/+' <<<"$payload")$(printf '%*s' "$pad" '' | tr ' ' '=')"
printf '%s' "$b64" | base64 -d 2>/dev/null | jq -r '.exp // empty' 2>/dev/null || true
}
exp="$(jwt_exp "$TOKEN")"
if [ -n "$exp" ]; then
info "token expires: $(date -u -d "@${exp}" '+%Y-%m-%dT%H:%M:%SZ' 2>/dev/null || echo "epoch ${exp}") — re-run this script to rotate"
else
warn "could not decode the token's exp claim; lifetime is UNVERIFIED (requested ${DURATION})"
fi
# ========================================================================================
# 3. ASSEMBLE the kubeconfig — SA token + cluster CA + server. Nothing else.
#
# The CA and the server URL are read from the operator's context, but the operator's
# client certificate/key are NEVER copied: only .cluster fields are extracted. `config
# view --raw` output is held in a variable and discarded, never printed.
# Built with jq (not `kubectl config set-credentials --token=…`) so the token never
# appears in argv where `ps` can see it.
# ========================================================================================
step "assembling the kubeconfig (SA token only — never cluster-admin, never the operator's)"
raw="$(kubectl --context "$KCTX" config view --raw --minify -o json)" \
|| die "could not read context '${KCTX}' from the operator kubeconfig"
SERVER="$(jq -r '.clusters[0].cluster.server // empty' <<<"$raw")"
CADATA="$(jq -r '.clusters[0].cluster["certificate-authority-data"] // empty' <<<"$raw")"
CAFILE="$(jq -r '.clusters[0].cluster["certificate-authority"] // empty' <<<"$raw")"
INSECURE="$(jq -r '.clusters[0].cluster["insecure-skip-tls-verify"] // false | tostring' <<<"$raw")"
unset raw
[ -n "$SERVER" ] || die "context '${KCTX}' has no cluster.server"
if [ -n "$SERVER_OVERRIDE" ]; then
info "overriding server ${SERVER} -> ${SERVER_OVERRIDE}"
SERVER="$SERVER_OVERRIDE"
fi
case "$SERVER" in
*localhost*|*127.0.0.1*|*://0.0.0.0*)
die "server '${SERVER}' is loopback — the CI runner pod cannot reach it. Pass --server with the address the runner uses (the kube-API VIP)." ;;
esac
if [ -z "$CADATA" ] && [ -n "$CAFILE" ]; then
[ -r "$CAFILE" ] || die "certificate-authority file '${CAFILE}' is not readable"
CADATA="$(base64 -w0 <"$CAFILE")"
fi
[ -n "$CADATA" ] || die "no cluster CA available for '${KCTX}' (insecure-skip-tls-verify=${INSECURE}); refusing to emit a kubeconfig that trusts anything"
[ "$INSECURE" = "false" ] || die "context '${KCTX}' sets insecure-skip-tls-verify; refusing to propagate that into CI"
info "server: ${SERVER}"
info "CA: ${#CADATA} base64 characters embedded (public certificate, not a secret)"
jq -n --arg srv "$SERVER" --arg ca "$CADATA" --arg tok "$TOKEN" --arg ns "$NS" --arg sa "$SA" '
{
apiVersion: "v1",
kind: "Config",
clusters: [ { name: "conformance", cluster: { server: $srv, "certificate-authority-data": $ca } } ],
users: [ { name: $sa, user: { token: $tok } } ],
contexts: [ { name: "conformance", context: { cluster: "conformance", user: $sa, namespace: $ns } } ],
"current-context": "conformance"
}' >"$KC"
chmod 600 "$KC"
info "kubeconfig written to the temp dir: $(wc -c <"$KC") bytes (content NOT shown)"
# Prove kubectl can PARSE the file we generated (JSON is valid YAML, but assert it rather
# than assume it) — this prints only the server we already printed.
parsed="$(kubectl --kubeconfig "$KC" config view -o jsonpath='{.clusters[0].cluster.server}')" \
|| die "kubectl cannot parse the generated kubeconfig"
[ "$parsed" = "$SERVER" ] || die "generated kubeconfig parses to server '${parsed}', expected '${SERVER}'"
pass "kubectl parses the generated kubeconfig"
# ========================================================================================
# 4. PROVE it is the narrow identity. Two independent proofs:
# a) who the token authenticates AS (SelfSubjectReview)
# b) an allow/deny matrix (SelfSubjectAccessReview) — both halves matter: "can create
# jobs" proves it works, "cannot get secrets / delete jobs / exec" proves it is not
# something wider that happens to include the trigger rights.
# All read-only. `auth can-i` answers from the API server's own authorizer.
# ========================================================================================
step "verifying the credential is the NARROW trigger identity"
EXPECT_USER="system:serviceaccount:${NS}:${SA}"
if who="$(kubectl --kubeconfig "$KC" auth whoami -o jsonpath='{.status.userInfo.username}' 2>/dev/null)"; then
[ "$who" = "$EXPECT_USER" ] || die "token authenticates as '${who}', expected '${EXPECT_USER}' — WRONG CREDENTIAL, nothing stored"
pass "authenticates as ${who}"
else
warn "'kubectl auth whoami' unavailable (older kubectl / SelfSubjectReview denied) — identity UNVERIFIED by this check; the matrix below still constrains it"
fi
# A real authorization round-trip, not just a review: `list jobs` is granted by the Role.
kubectl --kubeconfig "$KC" -n "$NS" get jobs -o name >/dev/null \
|| die "the minted credential cannot list Jobs in ns ${NS} — the RoleBinding is not effective; nothing stored"
pass "can list Jobs in ns ${NS} (end-to-end authentication + authorization)"
cani() { # cani <expected yes|no> <verb> <resource...>
local want="$1"; shift
local got
got="$(kubectl --kubeconfig "$KC" auth can-i "$@" 2>/dev/null || true)"
case "$got" in
yes|no) : ;;
*) die "auth can-i '$*' returned '${got:-<nothing>}' — cannot measure the identity's rights; refusing to store an unverified credential" ;;
esac
if [ "$got" = "$want" ]; then
pass "can-i $* -> ${got} (expected)"
else
die "can-i $* -> ${got}, expected ${want}. This credential is NOT the narrow trigger identity described in conformance/incluster/rbac.yaml. Nothing stored. Do NOT widen that Role to make this pass — investigate what it is actually bound to."
fi
}
# Granted by the trigger Role (create/get/list/watch jobs; get/list/watch pods + pods/log).
cani yes create jobs.batch -n "$NS"
cani yes get pods -n "$NS"
cani yes get pods/log -n "$NS"
# Explicitly NOT granted — each is a specific escape route the Role's comments call out.
cani no delete jobs.batch -n "$NS" # a trigger that can delete can hide a failed run
cani no create pods/exec -n "$NS" # no execution channel into the runner pod
cani no get secrets -n "$NS"
cani no create jobs.batch -n default # namespaced Role: no reach outside ns conformance
cani no get nodes # the RUNNER has nodes/patch; the trigger must not
cani no '*' '*' --all-namespaces # not cluster-admin, stated as a measurement
# ========================================================================================
# 5. STORE as the Gitea Actions repo secret, then VERIFY BY READING IT BACK (lesson 2).
# The body goes in a 0600 file so the token is not in argv.
# ========================================================================================
step "storing the kubeconfig as repo Actions secret ${SECRET}"
jq -n --rawfile d "$KC" '{data:$d}' >"${WORK}/body.json"
chmod 600 "${WORK}/body.json"
code="$(basic -o "${WORK}/put.out" -w '%{http_code}' -X PUT \
-H 'Content-Type: application/json' --data @"${WORK}/body.json" \
"${API}/actions/secrets/${SECRET}")"
unset TOKEN
rm -f "${WORK}/body.json" "$KC"
if [ "$code" != 201 ] && [ "$code" != 204 ]; then
# Safe to echo: the response body is Gitea's error message, not our payload. HTTP 400
# "invalid secret name" here means the name hit the reserved GITEA_ prefix (lesson 1).
die "PUT ${SECRET} failed: HTTP ${code} -- $(head -c 300 "${WORK}/put.out" 2>/dev/null)"
fi
info "PUT returned HTTP ${code}"
# curl -sS exits 0 on an HTTP error, so the status check above is necessary but NOT
# sufficient: read the object back. Gitea never returns secret VALUES, so this proves
# PRESENCE BY NAME only — say so rather than implying the content was checked.
names="$(basic "${API}/actions/secrets" | jq -rc '[.. | objects | select(has("name")) | .name] | unique')"
grep -q "$SECRET" <<<"$names" \
|| die "PUT reported HTTP ${code} but ${SECRET} is NOT in ${names} — refusing to claim success"
pass "stored and VERIFIED PRESENT by read-back: ${names}"
info "(read-back proves the secret EXISTS; Gitea never returns secret values, so its CONTENT"
info " is proven only by the first CI run that uses it — see the NOTE below.)"
info ""
info "FOR WHOEVER WRITES plan §3 item 4: this secret holds the kubeconfig CONTENT, not a path."
info "The plan's sketch writes 'kubectl --kubeconfig=\$CONFORMANCE_TRIGGER_KUBECONFIG', which"
info "would treat it as a filename. ci/conformance.sh must write the env var into a 0600 temp"
info "file (umask 077, trap-removed) and pass THAT path — an Actions secret is a string, so a"
info "kubeconfig cannot reach CI any other way."
# ========================================================================================
# 6. vars.CLUSTER_AVAILABLE — the loaded gun. Only here, only on request, only confirmed.
# ========================================================================================
if ! $ARM; then
cat <<EOF
==> ${VARIABLE} NOT TOUCHED (no --arm-cluster-available). This is the safe default.
The secret is in place; the conformance job stays gated off until the variable is set. That
is plan §5's order: land items 1-6, prove them, and only then flip step 7's switch.
EOF
else
step "arming vars.${VARIABLE}=true — every precondition above is green"
cat <<EOF
This makes .gitea/workflows/gate.yml's conformance job RUN on every PR. If the
in-cluster trigger path misbehaves, ci/conformance.sh will BLOCK (exit 1) rather than
fake a green — so the failure mode is loud PRs, not silent false confidence. Undo with:
curl -X DELETE .../repos/${OWNER}/${REPO}/actions/variables/${VARIABLE}
EOF
[ -t 0 ] || die "refusing to arm ${VARIABLE} non-interactively — this needs a human confirmation"
printf ' Type exactly "ARM %s" to proceed: ' "$VARIABLE"
read -r confirm
[ "$confirm" = "ARM ${VARIABLE}" ] || die "not confirmed (got '${confirm}') — ${VARIABLE} unchanged. The secret IS stored."
# Gitea's variables API: POST creates, PUT updates, and the body field is `value` — NOT
# `data` as in the secrets API above. Try create, fall back to update. The value "true" is
# not a secret, so response bodies are printed here to make a failure diagnosable.
vcode="$(basic -o "${WORK}/var.out" -w '%{http_code}' -X POST \
-H 'Content-Type: application/json' --data '{"value":"true"}' \
"${API}/actions/variables/${VARIABLE}")"
if [ "$vcode" != 201 ] && [ "$vcode" != 204 ]; then
info "POST returned HTTP ${vcode} ($(head -c 200 "${WORK}/var.out" 2>/dev/null)); trying PUT (variable may already exist)"
vcode="$(basic -o "${WORK}/var.out" -w '%{http_code}' -X PUT \
-H 'Content-Type: application/json' \
--data "$(jq -n --arg n "$VARIABLE" '{name:$n,value:"true"}')" \
"${API}/actions/variables/${VARIABLE}")"
fi
[ "$vcode" = 201 ] || [ "$vcode" = 204 ] \
|| die "could not set ${VARIABLE}: HTTP ${vcode} -- $(head -c 300 "${WORK}/var.out" 2>/dev/null)"
info "write returned HTTP ${vcode}"
# Unlike a secret, a variable's value IS readable — so verify the VALUE, not just presence.
# gate.yml compares `vars.CLUSTER_AVAILABLE == 'true'`, so it must be exactly "true".
gcode="$(basic -o "${WORK}/varget.json" -w '%{http_code}' "${API}/actions/variables/${VARIABLE}")"
[ "$gcode" = 200 ] || die "stored HTTP ${vcode} but GET ${VARIABLE} returned HTTP ${gcode} — refusing to claim success"
val="$(jq -r '.data // .value // empty' "${WORK}/varget.json")"
[ "$val" = "true" ] || die "${VARIABLE} reads back as '${val}', not 'true' — gate.yml compares against 'true' exactly"
pass "${VARIABLE}=${val} stored and VERIFIED by read-back"
fi
cat <<EOF
NEXT — and none of it is proven until you watch it happen:
1. Open a THROWAWAY PR. Read the conformance job's LOG, not its tick: a skipped Gitea job
reports success ("Has been skipped"), and a green tick is not evidence anything ran.
2. Confirm a Job appeared in ns ${NS}, ran the touched --requirements subset, and that its
artifacts came back. Then break an L3 test deliberately and confirm the gate BLOCKS.
3. Only after seeing BOTH directions should the conformance check be added to branch
protection (infra/gitea/require-gate.sh's status_check_contexts).
4. Diary the token expiry printed above. Re-run this script to rotate; CI fails loudly
(401 on the trigger) rather than skipping when it lapses.
ROLLBACK: DELETE the ${VARIABLE} variable (instant SKIP, gate reverts to L0/L1-only) and/or
DELETE the ${SECRET} secret. The token itself is revoked by deleting the ${NS}/${SA}
ServiceAccount, which Flux will recreate with a fresh identity.
EOF