jakman85andClaude Opus 5
19f9a8ae0c
docs/plans/l2-l3-conformance-ci-wiring.md §3 item 6, as an operator script:
infra/gitea/conformance-trigger-kubeconfig.sh. UNEXECUTED here -- minting a
cluster credential and changing Gitea variables are operator actions.
It mints a token for conformance/conformance-trigger (TokenRequest), wraps ONLY
that token with the cluster CA and server into a kubeconfig, and stores it as the
Actions secret CONFORMANCE_TRIGGER_KUBECONFIG. Never cluster-admin, never a copy
of the operator's own kubeconfig. The token is never printed and never reaches
argv: it goes into a 0600 file and into the PUT body FILE (--data @file), and the
script refuses to run under `set -x` because xtrace would print it.
Two traps this repo already paid for are designed around:
* GITEA_ is a RESERVED secret-name prefix (HTTP 400 "invalid secret name"), so
the name is CONFORMANCE_TRIGGER_KUBECONFIG -- which is also what gate.yml and
merge-gate.yaml already reference.
* `curl -sS` exits 0 on an HTTP error. Every write checks %{http_code} AND reads
the object back. The secret read-back proves presence by NAME only (Gitea
never returns secret values) and the script says so rather than implying its
content was checked; the variable read-back asserts the value is exactly
"true", which is what gate.yml compares against.
PRECONDITIONS ARE SPLIT IN TWO, and the split is the security argument:
mint-blocking (cluster) -- SA/Role/RoleBinding present, and ClusterPolicy
conformance-trigger-job-shape present + Enforce + failurePolicy Fail + Ready.
Any failure and NOTHING IS MINTED. Kubernetes places no `use` check on a Job's
serviceAccountName, so a `create jobs` token in that namespace IS the privileged
conformance-runner identity unless admission bounds its shape. RBAC cannot close
that (plan §4); minting before the boundary is live hands CI the escalation the
boundary exists to close.
arm-blocking (repo) -- digest-pinned CONFORMANCE_RUNNER_IMAGE, an initContainer
plus the admitted image in chainsaw-job.yaml, and CONFORMANCE_TRIGGER_KUBECONFIG
referenced in ci/conformance.sh CODE rather than a comment. These block only
vars.CLUSTER_AVAILABLE, because storing the secret early is harmless (the job is
gated off) and is the plan's own order.
All three arm-blocking checks FAIL in the tree today, so a live run would store
the secret and then REFUSE to arm CLUSTER_AVAILABLE, naming each missing item.
That is the intended behaviour: measured 2026-09-29, CLUSTER_AVAILABLE=true with
no trigger path scaffold-skipped all 54 tests at rc 0 and the stage printed "L2/L3
conformance passed". ci/conformance.sh now blocks that case, so arming early does
not enable L2/L3 -- it only breaks every PR. Arming needs the explicit flag, every
precondition green, and a typed confirmation on a TTY.
Before storing anything, the credential is PROVEN narrow: auth whoami must be
system:serviceaccount:conformance:conformance-trigger, a real `get jobs` round-trip
must succeed, and an auth can-i matrix asserts both halves -- granted (create/get
jobs, get pods, get pods/log) and DENIED (delete jobs, create pods/exec, get
secrets, create jobs in default, get nodes, '*' '*' --all-namespaces). A mismatch
aborts before any store, and says explicitly not to widen
conformance/incluster/rbac.yaml to make it pass.
Verified only by `bash -n` (clean) and reading. The Gitea variables API shape
(POST creates, PUT updates, body field `value` not `data`) is from documentation,
not measurement: the script tries POST then PUT, prints the response body on
failure, and treats the read-back as the authority. Kyverno readiness is accepted
from either .status.ready or a Ready condition, and "neither present" blocks
rather than assuming ready.
Gate (pinned toolbox): MERGE GATE: UNDETERMINED (rc 3) -- spec-immutability and
gitea-settings unrun, the expected local result.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
576 lines
33 KiB
Bash
Executable File
576 lines
33 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# infra/gitea/conformance-trigger-kubeconfig.sh
|
|
#
|
|
# OPERATOR script for docs/plans/l2-l3-conformance-ci-wiring.md §3 item 6 (= §5 step 7):
|
|
# provision the NARROW CI trigger credential for L2/L3 conformance, and — only when the
|
|
# rest of the trigger path actually exists — arm `vars.CLUSTER_AVAILABLE`.
|
|
#
|
|
# What it does, in order:
|
|
# 1. measures the preconditions (cluster-side AND repo-side) instead of assuming them
|
|
# 2. mints a bounded-lifetime ServiceAccount token for `conformance-trigger` in ns
|
|
# `conformance` (kubectl create token -> TokenRequest API)
|
|
# 3. assembles a kubeconfig containing ONLY that token + the cluster CA + the API server
|
|
# 4. PROVES the kubeconfig is the narrow identity (auth whoami + an allow/deny matrix)
|
|
# 5. stores it as the Gitea Actions repo secret CONFORMANCE_TRIGGER_KUBECONFIG, and
|
|
# VERIFIES the store by reading the secret list back
|
|
# 6. sets vars.CLUSTER_AVAILABLE=true ONLY under --arm-cluster-available, only with every
|
|
# precondition green, and only after a typed confirmation. Otherwise it REFUSES.
|
|
#
|
|
# NOTHING SECRET IS EVER PRINTED. The token exists in a shell variable and in a 0600 file
|
|
# under a 0700 temp dir that is removed on exit; it reaches the Gitea API in a request-body
|
|
# FILE (`--data @file`), never in argv, never on stdout. This script prints names, byte
|
|
# lengths, HTTP status codes, the token's expiry timestamp, and yes/no authorization answers.
|
|
#
|
|
# ----------------------------------------------------------------------------------------
|
|
# THREE HARD-WON LESSONS THIS SCRIPT IS BUILT AROUND
|
|
# ----------------------------------------------------------------------------------------
|
|
# 1. GITEA RESERVES THE `GITEA_` PREFIX FOR SECRET NAMES.
|
|
# `PUT .../actions/secrets/GITEA_ANYTHING` returns HTTP 400 {"message":"invalid secret
|
|
# name"}. That is why set-ci-token.sh stores SETTINGS_API_TOKEN and not
|
|
# GITEA_SETTINGS_TOKEN. `CONFORMANCE_TRIGGER_KUBECONFIG` is legal, and is the exact name
|
|
# .gitea/workflows/{gate.yml,merge-gate.yaml} already reference — do not "tidy" it.
|
|
#
|
|
# 2. `curl -sS` EXITS 0 ON AN HTTP ERROR. The first version of set-ci-token.sh sent its PUT
|
|
# to /dev/null and printed "stored" unconditionally; the secret never existed and nobody
|
|
# knew for hours. So every write here captures `-w '%{http_code}'`, checks it, AND reads
|
|
# the object back afterwards. A store is not claimed unless the read-back shows it.
|
|
#
|
|
# 3. `CLUSTER_AVAILABLE` IS A LOADED GUN — SEE THE 2026-09-29 CORRECTION IN THE PLAN.
|
|
# `conformance/run.sh` exits 0 in scaffold mode with all 54 tests `<skipped>` when it
|
|
# cannot reach a cluster. Measured on 2026-09-29: with CLUSTER_AVAILABLE=true and no
|
|
# trigger path, ci/conformance.sh printed "[ ok ] L2/L3 conformance passed" at rc 0 — a
|
|
# FAKE GREEN on an enforcing required status check. ci/conformance.sh now BLOCKS that
|
|
# case (exit 1), so today the variable does not enable L2/L3: it only changes which way
|
|
# the stage fails. Arming it before §3 items 1-4 land therefore buys nothing and breaks
|
|
# every PR. This script refuses to do it.
|
|
#
|
|
# ----------------------------------------------------------------------------------------
|
|
# WHY THE PRECONDITIONS ARE SPLIT INTO TWO CLASSES (deliberate, and load-bearing)
|
|
# ----------------------------------------------------------------------------------------
|
|
# MINT-BLOCKING (cluster-side). Kubernetes places NO `use` check on a Job's
|
|
# `serviceAccountName`, so a token with plain `create jobs` in ns `conformance` can create a
|
|
# Job running as the PRIVILEGED `conformance-runner` SA with an arbitrary command. The
|
|
# trigger Role is only the necessary half; the sufficient half is the Kyverno ClusterPolicy
|
|
# `conformance-trigger-job-shape` (platform/kyverno/admission/), which must be present AND
|
|
# Enforce AND Ready. Minting this token while that policy is absent or in Audit hands CI a
|
|
# privileged identity. So those checks block the MINT itself — nothing is created if they
|
|
# fail. (See conformance/incluster/rbac.yaml's trigger header and plan §4.)
|
|
#
|
|
# ARM-BLOCKING (repo-side). The runner image, the Job template's init-container, and
|
|
# ci/conformance.sh's in-cluster trigger flow (plan §3 items 1, 2, 4) are what make a real
|
|
# L2/L3 run possible. Storing the kubeconfig secret without them is harmless — the
|
|
# conformance job is gated on the variable and never starts — and the plan explicitly lands
|
|
# the secret before the switch. So these checks block ONLY the variable.
|
|
#
|
|
# Net effect: you can safely run this script today to provision the secret. It will then
|
|
# REFUSE to arm the variable, and tell you exactly which repo-side item is missing.
|
|
#
|
|
# ----------------------------------------------------------------------------------------
|
|
# USAGE
|
|
# ----------------------------------------------------------------------------------------
|
|
# ./infra/gitea/conformance-trigger-kubeconfig.sh --dry-run
|
|
# Read-only. Measures every precondition, mints nothing, writes nothing.
|
|
# ./infra/gitea/conformance-trigger-kubeconfig.sh [--server URL]
|
|
# Mint + store the secret. Does NOT touch CLUSTER_AVAILABLE.
|
|
# ./infra/gitea/conformance-trigger-kubeconfig.sh --arm-cluster-available
|
|
# The above, then set vars.CLUSTER_AVAILABLE=true after a typed confirmation —
|
|
# refused unless every precondition is green.
|
|
#
|
|
# Options:
|
|
# --dry-run measure only; no TokenRequest, no PUT/POST
|
|
# --arm-cluster-available additionally set vars.CLUSTER_AVAILABLE=true (guarded)
|
|
# --server URL API server URL to embed (default: the operator context's own)
|
|
# --context NAME kube context to read the CA/server from (default: k3s-prod)
|
|
# --duration D requested token lifetime (default: 720h = 30d)
|
|
# -h | --help this text
|
|
#
|
|
# ON --server: the embedded server URL must be reachable FROM THE CI RUNNER POD, which is
|
|
# not necessarily the address your workstation uses. The runner has broad egress and reaches
|
|
# the kube-API VIP (per the plan: 192.168.1.49:6443). If your operator kubeconfig points at
|
|
# a workstation-only name or a localhost tunnel, pass --server explicitly; the script warns
|
|
# on a loopback server but cannot know your topology for you.
|
|
#
|
|
# ON TOKEN LIFETIME: this is a TokenRequest token and it EXPIRES. The API server may cap the
|
|
# requested duration, so the script decodes the JWT `exp` claim and reports the ACTUAL
|
|
# expiry rather than trusting --duration. Re-run this script to rotate. After expiry the
|
|
# trigger gets 401 and the conformance stage FAILS (loudly) rather than skipping — which is
|
|
# the correct direction, but put the expiry date in your calendar.
|
|
#
|
|
# DELIBERATELY NOT DONE: SOPS-sealing the token into secrets/conformance-trigger.sops.yaml
|
|
# (plan §3 item 6's "for reproducibility/rotation"). A bearer token with a 30-day life
|
|
# cannot usefully survive the branch -> PR -> review -> merge flow (CLAUDE.md §6), and
|
|
# committing one buys re-issuance we get for free by re-running this script against the
|
|
# live SA. If the operator still wants it sealed, that is a separate reviewed change.
|
|
#
|
|
# FORBIDDEN HERE, ON PURPOSE: no cluster-admin kubeconfig, no copy of the operator's own
|
|
# kubeconfig, no widening of conformance/incluster/rbac.yaml, no mutating kubectl beyond the
|
|
# TokenRequest itself, no `kubectl apply`.
|
|
set -euo pipefail
|
|
|
|
# xtrace would print the token. Refuse rather than leak.
|
|
case "$-" in
|
|
*x*) echo "FATAL: refusing to run under 'set -x' — xtrace would print the token" >&2; exit 1 ;;
|
|
esac
|
|
umask 077
|
|
|
|
HOST=gitea.home.keithjackson.dev
|
|
OWNER=Home-Network
|
|
REPO=home-network-k3s-impl
|
|
# Legal secret name: NOT prefixed GITEA_ (lesson 1), and the name the workflows already use.
|
|
SECRET=CONFORMANCE_TRIGGER_KUBECONFIG
|
|
VARIABLE=CLUSTER_AVAILABLE
|
|
NS=conformance
|
|
SA=conformance-trigger
|
|
RUNNER_SA=conformance-runner
|
|
POLICY=conformance-trigger-job-shape
|
|
# The image prefix the admission policy admits. Kept in sync with
|
|
# platform/kyverno/admission/conformance-trigger-job-shape.yaml by the P2 check below.
|
|
RUNNER_IMAGE_PREFIX='gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:'
|
|
|
|
KCTX=k3s-prod
|
|
DURATION=720h
|
|
DRY_RUN=false
|
|
ARM=false
|
|
SERVER_OVERRIDE=""
|
|
|
|
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
|
|
|
step() { printf '\n==> %s\n' "$*"; }
|
|
info() { printf ' %s\n' "$*"; }
|
|
warn() { printf ' WARN: %s\n' "$*" >&2; }
|
|
die() { printf 'FATAL: %s\n' "$*" >&2; exit 1; }
|
|
|
|
usage() {
|
|
cat <<'USAGE'
|
|
conformance-trigger-kubeconfig.sh — provision the narrow L2/L3 conformance CI trigger credential.
|
|
|
|
--dry-run measure every precondition; mint nothing, write nothing (read-only)
|
|
--arm-cluster-available additionally set vars.CLUSTER_AVAILABLE=true (guarded + confirmed)
|
|
--server URL API server URL to embed (default: the operator context's own;
|
|
must be reachable FROM THE CI RUNNER POD — the kube-API VIP)
|
|
--context NAME kube context to read the CA/server from (default: k3s-prod)
|
|
--duration D requested token lifetime (default: 720h)
|
|
-h | --help this text
|
|
|
|
Read the header of this file before running it: the CLUSTER_AVAILABLE guard, the split between
|
|
mint-blocking and arm-blocking preconditions, and why nothing secret is ever printed.
|
|
USAGE
|
|
}
|
|
|
|
while [ $# -gt 0 ]; do
|
|
case "$1" in
|
|
--dry-run) DRY_RUN=true ;;
|
|
--arm-cluster-available) ARM=true ;;
|
|
--server) SERVER_OVERRIDE="${2:?--server needs a URL}"; shift ;;
|
|
--context) KCTX="${2:?--context needs a name}"; shift ;;
|
|
--duration) DURATION="${2:?--duration needs a value}"; shift ;;
|
|
-h|--help) usage; exit 0 ;;
|
|
*) die "unknown argument: $1 (try --help)" ;;
|
|
esac
|
|
shift
|
|
done
|
|
if $DRY_RUN && $ARM; then die "--dry-run and --arm-cluster-available are contradictory"; fi
|
|
|
|
for t in kubectl jq curl git base64; do
|
|
command -v "$t" >/dev/null || die "missing required tool: $t"
|
|
done
|
|
|
|
WORK="$(mktemp -d)"
|
|
KC="${WORK}/conformance-trigger.kubeconfig"
|
|
cleanup() { rm -rf "$WORK"; }
|
|
trap cleanup EXIT
|
|
trap 'cleanup; exit 130' INT TERM
|
|
|
|
# ========================================================================================
|
|
# 0. Gitea credentials — from the credential helper git already has for this host, exactly
|
|
# as branch-protection.sh / set-ci-token.sh / require-gate.sh do. Never in argv: curl
|
|
# reads them from a --config file descriptor.
|
|
# ========================================================================================
|
|
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
|
U="$(sed -n 's/^username=//p' <<<"$creds")"
|
|
P="$(sed -n 's/^password=//p' <<<"$creds")"
|
|
unset creds
|
|
[ -n "$U" ] && [ -n "$P" ] || die "no stored credential for ${HOST} (git credential fill returned nothing)"
|
|
basic() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
|
API="https://${HOST}/api/v1/repos/${OWNER}/${REPO}"
|
|
|
|
# ========================================================================================
|
|
# 1. PRECONDITIONS — measured, never assumed (CLAUDE.md §2b: query the thing itself).
|
|
# Two independent tallies, per the split documented in the header.
|
|
# ========================================================================================
|
|
MINT_FAIL=0
|
|
ARM_FAIL=0
|
|
pass() { printf ' [ ok ] %s\n' "$*"; }
|
|
fail_mint() { printf ' [MINT-BLOCK] %s\n' "$*" >&2; MINT_FAIL=$((MINT_FAIL + 1)); }
|
|
fail_arm() { printf ' [ARM-BLOCK] %s\n' "$*" >&2; ARM_FAIL=$((ARM_FAIL + 1)); }
|
|
|
|
step "preconditions — cluster side (these block the MINT: without them the token IS privileged)"
|
|
|
|
kubectl --context "$KCTX" get --raw /version >/dev/null 2>&1 \
|
|
|| die "cannot reach the API server via context '${KCTX}' — nothing below can be measured"
|
|
pass "API server reachable via context ${KCTX}"
|
|
|
|
# C1: the trigger identity exists (created by Flux from conformance/incluster/rbac.yaml via
|
|
# clusters/base/conformance.yaml). We print object names only, never contents.
|
|
if kubectl --context "$KCTX" -n "$NS" get serviceaccount "$SA" -o name >/dev/null 2>&1; then
|
|
pass "ServiceAccount ${NS}/${SA} exists"
|
|
else
|
|
fail_mint "ServiceAccount ${NS}/${SA} is ABSENT — is the 'conformance' Flux Kustomization reconciled?"
|
|
fi
|
|
for kind in role rolebinding; do
|
|
if kubectl --context "$KCTX" -n "$NS" get "$kind" "$SA" -o name >/dev/null 2>&1; then
|
|
pass "${kind} ${NS}/${SA} exists"
|
|
else
|
|
fail_mint "${kind} ${NS}/${SA} is ABSENT — the token would authenticate but authorize nothing"
|
|
fi
|
|
done
|
|
|
|
# C2: THE SECURITY BOUNDARY. Present, Enforce, and Ready. Anything else and we do not mint.
|
|
# `validationFailureAction: Audit` is not a boundary; an unready policy is not a boundary.
|
|
if pol="$(kubectl --context "$KCTX" get clusterpolicy "$POLICY" -o json 2>/dev/null)"; then
|
|
action="$(jq -r '.spec.validationFailureAction // .spec.rules[0].validate.failureAction // "UNSET"' <<<"$pol")"
|
|
failpol="$(jq -r '.spec.failurePolicy // "UNSET"' <<<"$pol")"
|
|
# Kyverno moved readiness from .status.ready to a Ready condition; accept either, and
|
|
# treat "neither key present" as UNMEASURABLE -> block (fail closed, do not assume ready).
|
|
ready="$(jq -r '
|
|
if (.status.ready != null) then (.status.ready | tostring)
|
|
elif (.status.conditions != null) then
|
|
((.status.conditions[] | select(.type == "Ready") | .status) // "UNKNOWN")
|
|
else "UNKNOWN" end' <<<"$pol")"
|
|
info "ClusterPolicy ${POLICY}: validationFailureAction=${action} failurePolicy=${failpol} ready=${ready}"
|
|
[ "$action" = "Enforce" ] || fail_mint "ClusterPolicy ${POLICY} is '${action}', not Enforce — the trigger would be unbounded"
|
|
[ "$failpol" = "Fail" ] || fail_mint "ClusterPolicy ${POLICY} failurePolicy='${failpol}', not Fail — a Kyverno outage would admit arbitrary Jobs"
|
|
case "$ready" in
|
|
true|True) pass "ClusterPolicy ${POLICY} reports Ready" ;;
|
|
*) fail_mint "ClusterPolicy ${POLICY} does not report Ready (got '${ready}') — refusing to assume it is loaded" ;;
|
|
esac
|
|
else
|
|
fail_mint "ClusterPolicy ${POLICY} is ABSENT — plan §4's boundary does not exist; a token with 'create jobs' here can run arbitrary code as ${RUNNER_SA}"
|
|
fi
|
|
|
|
step "preconditions — repo side (these block ONLY arming ${VARIABLE}; the secret may be stored first)"
|
|
|
|
# R1: the runner image is built and digest-pinned. Without it the Job cannot run run.sh at
|
|
# all (the stock chainsaw image has no bash/python3/kubectl/git) — plan §3 item 1.
|
|
VERSIONS="${REPO_ROOT}/conformance/versions.env"
|
|
if grep -Eq '^[[:space:]]*CONFORMANCE_RUNNER_IMAGE=.*@sha256:[0-9a-f]{64}' "$VERSIONS" 2>/dev/null; then
|
|
pass "CONFORMANCE_RUNNER_IMAGE is digest-pinned in conformance/versions.env"
|
|
else
|
|
fail_arm "conformance/versions.env has no digest-pinned CONFORMANCE_RUNNER_IMAGE (plan §3 item 1) — the Job cannot run ./run.sh"
|
|
fi
|
|
|
|
# R2: the Job template is the real thing, not the skeleton: the admitted image AND an
|
|
# init-container that populates the empty `workspace` volume with the suite at the PR SHA
|
|
# (plan §3 item 2). A template whose image the policy will not admit is a guaranteed denial.
|
|
JOBTPL="${REPO_ROOT}/conformance/incluster/chainsaw-job.yaml"
|
|
if grep -q 'initContainers' "$JOBTPL" 2>/dev/null; then
|
|
pass "chainsaw-job.yaml declares an initContainers block (workspace population)"
|
|
else
|
|
fail_arm "chainsaw-job.yaml has no initContainers — 'workspace' is an empty emptyDir and the suite is never delivered (plan §3 item 2)"
|
|
fi
|
|
if grep -q "$RUNNER_IMAGE_PREFIX" "$JOBTPL" 2>/dev/null; then
|
|
pass "chainsaw-job.yaml uses the digest-pinned image the admission policy admits"
|
|
else
|
|
fail_arm "chainsaw-job.yaml does not use ${RUNNER_IMAGE_PREFIX}… — every triggered Job would be DENIED by ${POLICY}"
|
|
fi
|
|
|
|
# R3: ci/conformance.sh actually drives the in-cluster Job. Today it mentions
|
|
# CONFORMANCE_TRIGGER_KUBECONFIG only in a TODO comment and otherwise runs run.sh LOCALLY in
|
|
# the CI job — where there is no credential at all (the runner sets
|
|
# automountServiceAccountToken: false by design), so it can only scaffold-skip. Hence the
|
|
# check must ignore comment lines: a mention in prose is not an implementation.
|
|
#
|
|
# ONE awk, NOT `grep -v … | grep -q …`: under `set -o pipefail` a successful `grep -q` exits
|
|
# early, the upstream grep takes EPIPE, and the pipeline reports non-zero — which would read
|
|
# here as "unwired" exactly when the seam IS wired. A single process cannot have that bug.
|
|
CONF_STAGE="${REPO_ROOT}/ci/conformance.sh"
|
|
if awk '!/^[[:space:]]*#/ && /CONFORMANCE_TRIGGER_KUBECONFIG/ { found = 1 } END { exit !found }' \
|
|
"$CONF_STAGE" 2>/dev/null; then
|
|
pass "ci/conformance.sh references CONFORMANCE_TRIGGER_KUBECONFIG in code (in-cluster trigger path wired)"
|
|
else
|
|
fail_arm "ci/conformance.sh mentions CONFORMANCE_TRIGGER_KUBECONFIG only in comments — the trigger path is UNIMPLEMENTED (plan §3 item 4); arming ${VARIABLE} now only changes which way the stage fails"
|
|
fi
|
|
|
|
info ""
|
|
info "precondition tally: mint-blocking failures=${MINT_FAIL} arm-blocking failures=${ARM_FAIL}"
|
|
|
|
if [ "$MINT_FAIL" -gt 0 ]; then
|
|
cat >&2 <<EOF
|
|
|
|
REFUSING TO MINT. ${MINT_FAIL} cluster-side precondition(s) failed.
|
|
|
|
Minting a token for ${NS}/${SA} while the admission boundary is absent/Audit/unready hands CI
|
|
an identity that can create a Job running as ${RUNNER_SA} (nodes/patch, pods/exec,
|
|
namespaces/patch) with an arbitrary command. That is precisely the escalation
|
|
docs/plans/l2-l3-conformance-ci-wiring.md §4 exists to close, and RBAC cannot close it.
|
|
|
|
Fix the cluster side first (reconcile platform/kyverno/admission/ and conformance/incluster/,
|
|
confirm the ClusterPolicy is Enforce + Ready), then re-run. Nothing was created.
|
|
EOF
|
|
exit 1
|
|
fi
|
|
|
|
if $ARM && [ "$ARM_FAIL" -gt 0 ]; then
|
|
cat >&2 <<EOF
|
|
|
|
REFUSING TO ARM ${VARIABLE}. ${ARM_FAIL} repo-side precondition(s) failed.
|
|
|
|
Measured 2026-09-29: with ${VARIABLE}=true and no working trigger path, conformance/run.sh
|
|
scaffold-skips all 54 tests at rc 0 and ci/conformance.sh used to print
|
|
"[ ok ] L2/L3 conformance passed" — a fake green on an enforcing required status check. The
|
|
guard now in ci/conformance.sh turns that into exit 1 instead, so arming today does not
|
|
enable L2/L3: it breaks every PR while proving nothing.
|
|
|
|
Land plan §3 items 1, 2 and 4 first. Re-run WITHOUT --arm-cluster-available to provision the
|
|
secret in the meantime — that part is safe and is step 6's intended order.
|
|
EOF
|
|
exit 1
|
|
fi
|
|
|
|
if $DRY_RUN; then
|
|
step "DRY RUN — stopping here. Nothing was minted, nothing was written."
|
|
info "secret that WOULD be written: ${SECRET} (repo ${OWNER}/${REPO})"
|
|
info "variable that WOULD be set: none — ${VARIABLE} is never touched in --dry-run"
|
|
# Read-only visibility into what is already there. Gitea never returns secret VALUES.
|
|
names="$(basic "${API}/actions/secrets" | jq -rc '[.. | objects | select(has("name")) | .name] | unique')" || true
|
|
info "existing Actions secret names: ${names:-<unreadable>}"
|
|
cur="$(basic -o "${WORK}/var.json" -w '%{http_code}' "${API}/actions/variables/${VARIABLE}")" || true
|
|
if [ "$cur" = 200 ]; then
|
|
info "existing ${VARIABLE} = $(jq -r '.data // "<no data field>"' "${WORK}/var.json")"
|
|
else
|
|
info "existing ${VARIABLE}: not set (HTTP ${cur})"
|
|
fi
|
|
exit 0
|
|
fi
|
|
|
|
# ========================================================================================
|
|
# 2. MINT the ServiceAccount token (TokenRequest). This is the only write this script makes
|
|
# against the cluster, and it creates no durable object.
|
|
# ========================================================================================
|
|
step "minting a ServiceAccount token for ${NS}/${SA} (requested duration ${DURATION})"
|
|
TOKEN="$(kubectl --context "$KCTX" -n "$NS" create token "$SA" --duration="$DURATION")" \
|
|
|| die "kubectl create token failed for ${NS}/${SA}"
|
|
[ -n "$TOKEN" ] || die "kubectl create token returned an empty token"
|
|
info "got a ${#TOKEN}-character token (value NOT shown)"
|
|
|
|
# Report the ACTUAL expiry from the JWT rather than trusting --duration: the API server caps
|
|
# TokenRequest lifetimes (--service-account-max-token-expiration) and silently shortens.
|
|
jwt_exp() {
|
|
local payload pad b64
|
|
payload="$(cut -d. -f2 <<<"$1")"
|
|
pad=$(( (4 - ${#payload} % 4) % 4 ))
|
|
b64="$(tr '_-' '/+' <<<"$payload")$(printf '%*s' "$pad" '' | tr ' ' '=')"
|
|
printf '%s' "$b64" | base64 -d 2>/dev/null | jq -r '.exp // empty' 2>/dev/null || true
|
|
}
|
|
exp="$(jwt_exp "$TOKEN")"
|
|
if [ -n "$exp" ]; then
|
|
info "token expires: $(date -u -d "@${exp}" '+%Y-%m-%dT%H:%M:%SZ' 2>/dev/null || echo "epoch ${exp}") — re-run this script to rotate"
|
|
else
|
|
warn "could not decode the token's exp claim; lifetime is UNVERIFIED (requested ${DURATION})"
|
|
fi
|
|
|
|
# ========================================================================================
|
|
# 3. ASSEMBLE the kubeconfig — SA token + cluster CA + server. Nothing else.
|
|
#
|
|
# The CA and the server URL are read from the operator's context, but the operator's
|
|
# client certificate/key are NEVER copied: only .cluster fields are extracted. `config
|
|
# view --raw` output is held in a variable and discarded, never printed.
|
|
# Built with jq (not `kubectl config set-credentials --token=…`) so the token never
|
|
# appears in argv where `ps` can see it.
|
|
# ========================================================================================
|
|
step "assembling the kubeconfig (SA token only — never cluster-admin, never the operator's)"
|
|
raw="$(kubectl --context "$KCTX" config view --raw --minify -o json)" \
|
|
|| die "could not read context '${KCTX}' from the operator kubeconfig"
|
|
SERVER="$(jq -r '.clusters[0].cluster.server // empty' <<<"$raw")"
|
|
CADATA="$(jq -r '.clusters[0].cluster["certificate-authority-data"] // empty' <<<"$raw")"
|
|
CAFILE="$(jq -r '.clusters[0].cluster["certificate-authority"] // empty' <<<"$raw")"
|
|
INSECURE="$(jq -r '.clusters[0].cluster["insecure-skip-tls-verify"] // false | tostring' <<<"$raw")"
|
|
unset raw
|
|
|
|
[ -n "$SERVER" ] || die "context '${KCTX}' has no cluster.server"
|
|
if [ -n "$SERVER_OVERRIDE" ]; then
|
|
info "overriding server ${SERVER} -> ${SERVER_OVERRIDE}"
|
|
SERVER="$SERVER_OVERRIDE"
|
|
fi
|
|
case "$SERVER" in
|
|
*localhost*|*127.0.0.1*|*://0.0.0.0*)
|
|
die "server '${SERVER}' is loopback — the CI runner pod cannot reach it. Pass --server with the address the runner uses (the kube-API VIP)." ;;
|
|
esac
|
|
if [ -z "$CADATA" ] && [ -n "$CAFILE" ]; then
|
|
[ -r "$CAFILE" ] || die "certificate-authority file '${CAFILE}' is not readable"
|
|
CADATA="$(base64 -w0 <"$CAFILE")"
|
|
fi
|
|
[ -n "$CADATA" ] || die "no cluster CA available for '${KCTX}' (insecure-skip-tls-verify=${INSECURE}); refusing to emit a kubeconfig that trusts anything"
|
|
[ "$INSECURE" = "false" ] || die "context '${KCTX}' sets insecure-skip-tls-verify; refusing to propagate that into CI"
|
|
info "server: ${SERVER}"
|
|
info "CA: ${#CADATA} base64 characters embedded (public certificate, not a secret)"
|
|
|
|
jq -n --arg srv "$SERVER" --arg ca "$CADATA" --arg tok "$TOKEN" --arg ns "$NS" --arg sa "$SA" '
|
|
{
|
|
apiVersion: "v1",
|
|
kind: "Config",
|
|
clusters: [ { name: "conformance", cluster: { server: $srv, "certificate-authority-data": $ca } } ],
|
|
users: [ { name: $sa, user: { token: $tok } } ],
|
|
contexts: [ { name: "conformance", context: { cluster: "conformance", user: $sa, namespace: $ns } } ],
|
|
"current-context": "conformance"
|
|
}' >"$KC"
|
|
chmod 600 "$KC"
|
|
info "kubeconfig written to the temp dir: $(wc -c <"$KC") bytes (content NOT shown)"
|
|
|
|
# Prove kubectl can PARSE the file we generated (JSON is valid YAML, but assert it rather
|
|
# than assume it) — this prints only the server we already printed.
|
|
parsed="$(kubectl --kubeconfig "$KC" config view -o jsonpath='{.clusters[0].cluster.server}')" \
|
|
|| die "kubectl cannot parse the generated kubeconfig"
|
|
[ "$parsed" = "$SERVER" ] || die "generated kubeconfig parses to server '${parsed}', expected '${SERVER}'"
|
|
pass "kubectl parses the generated kubeconfig"
|
|
|
|
# ========================================================================================
|
|
# 4. PROVE it is the narrow identity. Two independent proofs:
|
|
# a) who the token authenticates AS (SelfSubjectReview)
|
|
# b) an allow/deny matrix (SelfSubjectAccessReview) — both halves matter: "can create
|
|
# jobs" proves it works, "cannot get secrets / delete jobs / exec" proves it is not
|
|
# something wider that happens to include the trigger rights.
|
|
# All read-only. `auth can-i` answers from the API server's own authorizer.
|
|
# ========================================================================================
|
|
step "verifying the credential is the NARROW trigger identity"
|
|
EXPECT_USER="system:serviceaccount:${NS}:${SA}"
|
|
if who="$(kubectl --kubeconfig "$KC" auth whoami -o jsonpath='{.status.userInfo.username}' 2>/dev/null)"; then
|
|
[ "$who" = "$EXPECT_USER" ] || die "token authenticates as '${who}', expected '${EXPECT_USER}' — WRONG CREDENTIAL, nothing stored"
|
|
pass "authenticates as ${who}"
|
|
else
|
|
warn "'kubectl auth whoami' unavailable (older kubectl / SelfSubjectReview denied) — identity UNVERIFIED by this check; the matrix below still constrains it"
|
|
fi
|
|
|
|
# A real authorization round-trip, not just a review: `list jobs` is granted by the Role.
|
|
kubectl --kubeconfig "$KC" -n "$NS" get jobs -o name >/dev/null \
|
|
|| die "the minted credential cannot list Jobs in ns ${NS} — the RoleBinding is not effective; nothing stored"
|
|
pass "can list Jobs in ns ${NS} (end-to-end authentication + authorization)"
|
|
|
|
cani() { # cani <expected yes|no> <verb> <resource...>
|
|
local want="$1"; shift
|
|
local got
|
|
got="$(kubectl --kubeconfig "$KC" auth can-i "$@" 2>/dev/null || true)"
|
|
case "$got" in
|
|
yes|no) : ;;
|
|
*) die "auth can-i '$*' returned '${got:-<nothing>}' — cannot measure the identity's rights; refusing to store an unverified credential" ;;
|
|
esac
|
|
if [ "$got" = "$want" ]; then
|
|
pass "can-i $* -> ${got} (expected)"
|
|
else
|
|
die "can-i $* -> ${got}, expected ${want}. This credential is NOT the narrow trigger identity described in conformance/incluster/rbac.yaml. Nothing stored. Do NOT widen that Role to make this pass — investigate what it is actually bound to."
|
|
fi
|
|
}
|
|
# Granted by the trigger Role (create/get/list/watch jobs; get/list/watch pods + pods/log).
|
|
cani yes create jobs.batch -n "$NS"
|
|
cani yes get pods -n "$NS"
|
|
cani yes get pods/log -n "$NS"
|
|
# Explicitly NOT granted — each is a specific escape route the Role's comments call out.
|
|
cani no delete jobs.batch -n "$NS" # a trigger that can delete can hide a failed run
|
|
cani no create pods/exec -n "$NS" # no execution channel into the runner pod
|
|
cani no get secrets -n "$NS"
|
|
cani no create jobs.batch -n default # namespaced Role: no reach outside ns conformance
|
|
cani no get nodes # the RUNNER has nodes/patch; the trigger must not
|
|
cani no '*' '*' --all-namespaces # not cluster-admin, stated as a measurement
|
|
|
|
# ========================================================================================
|
|
# 5. STORE as the Gitea Actions repo secret, then VERIFY BY READING IT BACK (lesson 2).
|
|
# The body goes in a 0600 file so the token is not in argv.
|
|
# ========================================================================================
|
|
step "storing the kubeconfig as repo Actions secret ${SECRET}"
|
|
jq -n --rawfile d "$KC" '{data:$d}' >"${WORK}/body.json"
|
|
chmod 600 "${WORK}/body.json"
|
|
code="$(basic -o "${WORK}/put.out" -w '%{http_code}' -X PUT \
|
|
-H 'Content-Type: application/json' --data @"${WORK}/body.json" \
|
|
"${API}/actions/secrets/${SECRET}")"
|
|
unset TOKEN
|
|
rm -f "${WORK}/body.json" "$KC"
|
|
if [ "$code" != 201 ] && [ "$code" != 204 ]; then
|
|
# Safe to echo: the response body is Gitea's error message, not our payload. HTTP 400
|
|
# "invalid secret name" here means the name hit the reserved GITEA_ prefix (lesson 1).
|
|
die "PUT ${SECRET} failed: HTTP ${code} -- $(head -c 300 "${WORK}/put.out" 2>/dev/null)"
|
|
fi
|
|
info "PUT returned HTTP ${code}"
|
|
|
|
# curl -sS exits 0 on an HTTP error, so the status check above is necessary but NOT
|
|
# sufficient: read the object back. Gitea never returns secret VALUES, so this proves
|
|
# PRESENCE BY NAME only — say so rather than implying the content was checked.
|
|
names="$(basic "${API}/actions/secrets" | jq -rc '[.. | objects | select(has("name")) | .name] | unique')"
|
|
grep -q "$SECRET" <<<"$names" \
|
|
|| die "PUT reported HTTP ${code} but ${SECRET} is NOT in ${names} — refusing to claim success"
|
|
pass "stored and VERIFIED PRESENT by read-back: ${names}"
|
|
info "(read-back proves the secret EXISTS; Gitea never returns secret values, so its CONTENT"
|
|
info " is proven only by the first CI run that uses it — see the NOTE below.)"
|
|
info ""
|
|
info "FOR WHOEVER WRITES plan §3 item 4: this secret holds the kubeconfig CONTENT, not a path."
|
|
info "The plan's sketch writes 'kubectl --kubeconfig=\$CONFORMANCE_TRIGGER_KUBECONFIG', which"
|
|
info "would treat it as a filename. ci/conformance.sh must write the env var into a 0600 temp"
|
|
info "file (umask 077, trap-removed) and pass THAT path — an Actions secret is a string, so a"
|
|
info "kubeconfig cannot reach CI any other way."
|
|
|
|
# ========================================================================================
|
|
# 6. vars.CLUSTER_AVAILABLE — the loaded gun. Only here, only on request, only confirmed.
|
|
# ========================================================================================
|
|
if ! $ARM; then
|
|
cat <<EOF
|
|
|
|
==> ${VARIABLE} NOT TOUCHED (no --arm-cluster-available). This is the safe default.
|
|
|
|
The secret is in place; the conformance job stays gated off until the variable is set. That
|
|
is plan §5's order: land items 1-6, prove them, and only then flip step 7's switch.
|
|
EOF
|
|
else
|
|
step "arming vars.${VARIABLE}=true — every precondition above is green"
|
|
cat <<EOF
|
|
This makes .gitea/workflows/gate.yml's conformance job RUN on every PR. If the
|
|
in-cluster trigger path misbehaves, ci/conformance.sh will BLOCK (exit 1) rather than
|
|
fake a green — so the failure mode is loud PRs, not silent false confidence. Undo with:
|
|
curl -X DELETE .../repos/${OWNER}/${REPO}/actions/variables/${VARIABLE}
|
|
EOF
|
|
[ -t 0 ] || die "refusing to arm ${VARIABLE} non-interactively — this needs a human confirmation"
|
|
printf ' Type exactly "ARM %s" to proceed: ' "$VARIABLE"
|
|
read -r confirm
|
|
[ "$confirm" = "ARM ${VARIABLE}" ] || die "not confirmed (got '${confirm}') — ${VARIABLE} unchanged. The secret IS stored."
|
|
|
|
# Gitea's variables API: POST creates, PUT updates, and the body field is `value` — NOT
|
|
# `data` as in the secrets API above. Try create, fall back to update. The value "true" is
|
|
# not a secret, so response bodies are printed here to make a failure diagnosable.
|
|
vcode="$(basic -o "${WORK}/var.out" -w '%{http_code}' -X POST \
|
|
-H 'Content-Type: application/json' --data '{"value":"true"}' \
|
|
"${API}/actions/variables/${VARIABLE}")"
|
|
if [ "$vcode" != 201 ] && [ "$vcode" != 204 ]; then
|
|
info "POST returned HTTP ${vcode} ($(head -c 200 "${WORK}/var.out" 2>/dev/null)); trying PUT (variable may already exist)"
|
|
vcode="$(basic -o "${WORK}/var.out" -w '%{http_code}' -X PUT \
|
|
-H 'Content-Type: application/json' \
|
|
--data "$(jq -n --arg n "$VARIABLE" '{name:$n,value:"true"}')" \
|
|
"${API}/actions/variables/${VARIABLE}")"
|
|
fi
|
|
[ "$vcode" = 201 ] || [ "$vcode" = 204 ] \
|
|
|| die "could not set ${VARIABLE}: HTTP ${vcode} -- $(head -c 300 "${WORK}/var.out" 2>/dev/null)"
|
|
info "write returned HTTP ${vcode}"
|
|
|
|
# Unlike a secret, a variable's value IS readable — so verify the VALUE, not just presence.
|
|
# gate.yml compares `vars.CLUSTER_AVAILABLE == 'true'`, so it must be exactly "true".
|
|
gcode="$(basic -o "${WORK}/varget.json" -w '%{http_code}' "${API}/actions/variables/${VARIABLE}")"
|
|
[ "$gcode" = 200 ] || die "stored HTTP ${vcode} but GET ${VARIABLE} returned HTTP ${gcode} — refusing to claim success"
|
|
val="$(jq -r '.data // .value // empty' "${WORK}/varget.json")"
|
|
[ "$val" = "true" ] || die "${VARIABLE} reads back as '${val}', not 'true' — gate.yml compares against 'true' exactly"
|
|
pass "${VARIABLE}=${val} stored and VERIFIED by read-back"
|
|
fi
|
|
|
|
cat <<EOF
|
|
|
|
NEXT — and none of it is proven until you watch it happen:
|
|
1. Open a THROWAWAY PR. Read the conformance job's LOG, not its tick: a skipped Gitea job
|
|
reports success ("Has been skipped"), and a green tick is not evidence anything ran.
|
|
2. Confirm a Job appeared in ns ${NS}, ran the touched --requirements subset, and that its
|
|
artifacts came back. Then break an L3 test deliberately and confirm the gate BLOCKS.
|
|
3. Only after seeing BOTH directions should the conformance check be added to branch
|
|
protection (infra/gitea/require-gate.sh's status_check_contexts).
|
|
4. Diary the token expiry printed above. Re-run this script to rotate; CI fails loudly
|
|
(401 on the trigger) rather than skipping when it lapses.
|
|
|
|
ROLLBACK: DELETE the ${VARIABLE} variable (instant SKIP, gate reverts to L0/L1-only) and/or
|
|
DELETE the ${SECRET} secret. The token itself is revoked by deleting the ${NS}/${SA}
|
|
ServiceAccount, which Flux will recreate with a fresh identity.
|
|
EOF
|