Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
  jakman85andClaude Opus 5
|
11a74cce53 | ||
|
jakman85andClaude Opus 5
|
9f018aa101 | ||
|
jakman85andClaude Opus 5
|
b1c99e51bb |
No files matched your search
@@ -1,200 +0,0 @@
|
||||
# =============================================================================
|
||||
# conformance-runner — the image the in-cluster L2/L3 Chainsaw Job actually runs.
|
||||
# Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
|
||||
#
|
||||
# WHY THIS EXISTS. chainsaw-job.yaml pins `ghcr.io/kyverno/chainsaw:v0.2.12` and
|
||||
# `command: ["./run.sh"]`. That image carries the chainsaw binary and almost nothing else —
|
||||
# no bash, no python3, no kubectl, no git. So the committed Job could never have run: it
|
||||
# would have failed at exec of `./run.sh`. This image carries exactly what
|
||||
# `conformance/run.sh` needs:
|
||||
#
|
||||
# bash run.sh and every tests/**/check.sh are bash (arrays, mapfile, BASH_SOURCE)
|
||||
# python3 lib/{run_l1_offline,gen_coverage,merge_junit,emit_skipped_junit,status}.py
|
||||
# (stdlib only — conformance/versions.env PYTHON_MIN=3.9, so no pip layer)
|
||||
# chainsaw the L1/L2/L3 runner itself (run_chainsaw / lint_tests)
|
||||
# kubectl conf_cluster_reachable()'s `kubectl cluster-info` preflight + probe scripts
|
||||
# git the init-container's `clone --depth 1` + `checkout <SHA>` (§3 item 2) and
|
||||
# run.sh's `rev-parse --short HEAD` for the run id
|
||||
# tar/gzip/coreutils the `=== ARTIFACTS-TGZ-BASE64 ===` results channel (§3 item 3)
|
||||
#
|
||||
# It also carries `/usr/local/bin/conformance-clone` (0755 root:root) — the init container's
|
||||
# ONLY entrypoint, a fixed path with no argument surface. That exists to close an admission
|
||||
# hole, not for convenience: see the COPY below and the script's own header.
|
||||
#
|
||||
# VERSIONS COME FROM infra/devcontainer/versions.env — NOT from new pins invented here.
|
||||
# That file is the single source of truth for CHAINSAW_VERSION / KUBECTL_VERSION /
|
||||
# FEDORA_VERSION, and build-image.sh sources it and passes them in as --build-arg. The
|
||||
# ARG defaults below are deliberately EMPTY so a build that forgets to pass them fails
|
||||
# loudly (see the assertion in the first RUN) instead of silently baking a stale version.
|
||||
#
|
||||
# KNOWN LIMIT — READ BEFORE CHOOSING THE JOB'S `args`.
|
||||
# `run.sh` runs the selected OFFLINE L0/L1 checks (run_offline_checks) before it ever
|
||||
# touches the cluster, for ANY selection that is not offline-only. Those checks shell out
|
||||
# to flux / promtool / kyverno / kustomize, which this image deliberately does NOT carry —
|
||||
# baking the whole devcontainer toolchain into the runner would roughly double its size to
|
||||
# re-run work that ci/l0.sh + ci/l1.sh already did offline in the same CI run. So the Job
|
||||
# must select the cluster-gated layers only, e.g. `args: ["--layer", "l2"]` /
|
||||
# `["--layer", "l3"]`, where offline_selected_dirs() resolves to nothing and
|
||||
# run_offline_checks() returns 0. `--layer all` and a bare `--requirements <IDs>` would drag
|
||||
# the L0 checks in and fail on a missing tool. Flagged to the plan owner; choosing the Job's
|
||||
# args is T-6.1b's call, not this image's.
|
||||
#
|
||||
# THE ADMISSION CONTRACT THIS IMAGE MUST SATISFY.
|
||||
# platform/kyverno/admission/conformance-trigger-job-shape.yaml is a fail-closed Enforce
|
||||
# ClusterPolicy (R-SEC-02) and it is the security boundary. It requires the Job's container
|
||||
# to be `gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*` — a DIGEST,
|
||||
# never a tag — with `runAsNonRoot: true` and `allowPrivilegeEscalation: false`. Two
|
||||
# consequences are baked in here:
|
||||
#
|
||||
# 1. `USER` is NUMERIC. With `runAsNonRoot: true` and no explicit `runAsUser`, the kubelet
|
||||
# REJECTS an image whose USER is a name it cannot resolve to a uid
|
||||
# ("container has runAsNonRoot and image has non-numeric user"). So: USER 65532:65532.
|
||||
# 2. The pod is `readOnlyRootFilesystem: true` and the policy admits only the `workspace`
|
||||
# and `results` volumes — there is no writable /tmp and one cannot be added without
|
||||
# changing the boundary. So HOME and every cache path point INTO /workspace (the
|
||||
# emptyDir), not at the read-only root. That is why HOME=/workspace below.
|
||||
# =============================================================================
|
||||
|
||||
# --- builder: fetch the pinned release binaries ------------------------------
|
||||
# A full Fedora for the download stage (it has curl + tar); none of it ships in the runtime
|
||||
# image, so the runner does not carry a downloader it has no business having at runtime.
|
||||
ARG FEDORA_VERSION=44
|
||||
FROM registry.fedoraproject.org/fedora:${FEDORA_VERSION} AS builder
|
||||
|
||||
ARG CHAINSAW_VERSION=
|
||||
ARG KUBECTL_VERSION=
|
||||
ARG TARGETARCH=amd64
|
||||
|
||||
# Fail the BUILD — not the Job, three weeks later — if a pin was not passed in.
|
||||
RUN set -eux; \
|
||||
: "${CHAINSAW_VERSION:?pass --build-arg CHAINSAW_VERSION from infra/devcontainer/versions.env}"; \
|
||||
: "${KUBECTL_VERSION:?pass --build-arg KUBECTL_VERSION from infra/devcontainer/versions.env}"
|
||||
|
||||
# Same URLs and the same arch handling as infra/devcontainer/Containerfile, so the chainsaw
|
||||
# and kubectl in this image are the same builds the toolbox gate runs with.
|
||||
#
|
||||
# INTEGRITY, MEASURED NOT ASSUMED. kubectl publishes a plain `<url>.sha256`, so it is verified
|
||||
# below. chainsaw publishes NO checksums file — the v0.2.12 release assets are the tarballs plus
|
||||
# per-asset cosign `.sig`/`.pem` and an SBOM (listed via the GitHub releases API). So chainsaw's
|
||||
# integrity here rests on the tag + TLS, and afterwards on this image's own digest.
|
||||
# TODO, worth doing and deliberately not smuggled in here: verify chainsaw with
|
||||
# `cosign verify-blob --certificate chainsaw_linux_amd64.tar.gz.pem
|
||||
# --signature chainsaw_linux_amd64.tar.gz.sig ...`. That needs cosign in the builder stage and
|
||||
# the correct keyless identity/issuer for kyverno's release workflow — a supply-chain decision
|
||||
# that belongs in its own reviewed change, not as a silent addition to a runner image.
|
||||
# infra/devcontainer/Containerfile has the same gap for the same binary.
|
||||
RUN set -eux; \
|
||||
mkdir -p /out; \
|
||||
curl -fsSL "https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/${TARGETARCH}/kubectl" \
|
||||
-o /out/kubectl; \
|
||||
curl -fsSL "https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/${TARGETARCH}/kubectl.sha256" \
|
||||
-o /tmp/kubectl.sha256; \
|
||||
echo "$(cat /tmp/kubectl.sha256) /out/kubectl" | sha256sum -c -; \
|
||||
curl -fsSL "https://github.com/kyverno/chainsaw/releases/download/v${CHAINSAW_VERSION}/chainsaw_linux_${TARGETARCH}.tar.gz" \
|
||||
| tar -xz -C /out chainsaw; \
|
||||
chmod 0755 /out/kubectl /out/chainsaw; \
|
||||
/out/kubectl version --client=true --output=yaml >/dev/null; \
|
||||
/out/chainsaw version >/dev/null
|
||||
|
||||
# --- runtime -----------------------------------------------------------------
|
||||
ARG FEDORA_VERSION=44
|
||||
FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION}
|
||||
|
||||
ARG CHAINSAW_VERSION=
|
||||
ARG KUBECTL_VERSION=
|
||||
ARG FEDORA_VERSION=44
|
||||
# Nonroot uid/gid. 65532 is the distroless `nonroot` convention; the value is arbitrary but
|
||||
# it MUST be numeric and non-zero — see "THE ADMISSION CONTRACT" above.
|
||||
ARG RUNNER_UID=65532
|
||||
ARG RUNNER_GID=65532
|
||||
|
||||
LABEL org.opencontainers.image.title="conformance-runner" \
|
||||
org.opencontainers.image.description="bash + python3 + chainsaw + kubectl + git for the in-cluster L2/L3 conformance Job (conformance/run.sh)" \
|
||||
org.opencontainers.image.source="https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl" \
|
||||
org.opencontainers.image.documentation="conformance/incluster/README.md" \
|
||||
homelab.keithjackson.dev/requirements="R-TEST-02,R-SEC-02"
|
||||
|
||||
# bash: run.sh + every check.sh. python3: the stdlib report generators.
|
||||
# git-core: the init-container clone (git-core, not git — drops the perl/svn tail).
|
||||
# coreutils/findutils/grep/sed/gawk: find, sort, base64, xargs, sed -nE, awk in lib/*.sh.
|
||||
# tar+gzip: the base64-tar results channel. diffutils: `diff` in the L1 golden compare.
|
||||
# ca-certificates: TLS for the init clone and for kubectl's in-cluster API calls.
|
||||
RUN set -eux; \
|
||||
microdnf -y install --setopt=install_weak_deps=0 --nodocs \
|
||||
bash python3 git-core coreutils findutils grep sed gawk tar gzip diffutils \
|
||||
ca-certificates; \
|
||||
microdnf clean all; \
|
||||
rm -rf /var/cache/* /var/log/dnf*
|
||||
|
||||
COPY --from=builder /out/kubectl /out/chainsaw /usr/local/bin/
|
||||
|
||||
# The init container's ONLY entrypoint. Copied as a file with a fixed path and NO argument
|
||||
# surface, because the admission policy pins the main container's `command` but never reads
|
||||
# `initContainers` — so an init container with `command: ["/bin/sh","-c"]` and an arbitrary
|
||||
# script is currently ADMITTED and runs as the privileged `conformance-runner` SA (reproduced
|
||||
# by the policy's author; pass 3, fail 0). A 4th policy rule requiring
|
||||
# `command: ["/usr/local/bin/conformance-clone"]` with no `args` is what makes this file
|
||||
# REQUIRED rather than merely available; that rule belongs in its own reviewed PR against the
|
||||
# boundary, not here. See the script's header for the env-injection half of the same hole.
|
||||
#
|
||||
# 0755 root:root: the container runs as 65532 and MUST NOT be able to replace the one
|
||||
# executable the policy will trust. Ownership is asserted below rather than assumed.
|
||||
COPY conformance-clone /usr/local/bin/conformance-clone
|
||||
RUN set -eux; \
|
||||
chown root:root /usr/local/bin/conformance-clone; \
|
||||
chmod 0755 /usr/local/bin/conformance-clone; \
|
||||
bash -n /usr/local/bin/conformance-clone; \
|
||||
/usr/local/bin/conformance-clone --anything 2>&1 | grep -q 'takes no arguments'; \
|
||||
test "$(stat -c '%U:%G %a' /usr/local/bin/conformance-clone)" = 'root:root 755'
|
||||
|
||||
# The nonroot account is written directly rather than pulling in shadow-utils to run useradd
|
||||
# once. HOME is /workspace because at runtime that emptyDir is the only writable path.
|
||||
RUN set -eux; \
|
||||
echo "runner:x:${RUNNER_UID}:${RUNNER_GID}:conformance runner:/workspace:/bin/bash" >> /etc/passwd; \
|
||||
echo "runner:x:${RUNNER_GID}:" >> /etc/group; \
|
||||
install -d -m 0775 -o "${RUNNER_UID}" -g "${RUNNER_GID}" /workspace
|
||||
|
||||
# SYSTEM-level config, because HOME is an emptyDir that starts empty on every run and the
|
||||
# root filesystem is read-only — a runtime `--global` write would fail.
|
||||
# safe.directory=*: the init-container clones into the shared emptyDir and the main container
|
||||
# then runs `rev-parse` over it; any uid mismatch between the two would otherwise abort with
|
||||
# "detected dubious ownership". The repo is fetched fresh per run into an emptyDir, so there
|
||||
# is no pre-existing hostile checkout this relaxes a check against.
|
||||
RUN set -eux; \
|
||||
printf '%s\n' \
|
||||
'[safe]' \
|
||||
' directory = *' \
|
||||
'[advice]' \
|
||||
' detachedHead = false' \
|
||||
> /etc/gitconfig; \
|
||||
chmod 0644 /etc/gitconfig
|
||||
|
||||
# Provenance: what is actually in this layer, readable from a running pod.
|
||||
RUN set -eux; \
|
||||
{ echo "fedora=${FEDORA_VERSION}"; \
|
||||
echo "chainsaw=${CHAINSAW_VERSION}"; \
|
||||
echo "kubectl=${KUBECTL_VERSION}"; \
|
||||
echo "bash=$(bash --version | head -1)"; \
|
||||
echo "python3=$(python3 --version)"; \
|
||||
echo "git=$(git --version)"; \
|
||||
echo "conformance-clone=$(sha256sum /usr/local/bin/conformance-clone | cut -d' ' -f1)"; \
|
||||
} > /etc/conformance-runner-versions; \
|
||||
chmod 0444 /etc/conformance-runner-versions
|
||||
|
||||
# Every writable path points into the `workspace` emptyDir — the root filesystem is
|
||||
# read-only and the admission policy forbids adding a third volume for scratch.
|
||||
ENV HOME=/workspace \
|
||||
XDG_CACHE_HOME=/workspace/.cache \
|
||||
XDG_CONFIG_HOME=/workspace/.config \
|
||||
TMPDIR=/workspace/.tmp \
|
||||
PYTHONDONTWRITEBYTECODE=1 \
|
||||
CONF_ROOT=/workspace/conformance
|
||||
|
||||
USER ${RUNNER_UID}:${RUNNER_GID}
|
||||
WORKDIR /workspace
|
||||
|
||||
# No ENTRYPOINT on purpose. The admission policy pins `command: ["./run.sh"]` with
|
||||
# `workingDir: /workspace/conformance`; an ENTRYPOINT here would either be ignored or
|
||||
# prepend something the policy cannot see. `bash` is the interactive fallback for a human
|
||||
# debugging the image locally.
|
||||
CMD ["/bin/bash"]
|
||||
@@ -31,43 +31,16 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02).
|
||||
substitution tokens and the admission constraints it satisfies, and records the
|
||||
`initContainers` hole that was found and closed alongside it — read it before editing
|
||||
either file.
|
||||
- `Containerfile` — the **`conformance-runner`** image: `bash` + `python3` + `chainsaw` +
|
||||
`kubectl` + `git`, non-root (numeric uid 65532) and working under a read-only root
|
||||
filesystem. This is what the Job actually needs; the stock chainsaw image carries none of
|
||||
it. Versions are `--build-arg`-injected from `infra/devcontainer/versions.env` — the single
|
||||
source of truth — so the chainsaw/kubectl in the runner are the same pins the toolbox gate
|
||||
uses. Read its header before changing the Job's `args`: `run.sh` runs the selected offline
|
||||
L0/L1 checks for any non-offline-only selection, and those need flux/promtool/kyverno/
|
||||
kustomize, which this image deliberately omits. Use `--layer l2` / `--layer l3`.
|
||||
- `conformance-clone` — the init container's **only** entrypoint, baked into the image at
|
||||
`/usr/local/bin/conformance-clone` (0755 **root:root**, so uid 65532 cannot replace it).
|
||||
It is a file in the image rather than an inline script in the Job precisely so admission
|
||||
can pin it: a policy cannot tell the committed script from an attacker's when both are
|
||||
just `args` on `/bin/bash -c`. Clones the impl repo at the commit under test: the SHA must
|
||||
match `^[0-9a-f]{40}$` and `rev-parse HEAD` is **asserted** equal to it afterwards, so an
|
||||
unrendered placeholder is a hard failure rather than a silent run of the branch tip. It
|
||||
takes **no arguments**, and the repository URL and clone target are **compiled in** —
|
||||
`CONFORMANCE_REPO_URL`/`CONFORMANCE_SRC` are hard-failed on a mismatch rather than
|
||||
honoured, because a URL taken from env would let the trigger choose which repository's
|
||||
code runs as `conformance-runner`. This branch's Job depends on that interface:
|
||||
- `conformance-clone` — **arrives with PR #18**, not this one. The clone init container's
|
||||
entire logic, baked into the runner image at `/usr/local/bin/conformance-clone`. It is a
|
||||
file in the image rather than an inline script in the Job precisely so admission can pin
|
||||
it: a policy cannot tell the committed script from an attacker's when both are just `args`
|
||||
on `/bin/bash -c`. The repository URL and clone target are hardcoded **there** on purpose,
|
||||
and `CONFORMANCE_REPO_URL`/`CONFORMANCE_SRC` are hard-failed on a mismatch rather than
|
||||
honoured — a URL taken from env would let the trigger choose which repository's code runs
|
||||
as `conformance-runner`. This branch's Job depends on that interface:
|
||||
`CONFORMANCE_GIT_SHA`, `GITEA_USERNAME`/`GITEA_TOKEN`, a clone target of `/workspace/src`
|
||||
(hence `workingDir: /workspace/src/conformance`), and `TMPDIR`/`HOME` under `/workspace`.
|
||||
> **The other half landed with this branch.** When `conformance-clone` arrived, the
|
||||
> admission policy pinned the main container's `command` but **never read
|
||||
> `initContainers`** — an init container with `command: ["/bin/sh","-c"]` and an arbitrary
|
||||
> script was ADMITTED and ran as the privileged `conformance-runner` SA, so this file was
|
||||
> merely *available* rather than *required*. The rule that closes it is
|
||||
> `trigger-initcontainers-must-be-the-committed-clone-step`: at most one init container,
|
||||
> digest-pinned to the conformance-runner image, `command` exactly
|
||||
> `/usr/local/bin/conformance-clone`, and **no `args` at all**. `ephemeralContainers` and
|
||||
> `env`/`envFrom` were the same shape of hole and are closed in the same policy.
|
||||
- `build-image.sh` — builds the above, **verifies the toolchain by running it** inside the
|
||||
image as uid 65532 with `--read-only`, and prints the digest. It **never pushes**: it
|
||||
prints the exact `podman push --digestfile …` command for the operator instead. See
|
||||
`conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE` for the post-push pin.
|
||||
`--verify-clone` additionally has `conformance-clone` really clone this repo at a real SHA
|
||||
and asserts the checkout matches (opt-in, because it needs the Gitea host reachable — the
|
||||
default verify stays offline so an air-gapped build is fully checked, not half-checked).
|
||||
|
||||
## Flow
|
||||
1. CI (Gitea Actions) renders `chainsaw-job.yaml` — two `@@TOKEN@@` substitutions, the
|
||||
|
||||
@@ -1,229 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
###############################################################################
|
||||
# conformance/incluster/build-image.sh — build the pinned `conformance-runner`
|
||||
# image and print its digest. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
|
||||
#
|
||||
# ./build-image.sh # build + verify + print the digest and push command
|
||||
# ./build-image.sh --no-verify # build only (skip the in-image tool proof)
|
||||
# ./build-image.sh --verify-clone # ALSO really clone this repo with conformance-clone.
|
||||
# # Opt-in because it needs the Gitea host reachable; the
|
||||
# # default verify is offline so an air-gapped build is
|
||||
# # still fully checked rather than half-checked.
|
||||
# ENGINE=docker ./build-image.sh # build with docker instead of podman
|
||||
#
|
||||
# THIS SCRIPT NEVER PUSHES. Publishing is an operator action: the registry credential is
|
||||
# not an agent's to hold, and the pushed digest is what a fail-closed admission policy
|
||||
# then trusts. So this builds locally, proves the toolchain works INSIDE the image as the
|
||||
# non-root uid the policy demands, and prints the exact `podman push` line for a human.
|
||||
#
|
||||
# VERSIONS ARE NOT DEFINED HERE. They are read from infra/devcontainer/versions.env, the
|
||||
# single source of truth (CHAINSAW_VERSION, KUBECTL_VERSION, FEDORA_VERSION) — the same
|
||||
# pins the toolbox gate runs with. Nothing in this file invents a version.
|
||||
#
|
||||
# THE DIGEST CAVEAT — read it before pasting a digest into anything.
|
||||
# The digest printed here is the LOCAL manifest digest. It is normally identical to the
|
||||
# registry digest (podman pushes the manifest it already has rather than recompressing),
|
||||
# but that is a property of the push, not a guarantee. So the push command below uses
|
||||
# `--digestfile`: the digest the registry actually returns is the authoritative one, and
|
||||
# that is the one that goes into conformance/versions.env.
|
||||
###############################################################################
|
||||
set -euo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "${HERE}/../.." && pwd)"
|
||||
ENGINE="${ENGINE:-podman}"
|
||||
|
||||
VERIFY=1
|
||||
VERIFY_CLONE=0
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--no-verify) VERIFY=0 ;;
|
||||
--verify-clone) VERIFY_CLONE=1 ;;
|
||||
-h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
|
||||
*) echo "unknown argument: $arg" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
|
||||
info() { printf ' %s\n' "$*"; }
|
||||
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
|
||||
|
||||
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
|
||||
|
||||
# --- pins: sourced, never invented -------------------------------------------
|
||||
TOOLCHAIN_ENV="${REPO_ROOT}/infra/devcontainer/versions.env"
|
||||
[ -f "${TOOLCHAIN_ENV}" ] || die "missing ${TOOLCHAIN_ENV} — the pins live there, not here"
|
||||
# shellcheck disable=SC1090
|
||||
source "${TOOLCHAIN_ENV}"
|
||||
: "${CHAINSAW_VERSION:?CHAINSAW_VERSION not set in ${TOOLCHAIN_ENV}}"
|
||||
: "${KUBECTL_VERSION:?KUBECTL_VERSION not set in ${TOOLCHAIN_ENV}}"
|
||||
: "${FEDORA_VERSION:?FEDORA_VERSION not set in ${TOOLCHAIN_ENV}}"
|
||||
|
||||
# --- image name: fixed by the admission policy -------------------------------
|
||||
# platform/kyverno/admission/conformance-trigger-job-shape.yaml admits ONLY
|
||||
# gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*. The repository path
|
||||
# is therefore not a preference; a different path is a DENIED Job. Do not "improve" it here.
|
||||
IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
|
||||
# The tag is build convenience only (a human-readable handle for the push). Admission
|
||||
# ignores it: only the digest is ever referenced from a Job.
|
||||
IMAGE_TAG="chainsaw${CHAINSAW_VERSION}-kubectl${KUBECTL_VERSION}-fc${FEDORA_VERSION}"
|
||||
IMAGE_REF="${IMAGE_REPO}:${IMAGE_TAG}"
|
||||
|
||||
log "Building ${IMAGE_REF}"
|
||||
info "engine ${ENGINE} ($(${ENGINE} --version 2>/dev/null | head -1))"
|
||||
info "pins from ${TOOLCHAIN_ENV}"
|
||||
info " CHAINSAW ${CHAINSAW_VERSION}"
|
||||
info " KUBECTL ${KUBECTL_VERSION}"
|
||||
info " FEDORA ${FEDORA_VERSION}"
|
||||
|
||||
"${ENGINE}" build \
|
||||
--file "${HERE}/Containerfile" \
|
||||
--tag "${IMAGE_REF}" \
|
||||
--build-arg "CHAINSAW_VERSION=${CHAINSAW_VERSION}" \
|
||||
--build-arg "KUBECTL_VERSION=${KUBECTL_VERSION}" \
|
||||
--build-arg "FEDORA_VERSION=${FEDORA_VERSION}" \
|
||||
"${HERE}"
|
||||
|
||||
# --- verify: the point of the whole exercise ---------------------------------
|
||||
# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So prove the
|
||||
# tools EXECUTE, and prove they execute as the NON-ROOT uid with a READ-ONLY root
|
||||
# filesystem — the shape the admission policy forces. A pass here is the only evidence
|
||||
# this image is usable; a build that merely succeeded is not.
|
||||
if [ "${VERIFY}" -eq 1 ]; then
|
||||
log "Verifying the toolchain inside the image (non-root, read-only rootfs)"
|
||||
"${ENGINE}" run --rm \
|
||||
--user 65532:65532 \
|
||||
--read-only \
|
||||
--tmpfs /workspace:rw,mode=1777 \
|
||||
--entrypoint "" \
|
||||
"${IMAGE_REF}" \
|
||||
/bin/bash -euo pipefail -c '
|
||||
echo "--- id ---"; id
|
||||
echo "--- HOME ---"; echo "HOME=${HOME}"
|
||||
echo "--- bash ---"; bash --version | head -1
|
||||
echo "--- python3 ---"; python3 --version
|
||||
echo "--- git ---"; git --version
|
||||
echo "--- kubectl ---"; kubectl version --client
|
||||
echo "--- chainsaw ---"; chainsaw version
|
||||
echo "--- baked versions ---"; cat /etc/conformance-runner-versions
|
||||
echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---"
|
||||
stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone
|
||||
if [ -w /usr/local/bin/conformance-clone ]; then
|
||||
echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1
|
||||
fi
|
||||
echo "confirmed: not writable by uid $(id -u)"
|
||||
# Each of these MUST fail. Assert the non-zero exit as well as the message: a check
|
||||
# that only greps stderr would pass a build where the guard had stopped rejecting.
|
||||
# (`|| true` inside the capture keeps `set -e`/`pipefail` from aborting on the
|
||||
# expected failure — without it the verify dies at the first negative case.)
|
||||
expect_fail() {
|
||||
local label="$1"; shift
|
||||
local out rc=0
|
||||
out="$(env "$@" 2>&1)" || rc=$?
|
||||
echo "--- ${label} ---"
|
||||
printf "%s\n" "${out}" | head -2
|
||||
if [ "${rc}" -eq 0 ]; then
|
||||
echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1
|
||||
fi
|
||||
echo "(rejected, exit ${rc})"
|
||||
}
|
||||
expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \
|
||||
conformance-clone --layer l2
|
||||
expect_fail "conformance-clone rejects a missing SHA" \
|
||||
conformance-clone
|
||||
expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \
|
||||
CONFORMANCE_GIT_SHA=main conformance-clone
|
||||
expect_fail "conformance-clone rejects an UNRENDERED placeholder" \
|
||||
CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone
|
||||
expect_fail "conformance-clone rejects a HOME off the workspace volume" \
|
||||
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
|
||||
HOME=/etc conformance-clone
|
||||
expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \
|
||||
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
|
||||
CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone
|
||||
expect_fail "conformance-clone refuses a clone target off the workspace volume" \
|
||||
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
|
||||
CONFORMANCE_SRC=/etc/src conformance-clone
|
||||
echo "--- rootfs is read-only (expected: a failure) ---"
|
||||
if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: rootfs is writable"; exit 1; fi
|
||||
echo "confirmed: / is read-only"
|
||||
echo "--- HOME is writable (the workspace emptyDir) ---"
|
||||
touch "${HOME}/.probe" && echo "confirmed: ${HOME} is writable"
|
||||
echo "--- uid is non-zero ---"
|
||||
[ "$(id -u)" -ne 0 ] && echo "confirmed: running as uid $(id -u), not root"
|
||||
'
|
||||
log "Toolchain verified"
|
||||
fi
|
||||
|
||||
# --- verify the clone helper against the REAL repository (opt-in: needs network) ----
|
||||
# The negative cases above prove conformance-clone refuses bad input. This proves the
|
||||
# positive case: it lands the EXACT commit asked for and can prove it did. Without this, the
|
||||
# rev-parse assertion is only an unexercised line of shell.
|
||||
if [ "${VERIFY_CLONE}" -eq 1 ]; then
|
||||
log "Verifying conformance-clone against the real repository"
|
||||
SHA="$(git -C "${REPO_ROOT}" rev-parse HEAD)"
|
||||
info "requesting commit ${SHA}"
|
||||
"${ENGINE}" run --rm \
|
||||
--user 65532:65532 \
|
||||
--read-only \
|
||||
--tmpfs /workspace:rw,mode=1777 \
|
||||
--network host \
|
||||
--entrypoint "" \
|
||||
-e "CONFORMANCE_GIT_SHA=${SHA}" \
|
||||
"${IMAGE_REF}" \
|
||||
/bin/bash -euo pipefail -c '
|
||||
conformance-clone
|
||||
echo "--- assert rev-parse HEAD == the requested SHA ---"
|
||||
got="$(git -C /workspace/src rev-parse HEAD)"
|
||||
echo "requested: ${CONFORMANCE_GIT_SHA}"
|
||||
echo "HEAD: ${got}"
|
||||
[ "${got}" = "${CONFORMANCE_GIT_SHA}" ] || { echo "MISMATCH"; exit 1; }
|
||||
echo "confirmed: the working tree is at the requested commit"
|
||||
test -x /workspace/src/conformance/run.sh && echo "confirmed: run.sh present and executable"
|
||||
'
|
||||
log "conformance-clone verified against the real repository"
|
||||
fi
|
||||
|
||||
# --- digest ------------------------------------------------------------------
|
||||
DIGEST="$("${ENGINE}" image inspect --format '{{.Digest}}' "${IMAGE_REF}")"
|
||||
IMAGE_ID="$("${ENGINE}" image inspect --format '{{.Id}}' "${IMAGE_REF}")"
|
||||
SIZE="$("${ENGINE}" image inspect --format '{{.Size}}' "${IMAGE_REF}")"
|
||||
|
||||
log "Built ${IMAGE_REF}"
|
||||
info "local manifest digest ${DIGEST}"
|
||||
info "image id ${IMAGE_ID}"
|
||||
info "size $(( SIZE / 1024 / 1024 )) MiB"
|
||||
|
||||
cat <<EOF
|
||||
|
||||
================================================================================
|
||||
NOT PUSHED. Publishing is an operator action — this script holds no registry
|
||||
credential and will never acquire one.
|
||||
|
||||
OPERATOR: run these two commands, then paste the digest from the digestfile into
|
||||
conformance/versions.env (CONFORMANCE_RUNNER_IMAGE) via a PR:
|
||||
|
||||
${ENGINE} login gitea.home.keithjackson.dev
|
||||
|
||||
${ENGINE} push --digestfile /tmp/conformance-runner.digest \\
|
||||
${IMAGE_REF}
|
||||
|
||||
Then:
|
||||
|
||||
cat /tmp/conformance-runner.digest
|
||||
# -> sha256:.... THIS is the authoritative digest, not the local one above.
|
||||
|
||||
# conformance/versions.env
|
||||
CONFORMANCE_RUNNER_IMAGE="${IMAGE_REPO}@\$(cat /tmp/conformance-runner.digest)"
|
||||
|
||||
WHY THE DIGEST AND NOT THE TAG: platform/kyverno/admission/conformance-trigger-job-shape.yaml
|
||||
is a fail-closed Enforce policy. It DENIES any conformance-trigger Job whose image is not
|
||||
${IMAGE_REPO}@sha256:<digest>. A tag reference is denied,
|
||||
and so is any other repository path.
|
||||
|
||||
TIGHTEN THE BOUNDARY AFTERWARDS: that policy's header asks for the wildcard
|
||||
\`${IMAGE_REPO}@sha256:*\` to be replaced with the
|
||||
LITERAL digest once the image exists, closing the "anyone who can push to this path" gap.
|
||||
That is a separate, human-reviewed PR against the security boundary.
|
||||
================================================================================
|
||||
EOF
|
||||
@@ -1,138 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
###############################################################################
|
||||
# conformance-clone — fetch the impl repo at the exact commit under test.
|
||||
#
|
||||
# Baked into the conformance-runner image at /usr/local/bin/conformance-clone, mode 0755
|
||||
# root:root, so the running container (uid 65532) cannot replace it. The Job's init
|
||||
# container runs it as `command: ["/usr/local/bin/conformance-clone"]` with NO args.
|
||||
#
|
||||
# WHY IT IS A FILE IN THE IMAGE AND NOT AN INLINE SCRIPT IN THE JOB
|
||||
# -----------------------------------------------------------------
|
||||
# platform/kyverno/admission/conformance-trigger-job-shape.yaml pins the MAIN container's
|
||||
# `command` to ["./run.sh"] and constrains its `args` — but it never reads `initContainers`.
|
||||
# So a Job from `conformance-trigger` could carry an init container with
|
||||
# `command: ["/bin/sh","-c"]` and an arbitrary script, running as the privileged
|
||||
# `conformance-runner` SA. Reproduced by the policy's author: such a Job is ADMITTED
|
||||
# (pass 3, fail 0).
|
||||
#
|
||||
# The fix has two halves and NEITHER works alone:
|
||||
# 1. this file — a fixed path with fixed behaviour and no argument surface, so there is
|
||||
# nothing for a caller to inject into; and
|
||||
# 2. a 4th policy rule requiring every initContainer's `command` to be exactly
|
||||
# ["/usr/local/bin/conformance-clone"] with no `args`. That rule lives in the policy
|
||||
# and is NOT this branch's to write — a security boundary changes in its own reviewed PR.
|
||||
# Without (2) this file is merely available, not required.
|
||||
#
|
||||
# THE ENV SURFACE IS THE OTHER HALF OF THE SAME HOLE — READ THIS BEFORE "SIMPLIFYING"
|
||||
# -----------------------------------------------------------------------------------
|
||||
# The admission policy says NOTHING about `env`. Verified: no `env`, `valueFrom` or
|
||||
# `secretKeyRef` clause appears in it. So a script that took its repository URL from the
|
||||
# environment would just move the injection one field over — the trigger sets
|
||||
# CONFORMANCE_REPO_URL to a repo it controls, this clones it, and `./run.sh` from THAT repo
|
||||
# executes under the privileged runner SA. Identical shape to the initContainers gap, and
|
||||
# "no args" would not have stopped it.
|
||||
#
|
||||
# Therefore the repository URL and the clone target are COMPILED IN below. The matching env
|
||||
# vars are still read, so the Job template can keep declaring them and stay self-documenting,
|
||||
# but a value that DISAGREES with the baked literal is a hard failure, never a redirect.
|
||||
# The only per-run input that is genuinely trusted is the commit SHA, and it must be 40 hex
|
||||
# characters of a commit that the baked repository actually contains.
|
||||
#
|
||||
# Inputs (environment; no arguments are accepted):
|
||||
# CONFORMANCE_GIT_SHA REQUIRED. 40 lowercase hex. The commit under test.
|
||||
# CONFORMANCE_REPO_URL optional. Must equal the baked URL if set.
|
||||
# CONFORMANCE_SRC optional. Must equal the baked target if set.
|
||||
# GITEA_USERNAME optional, from Secret `conformance-git-readonly`.
|
||||
# GITEA_TOKEN optional, from Secret `conformance-git-readonly`.
|
||||
# TMPDIR / HOME optional; must be under /workspace. Default to /workspace/{tmp,home}.
|
||||
#
|
||||
# Runs under: uid 65532, readOnlyRootFilesystem, only the `workspace` and `results` volumes.
|
||||
# Everything it writes therefore lives under /workspace. There is no writable /tmp.
|
||||
###############################################################################
|
||||
set -euo pipefail
|
||||
|
||||
# --- COMPILED-IN, NOT CONFIGURABLE (see the header) --------------------------
|
||||
readonly BAKED_REPO_URL="https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git"
|
||||
readonly BAKED_SRC="/workspace/src"
|
||||
|
||||
die() { printf 'FATAL: %s\n' "$*" >&2; exit 1; }
|
||||
log() { printf '==> %s\n' "$*"; }
|
||||
|
||||
[ "$#" -eq 0 ] || die "takes no arguments (got $#: $*). Arguments are where an injection would live."
|
||||
|
||||
# --- the one genuinely trusted input ----------------------------------------
|
||||
# A bad SHA must never degrade into "ran whatever the branch tip was": an unrendered
|
||||
# @@CONFORMANCE_GIT_SHA@@ placeholder has to be a hard failure, not a silent run of main.
|
||||
: "${CONFORMANCE_GIT_SHA:?CONFORMANCE_GIT_SHA is not set; the CI trigger did not render it}"
|
||||
printf '%s' "${CONFORMANCE_GIT_SHA}" | grep -Eq '^[0-9a-f]{40}$' \
|
||||
|| die "CONFORMANCE_GIT_SHA is not a 40-hex commit id: '${CONFORMANCE_GIT_SHA}'
|
||||
The CI trigger did not substitute it, or substituted junk."
|
||||
|
||||
# --- env that may agree with the baked values, but may not override them -----
|
||||
REPO_URL="${CONFORMANCE_REPO_URL:-${BAKED_REPO_URL}}"
|
||||
[ "${REPO_URL}" = "${BAKED_REPO_URL}" ] || die "CONFORMANCE_REPO_URL does not match the image's baked repository.
|
||||
got: ${REPO_URL}
|
||||
baked: ${BAKED_REPO_URL}
|
||||
Refusing. A caller that can choose the repository can choose the code that runs."
|
||||
|
||||
SRC="${CONFORMANCE_SRC:-${BAKED_SRC}}"
|
||||
[ "${SRC}" = "${BAKED_SRC}" ] || die "CONFORMANCE_SRC does not match the image's baked clone target.
|
||||
got: ${SRC}
|
||||
baked: ${BAKED_SRC}
|
||||
Refusing."
|
||||
|
||||
# Scratch paths must stay on the workspace volume; the root filesystem is read-only.
|
||||
# `/workspace` ITSELF is allowed, not just subdirectories: the image's own ENV sets
|
||||
# HOME=/workspace, and an earlier version of this guard rejected exactly that — it matched
|
||||
# only `/workspace/*` and so refused its own default. Caught by running it, not by reading it.
|
||||
: "${TMPDIR:=/workspace/tmp}"
|
||||
: "${HOME:=/workspace/home}"
|
||||
on_workspace() { case "$1" in /workspace|/workspace/*) return 0 ;; *) return 1 ;; esac; }
|
||||
on_workspace "${TMPDIR}" || die "TMPDIR must be /workspace or under it (got '${TMPDIR}')"
|
||||
on_workspace "${HOME}" || die "HOME must be /workspace or under it (got '${HOME}')"
|
||||
export TMPDIR HOME
|
||||
mkdir -p "${TMPDIR}" "${HOME}"
|
||||
|
||||
# --- credential (optional) ---------------------------------------------------
|
||||
# git's shell credential helper is invoked as `sh -c '<value> get'`, so it reads $1 and
|
||||
# answers from the environment. The token therefore never appears in argv and never on disk.
|
||||
#
|
||||
# ANONYMOUS FALLBACK, DELIBERATE AND ANNOUNCED. Read access to this repository currently
|
||||
# needs no credential (measured: `ls-remote` succeeds unauthenticated from inside this
|
||||
# image), so requiring the Secret would fail runs that would otherwise work — and would make
|
||||
# this script untestable without one. It is not a privilege downgrade: the integrity
|
||||
# guarantee here is the SHA assertion below, not the credential. Which mode was used is
|
||||
# logged, so a run that silently lost its Secret is visible in the log rather than inferred.
|
||||
CRED=''
|
||||
if [ -n "${GITEA_USERNAME:-}" ] && [ -n "${GITEA_TOKEN:-}" ]; then
|
||||
CRED='!f() { if [ "$1" = get ]; then printf "username=%s\npassword=%s\n" "$GITEA_USERNAME" "$GITEA_TOKEN"; fi; }; f'
|
||||
log "authenticating as '${GITEA_USERNAME}' (token from the environment, never argv)"
|
||||
else
|
||||
log "no GITEA_USERNAME/GITEA_TOKEN in the environment — cloning ANONYMOUSLY"
|
||||
fi
|
||||
|
||||
git_c() {
|
||||
if [ -n "${CRED}" ]; then git -c credential.helper="${CRED}" "$@"; else git "$@"; fi
|
||||
}
|
||||
|
||||
# --- clone, land the exact commit, and PROVE it ------------------------------
|
||||
log "shallow clone ${REPO_URL} -> ${SRC}"
|
||||
git_c clone --depth 1 --no-tags --single-branch "${REPO_URL}" "${SRC}"
|
||||
|
||||
if ! git -C "${SRC}" cat-file -e "${CONFORMANCE_GIT_SHA}^{commit}" 2>/dev/null; then
|
||||
log "${CONFORMANCE_GIT_SHA} is not in the shallow tip; fetching that commit"
|
||||
git_c -C "${SRC}" fetch --depth 1 --no-tags origin "${CONFORMANCE_GIT_SHA}"
|
||||
fi
|
||||
|
||||
git -C "${SRC}" checkout --detach "${CONFORMANCE_GIT_SHA}"
|
||||
|
||||
# The assertion is the point of the whole script: everything above is best-effort plumbing,
|
||||
# this is what makes "the suite ran at the commit under test" a fact rather than a hope.
|
||||
head="$(git -C "${SRC}" rev-parse HEAD)"
|
||||
[ "${head}" = "${CONFORMANCE_GIT_SHA}" ] \
|
||||
|| die "checked out ${head}, expected ${CONFORMANCE_GIT_SHA}."
|
||||
|
||||
test -x "${SRC}/conformance/run.sh" \
|
||||
|| die "no executable conformance/run.sh at ${head}."
|
||||
|
||||
log "suite ready at ${head}"
|
||||
@@ -6,10 +6,6 @@
|
||||
|
||||
# Chainsaw — L1 rendering / L2 behavioral / L3 fault-injection runner (ADR-0011).
|
||||
CHAINSAW_VERSION="v0.2.12"
|
||||
# The UPSTREAM chainsaw image. It is NOT what the in-cluster Job runs — see
|
||||
# CONFORMANCE_RUNNER_IMAGE below. It carries the chainsaw binary and essentially nothing
|
||||
# else (no bash, no python3, no kubectl, no git), so `command: ["./run.sh"]` on this image
|
||||
# fails at exec. Kept only as the provenance of the chainsaw binary we rebuild from.
|
||||
CHAINSAW_IMAGE="ghcr.io/kyverno/chainsaw:v0.2.12"
|
||||
|
||||
# kubectl — used by run.sh for cluster reachability preflight and by probe scripts.
|
||||
@@ -22,52 +18,3 @@ KYVERNO_VERSION="v1.19.0"
|
||||
# Python 3 — coverage generator + JUnit emitter use stdlib only (no pip deps), so the
|
||||
# only constraint is a modern interpreter.
|
||||
PYTHON_MIN="3.9"
|
||||
|
||||
# --- In-cluster conformance runner image -------------------------------------
|
||||
# Built from conformance/incluster/Containerfile by conformance/incluster/build-image.sh.
|
||||
# This — not CHAINSAW_IMAGE — is what the L2/L3 Job runs (bash + python3 + chainsaw +
|
||||
# kubectl + git). Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
|
||||
#
|
||||
# It ALSO carries /usr/local/bin/conformance-clone (0755 root:root, no argument surface) as the
|
||||
# init container's only entrypoint. That is a security fixture, not a convenience: the
|
||||
# admission policy never reads `initContainers`, so an init container with an inline shell
|
||||
# script is currently admitted with the privileged runner SA. The policy rule that makes the
|
||||
# fixed path REQUIRED is not in this branch — see conformance/incluster/README.md.
|
||||
#
|
||||
# THE REPOSITORY PATH IS NOT A PREFERENCE. platform/kyverno/admission/
|
||||
# conformance-trigger-job-shape.yaml is a fail-closed Enforce ClusterPolicy (R-SEC-02) that
|
||||
# ADMITS a conformance-trigger Job only when its image matches
|
||||
# gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*
|
||||
# Any other path, and any TAG reference, is DENIED at admission. Do not "tidy" these values.
|
||||
CONFORMANCE_RUNNER_IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
|
||||
# Build handle only. Admission never sees the tag; a Job must reference the digest.
|
||||
CONFORMANCE_RUNNER_IMAGE_TAG="chainsaw0.2.12-kubectl1.36.4-fc44"
|
||||
#
|
||||
# ############################################################################
|
||||
# # DIGEST DELIBERATELY EMPTY — THE IMAGE HAS NOT BEEN PUBLISHED YET. #
|
||||
# ############################################################################
|
||||
# The image has been BUILT and its toolchain verified locally (non-root uid 65532,
|
||||
# read-only rootfs: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4, chainsaw 0.2.12,
|
||||
# and `./run.sh --layer l2 --no-cluster` completing with rc 0). It has NOT been pushed:
|
||||
# pushing needs a registry credential, and publishing the artifact a fail-closed security
|
||||
# boundary then trusts is an OPERATOR action, not an agent's.
|
||||
#
|
||||
# OPERATOR, to complete the pin:
|
||||
# 1. ./conformance/incluster/build-image.sh # rebuild, verify, print the commands
|
||||
# 2. podman login gitea.home.keithjackson.dev
|
||||
# 3. podman push --digestfile /tmp/conformance-runner.digest \
|
||||
# "${CONFORMANCE_RUNNER_IMAGE_REPO}:${CONFORMANCE_RUNNER_IMAGE_TAG}"
|
||||
# 4. Set CONFORMANCE_RUNNER_IMAGE_DIGEST below to `cat /tmp/conformance-runner.digest`
|
||||
# — the digest the REGISTRY returned, not the local manifest digest the build printed.
|
||||
# 5. Land steps 4 in a PR (this file is GitOps input, not a live edit).
|
||||
#
|
||||
# UNTIL THEN: this is empty on purpose, NOT a placeholder to be guessed at. An empty value
|
||||
# makes any consumer that renders the Job fail visibly; a fake or stale digest would instead
|
||||
# produce an ImagePullBackOff that looks like a cluster fault. Do not invent one.
|
||||
CONFORMANCE_RUNNER_IMAGE_DIGEST=""
|
||||
CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_DIGEST:+${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}}"
|
||||
#
|
||||
# AFTERWARDS, TIGHTEN THE BOUNDARY: the header of conformance-trigger-job-shape.yaml asks for
|
||||
# its `@sha256:*` wildcard to be replaced with the LITERAL digest once the image exists —
|
||||
# that closes the residual "anyone who can push to this path" gap. Separate PR, human
|
||||
# reviewed, because it edits the security boundary itself.
|
||||
@@ -333,37 +333,15 @@ visible in the policy's prose — which is careful, accurate about what it *does
|
||||
explicit that it is "THE SECURITY BOUNDARY" — only in what the CLI did with a tampered resource. A
|
||||
policy is only as trustworthy as its negative fixtures.
|
||||
|
||||
### Addendum — rebased onto `main` after #18 landed (2026-09-29)
|
||||
|
||||
This branch originally appended its handoff to `DECISIONS-LOG.md`, which is why it was
|
||||
`mergeable: false`: every branch appended to that file's same last hunk. The append is now this
|
||||
file (see `decisions/README.md`).
|
||||
|
||||
The rebase onto post-#18 `main` hit one real content conflict — `conformance/incluster/README.md`,
|
||||
where #18 and this branch had each rewritten the same bullet list. Resolved by keeping #18's
|
||||
`Containerfile` / `conformance-clone` / `build-image.sh` bullets and this branch's
|
||||
`chainsaw-job.yaml` bullet, since this branch is what makes the Job a rendered run template.
|
||||
|
||||
One of #18's paragraphs had to be **rewritten rather than kept**: its `conformance-clone` bullet
|
||||
carried a blockquote saying *"This file is half a fix … a 4th policy rule demanding `command:
|
||||
["/usr/local/bin/conformance-clone"]` with no `args` … is **not** in this branch."* That rule **is**
|
||||
in this branch — `trigger-initcontainers-must-be-the-committed-clone-step`, plus the
|
||||
`ephemeralContainers` and `env`/`envFrom` rules beside it. Leaving #18's wording would have left
|
||||
`main` documenting an open hole that had just been closed, which is the same
|
||||
documentation-disagrees-with-mechanism failure this repo keeps hitting. No code changed in the
|
||||
rebase: every file but that README is byte-identical to the pre-rebase tip.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **Nothing here has ever executed against a cluster.** The `conformance-runner` image does not
|
||||
exist (plan §3 item 1), so the Job cannot be submitted, and the init container's
|
||||
clone/fetch/assert sequence, the credential helper and the envelope-over-`kubectl-logs` path are
|
||||
all **unexercised**. The envelope itself is verified; its transport is not.
|
||||
- **PR #18 has since merged** (`cac5c9f`), so `/usr/local/bin/conformance-clone` and the image are
|
||||
now on `main` and this branch was rebased onto them. What that changes is only *availability*:
|
||||
if the path were missing or named differently the init container would fail to start, and if the
|
||||
image is not the pinned digest admission denies the Job. **Neither has been exercised** — no
|
||||
image has been built or pushed, so the digest this branch pins still resolves to nothing.
|
||||
- **PR #18 must merge first.** The image digest and `/usr/local/bin/conformance-clone` both arrive
|
||||
with it. If that path is missing or named differently, the init container fails to start; if it
|
||||
is present but the image is not the pinned digest, admission denies the Job.
|
||||
- **Secret `conformance-git-readonly` does not exist** (plan §3 item 6, operator action). Only its
|
||||
name is referenced here.
|
||||
- **`uid 65532`** is asserted by the Job's `runAsUser`/`fsGroup`; the image that must provide it has
|
||||
|
||||
@@ -1,258 +0,0 @@
|
||||
---
|
||||
date: 2026-09-29
|
||||
branch: feat/conformance-runner-image
|
||||
title: The in-cluster conformance Job could never have run; built the image it needs (PR #18)
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Closes `docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 1. PR
|
||||
[#18](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl/pulls/18). **Not
|
||||
merged** — an agent cannot, and should not.
|
||||
|
||||
### The defect, stated plainly
|
||||
|
||||
`conformance/incluster/chainsaw-job.yaml` pinned `ghcr.io/kyverno/chainsaw:v0.2.12` **and**
|
||||
`command: ["./run.sh"]`. Both lines could not be true. That image carries the chainsaw binary
|
||||
and essentially nothing else — no `bash`, no `python3`, no `kubectl`, no `git`. The Job would
|
||||
have failed at exec of `./run.sh` before a single test ran, and the only place that would have
|
||||
surfaced is a pod log nobody was reading. This is the same failure shape as the entries around
|
||||
it in the archive: a manifest that *described* a working mechanism, with nothing measuring
|
||||
whether the mechanism existed.
|
||||
|
||||
### What landed
|
||||
|
||||
- `conformance/incluster/Containerfile` — the `conformance-runner` image: bash + python3 +
|
||||
chainsaw + kubectl + `git-core` + tar/gzip/coreutils/diffutils. Two-stage (a full Fedora
|
||||
fetches the binaries and verifies kubectl's published `sha256`; the runtime layer is
|
||||
`fedora-minimal` and carries no downloader).
|
||||
- `conformance/incluster/build-image.sh` — builds, **proves the toolchain by executing it**,
|
||||
prints the digest, and **never pushes**.
|
||||
- `conformance/incluster/conformance-clone` — the baked init-container clone helper (see the
|
||||
addendum below).
|
||||
- `conformance/versions.env` — `CONFORMANCE_RUNNER_IMAGE`, digest **empty** pending the push.
|
||||
- `conformance/incluster/README.md` — documents all of it, and flags the Job manifest's stale
|
||||
`image:`.
|
||||
|
||||
`infra/devcontainer/versions.env` stays the single source of truth. `build-image.sh` sources it
|
||||
and injects `CHAINSAW_VERSION` / `KUBECTL_VERSION` / `FEDORA_VERSION` as `--build-arg`; the
|
||||
Containerfile's `ARG` defaults are **empty** and asserted in its first `RUN`, so a build that
|
||||
forgets a pin fails loudly rather than baking a stale version. No new pin was invented.
|
||||
|
||||
Two things the admission boundary forced, both of which would otherwise have been discovered
|
||||
at first run: `USER` must be **numeric** (`runAsNonRoot` with no `runAsUser` makes the kubelet
|
||||
reject a named user), and every writable path must live under `/workspace` (the pod is
|
||||
`readOnlyRootFilesystem` and the policy admits no third volume, so there is no `/tmp`). git's
|
||||
`safe.directory` is therefore written to `/etc/gitconfig` at build time.
|
||||
`platform/kyverno/admission/conformance-trigger-job-shape.yaml` was **not modified**.
|
||||
|
||||
### The chainsaw supply-chain gap, stated accurately
|
||||
|
||||
The Containerfile originally said chainsaw "publishes no per-asset checksum file at a stable
|
||||
URL". Literally true, and misleading: it reads as "no integrity option exists". The release
|
||||
assets say otherwise — v0.2.12 ships per-asset cosign `.sig` + `.pem` and an SBOM alongside
|
||||
each tarball. So verification IS available; it needs cosign in the builder stage and the right
|
||||
keyless identity for kyverno's release workflow. The comment now says what is true: kubectl is
|
||||
sha256-verified, chainsaw rests on tag + TLS, and the cosign path is named as a TODO.
|
||||
Deliberately **not** implemented here — picking a trusted signer identity is a supply-chain
|
||||
decision that belongs in its own reviewed change. `infra/devcontainer/Containerfile` has the
|
||||
same gap for the same binary.
|
||||
|
||||
### Addendum — `/usr/local/bin/conformance-clone`
|
||||
|
||||
Added at the coordinator's request, to close half of the admission gap a sibling agent found:
|
||||
the policy pins the MAIN container's `command` but **never reads `initContainers`**, so a Job
|
||||
from `conformance-trigger` can carry an init container with `command: ["/bin/sh","-c"]` and an
|
||||
arbitrary script, running as the privileged `conformance-runner` SA. Reproduced by the policy's
|
||||
author (`pass: 3, fail: 0` on a privileged `alpine:latest` init container running
|
||||
`kubectl get secrets -A`).
|
||||
|
||||
The image now bakes `/usr/local/bin/conformance-clone`, **0755 root:root**, so uid 65532 cannot
|
||||
replace the one executable the policy will come to trust. Clone logic taken from PR #20's
|
||||
`clone-suite-at-commit`, with both load-bearing properties preserved: the SHA must match
|
||||
`^[0-9a-f]{40}$`, and `rev-parse HEAD` is asserted equal to it after checkout. No arguments are
|
||||
accepted.
|
||||
|
||||
**A DEVIATION FROM THE BRIEF, DELIBERATE — the repo URL and clone target are COMPILED IN, not
|
||||
taken from the environment.** The brief said to take SHA, repo URL and target dir from the
|
||||
environment. The policy contains no `env`, `valueFrom` or `secretKeyRef` clause at all, so a
|
||||
script that trusted `CONFORMANCE_REPO_URL` would have moved the injection one field over — the
|
||||
trigger points it at a repo it controls, `conformance-clone` fetches that, and `./run.sh` from
|
||||
the attacker's tree runs under the privileged SA. "No args" would not have stopped it. The env
|
||||
vars are still read so the Job template stays self-documenting, but a value that disagrees with
|
||||
the baked literal is a hard failure, never a redirect. The only genuinely trusted per-run input
|
||||
is the SHA.
|
||||
|
||||
**This file is half a fix and must not be mistaken for the whole one.** What makes the fixed
|
||||
path *required* rather than merely *available* is a 4th policy rule demanding
|
||||
`command: ["/usr/local/bin/conformance-clone"]` with no `args` on every initContainer. That
|
||||
rule edits the security boundary and belongs in its own reviewed PR — it is **not** in this
|
||||
branch. Until it lands, the hole is open.
|
||||
|
||||
The anonymous-clone fallback is deliberate and announced in the script's log line: read access
|
||||
to this repo needs no credential today, the integrity guarantee here is the SHA assertion
|
||||
rather than the credential, and requiring the Secret would have made the script untestable
|
||||
without one.
|
||||
|
||||
### A REAL CONSTRAINT THE PLAN DOES NOT ACCOUNT FOR — read before wiring T-6.1b
|
||||
|
||||
`run.sh` runs the selected **offline L0/L1 checks** before it touches the cluster, for **any**
|
||||
selection that is not offline-only. Those checks shell out to `flux`, `promtool`, `kyverno` and
|
||||
`kustomize`. This image deliberately omits them — baking the whole devcontainer toolchain into
|
||||
the runner would roughly double its size to re-run work `ci/l0.sh` and `ci/l1.sh` already did
|
||||
offline in the same CI run.
|
||||
|
||||
So **the plan's own invocation shapes do not work on this image.** `--layer all
|
||||
--strict-coverage` (the documented phase-exit shape) and a bare `--requirements <IDs>` (the
|
||||
documented merge-gate shape) both drag the L0 checks in and will fail on a missing tool. The Job
|
||||
must use `--layer l2` / `--layer l3`, which is verified below to resolve the offline selection
|
||||
to nothing.
|
||||
|
||||
**This is a decision for the plan owner, not a detail:** either the Job selects cluster-gated
|
||||
layers only, or the runner grows the full L0 toolchain, or `run.sh` learns to skip offline
|
||||
checks when their tools are absent (which would be a fail-open and is the wrong answer). It is
|
||||
documented in the Containerfile header and the incluster README so it cannot be rediscovered
|
||||
from a pod log.
|
||||
|
||||
### Follow-ups this creates
|
||||
|
||||
1. **Operator:** `podman login` + `podman push --digestfile`, then a PR setting
|
||||
`CONFORMANCE_RUNNER_IMAGE_DIGEST` from the digestfile — the digest the *registry* returned,
|
||||
not the local manifest digest the build printed.
|
||||
2. **Then tighten the boundary**, as that policy's own header asks: replace its `@sha256:*`
|
||||
wildcard with the literal digest, closing the residual "anyone who can push to this path"
|
||||
gap. Separate, human-reviewed PR — it edits the security boundary.
|
||||
3. **T-6.1b:** decide the Job's `args` per the constraint above, and stop `chainsaw-job.yaml`
|
||||
carrying an image reference that admission would deny.
|
||||
4. **The 4th policy rule** pinning every initContainer's `command` to `conformance-clone`.
|
||||
Without it, `conformance-clone` is available but not required, and the injection hole is
|
||||
still open.
|
||||
|
||||
### Rework note — why this entry is a file and not an append
|
||||
|
||||
This branch originally carried its handoff as a ~236-line append to the tail of
|
||||
`DECISIONS-LOG.md`. That append was the **only** conflicting path between this PR and `main`,
|
||||
and it made PR #18 `mergeable: false`. The append has been dropped from the branch entirely and
|
||||
its substance moved here verbatim in meaning; the branch now touches no file outside
|
||||
`conformance/`. Nothing about what the code claims to have proven was changed by that move.
|
||||
|
||||
## What I verified
|
||||
|
||||
### The image, by running it — not by a green build
|
||||
|
||||
As uid 65532 with `--read-only`: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4
|
||||
(client), chainsaw 0.2.12; `/` confirmed read-only, `/workspace` writable, uid confirmed
|
||||
non-zero. Then the real entrypoint in the same shape against the worktree: `./run.sh --lint`,
|
||||
`--coverage-only`, and `--layer l2 --no-cluster` all **rc 0**, emitting `coverage.md`,
|
||||
`coverage.json` and `report.junit.xml`. 408 MiB. Rebuilt after the chainsaw-comment edit; the
|
||||
in-image verification still passed.
|
||||
|
||||
### The init-container clone path, end to end
|
||||
|
||||
From inside the image, as uid 65532, `python3` completes a verifying TLS handshake to
|
||||
`gitea.home.keithjackson.dev` against the stock `ca-certificates` (issuer `Let's Encrypt / YR2`,
|
||||
notAfter `Dec 12 02:16:57 2026 GMT`), and `git ls-remote https://…/home-network-k3s-impl.git
|
||||
HEAD` returns `20ea70d` with **no credential**. So no internal CA bundle is needed, and
|
||||
anonymous read is currently open — the plan's read-only deploy token buys authorisation
|
||||
hygiene, not reachability.
|
||||
|
||||
`/usr/local/bin/conformance-clone` measured in the image as uid 65532 with `--read-only`:
|
||||
`root:root 755`, confirmed **not writable** by uid 65532. Seven negative cases, each asserted to
|
||||
exit non-zero (not merely to print something): arguments rejected; missing SHA; a branch name
|
||||
instead of a SHA; an unrendered `@@CONFORMANCE_GIT_SHA@@`; `HOME` off the workspace volume; a
|
||||
**redirected** `CONFORMANCE_REPO_URL=https://evil.invalid/x.git`; a `CONFORMANCE_SRC` off the
|
||||
workspace volume. And the positive case: a real anonymous clone of this repo at a real 40-hex
|
||||
SHA, with `rev-parse HEAD` proven equal to the requested SHA.
|
||||
|
||||
Two bugs of my own that only running it could have found, both fixed: the workspace-path guard
|
||||
matched `/workspace/*` and so **rejected the image's own `HOME=/workspace`**; and the negative
|
||||
checks were written as pipelines, which under `pipefail` aborted the verify at the first
|
||||
expected failure and silently skipped the remaining six. A verify that stops early looks
|
||||
identical to a verify that passed.
|
||||
|
||||
### There is no `approval-policy` CI stage, and `conformance/**` is not a protected path
|
||||
|
||||
My brief told me to expect this PR's CI to fail on an `approval-policy` stage because
|
||||
`conformance/**` is a security-relevant path requiring human approval. **I repeated that before
|
||||
checking it. It is not true:**
|
||||
|
||||
| Claim | Measured |
|
||||
|---|---|
|
||||
| an `approval-policy` CI stage exists | **No.** `grep -rn 'approval-policy\|approval_policy' .gitea/ ci/` → nothing. `.gitea/workflows/gate.yml` has jobs `gate` and `conformance`; `merge-gate.yaml` has `spec-immutability, lint, l0, render-golden, l1, conformance, merge-gate`. No approval job in either. |
|
||||
| `conformance/**` is a protected path | **No.** `GET /branch_protections` → `protected_file_patterns: ''`. No path-based rule of any kind, and no `CODEOWNERS` file in the repo. |
|
||||
| required status checks | `status_check_contexts: ['gate / gate (pull_request)']` — one context, and it is not an approval stage. |
|
||||
|
||||
What actually blocks this PR is `required_approvals: 1` with `block_admin_merge_override: true`,
|
||||
and `gitea-admin` authors every agent PR and cannot self-approve. That is a **merge** block, not
|
||||
a CI failure — the distinction matters, because "expect a red X" and "expect a green run that an
|
||||
agent still cannot merge" send a reviewer to different places. Left in as a correction rather
|
||||
than quietly deleted.
|
||||
|
||||
### What PR #18's CI actually did, read out of the job log
|
||||
|
||||
Read from `/Home-Network/home-network-k3s-impl/actions/runs/128/jobs/0/logs`, not off the tick,
|
||||
because a skipped Gitea job reports `success`:
|
||||
|
||||
[skip] stage spec-immutability: SKIPPED (not run)
|
||||
[ ok ] stage lint: pass
|
||||
[ ok ] stage l0: pass
|
||||
[ ok ] stage render-golden: pass
|
||||
[ ok ] stage l1: pass
|
||||
[ ok ] stage gitea-settings: pass
|
||||
[skip] stage conformance: SKIPPED (not run)
|
||||
[fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED.
|
||||
|
||||
`gate / gate (pull_request)` = **success in 1m36s**, and that success means **rc 3, tolerated**
|
||||
by `./ci/gate.sh || [ "$?" -eq 3 ]` — not rc 0. `gitea-settings` ran and passed in CI with
|
||||
`REQUIRE_GITEA_SETTINGS=true`, so CI is *more* complete there than the local toolbox run, where
|
||||
that stage can only skip. That measurement predates this rework's force-push; CI has not been
|
||||
re-read since.
|
||||
|
||||
### The gate, on the reworked branch, today
|
||||
|
||||
ENGINE=<worktree shim> DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
|
||||
→ rc 3, MERGE GATE: UNDETERMINED. Nothing FAILED.
|
||||
|
||||
[ ok ] spec-immutability [ ok ] decisions-log [ ok ] lint
|
||||
[ ok ] l0 [ ok ] render-golden [ ok ] l1
|
||||
[skip] gitea-settings (SETTINGS_API_TOKEN not set — unwired repo-wide)
|
||||
[skip] conformance (CLUSTER_AVAILABLE != true) — L2/L3 UNVERIFIED
|
||||
Stages that did not run: gitea-settings
|
||||
|
||||
`decisions-log` is the stage PR #24 added; it ran against this branch and passed — "archive
|
||||
frozen, 2 entries well-formed" — so this file's shape is machine-checked, not eyeballed.
|
||||
Diff base `origin/main @ c59fe9f923c4` (this branch was rebased onto main after #24 merged).
|
||||
Touched requirements resolved to
|
||||
`R-SEC-02 R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04`; all five remain **L2/L3-UNVERIFIED**. This
|
||||
is **not** a green run and is not offered as one.
|
||||
|
||||
### The rework itself
|
||||
|
||||
`git diff origin/main...HEAD --name-only` lists only the five `conformance/` paths plus this
|
||||
file — `DECISIONS-LOG.md` is gone from the branch. `git diff pr18-orig HEAD -- conformance/`
|
||||
is empty, so the code is byte-identical to the pre-rework tip, file modes included
|
||||
(`build-image.sh` and `conformance-clone` both `100755`).
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **The image has never run in the cluster.** Everything above is local podman. Admission has
|
||||
not been exercised against it, because there is no digest to reference yet.
|
||||
- **The image is not pushed anywhere**, so `CONFORMANCE_RUNNER_IMAGE_DIGEST` is empty. Empty on
|
||||
purpose: it fails visibly in whatever renders the Job, whereas a guessed digest would look
|
||||
like an `ImagePullBackOff` cluster fault. Pushing needs a registry credential and is an
|
||||
operator action — `build-image.sh` prints the exact commands.
|
||||
- **The image was not rebuilt during this rework.** Moving where the handoff lives must not
|
||||
change what the branch claims to have proven, so no build was run; the image measurements
|
||||
above are the ones taken when the code was written, against this same code.
|
||||
- **chainsaw's cosign signatures are not verified** by this image. Only kubectl's `sha256` is.
|
||||
The cosign path is named as a TODO, not implemented.
|
||||
- **The 4th policy rule does not exist**, so `conformance-clone` is available but not required.
|
||||
The initContainer injection hole is open until that separate PR lands.
|
||||
- **L2/L3 still do not run in CI** (`vars.CLUSTER_AVAILABLE` unset). This image is a
|
||||
prerequisite for closing that hole, not the closure. R-SEC-02 and R-TEST-01..04 are untested
|
||||
at L2/L3.
|
||||
- **`gitea-settings` skipped in my local gate run**, so branch protection and runner
|
||||
registration are UNVERIFIED by this run. The CI-side claims quoted above were measured
|
||||
earlier, against the pre-rework head.
|
||||
- **The post-force-push CI run has not been read.** The `gate / gate (pull_request)` result
|
||||
quoted above is from run 128 on the old head.
|
||||
@@ -1,125 +0,0 @@
|
||||
---
|
||||
date: 2026-09-29
|
||||
branch: feat/conformance-trigger-kubeconfig
|
||||
title: The trigger credential script — it REFUSES, on purpose, and it is UNEXECUTED
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Delivered `infra/gitea/conformance-trigger-kubeconfig.sh` —
|
||||
`docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 6 — plus its section in
|
||||
`infra/gitea/README.md`. **The script is UNEXECUTED.** Minting a cluster credential and
|
||||
changing Gitea variables are operator actions; nothing was minted, no Gitea object was
|
||||
written, no `kubectl` mutation was made. Everything below is a claim about the *script*,
|
||||
not about the platform.
|
||||
|
||||
*What it does.* Mints a token for `conformance/conformance-trigger` (TokenRequest), wraps
|
||||
**only that token** with the cluster CA and server into a kubeconfig, stores it as the
|
||||
Actions secret `CONFORMANCE_TRIGGER_KUBECONFIG` (legal name — **not** the reserved `GITEA_`
|
||||
prefix), and sets `vars.CLUSTER_AVAILABLE` only as a separately-requested,
|
||||
precondition-gated, TTY-confirmed final step. The token is never printed and never reaches
|
||||
argv: it goes into a 0600 file and into the PUT **body file**. The script refuses to run
|
||||
under `set -x`.
|
||||
|
||||
*The design decision worth reviewing: TWO precondition classes.*
|
||||
|
||||
- **mint-blocking (cluster-side):** SA/Role/RoleBinding present, and ClusterPolicy
|
||||
`conformance-trigger-job-shape` present + `Enforce` + `failurePolicy: Fail` + Ready. If any
|
||||
fails, **nothing is minted**. Rationale: Kubernetes puts no `use` check on a Job's
|
||||
`serviceAccountName`, so a `create jobs` token in that namespace IS the privileged
|
||||
`conformance-runner` identity unless the admission policy bounds it. RBAC cannot fix that
|
||||
(plan §4); minting before the boundary is live would hand CI the escalation the boundary
|
||||
exists to close.
|
||||
- **arm-blocking (repo-side):** digest-pinned `CONFORMANCE_RUNNER_IMAGE`, an init-container
|
||||
plus the admitted image in `chainsaw-job.yaml`, and `CONFORMANCE_TRIGGER_KUBECONFIG`
|
||||
referenced in `ci/conformance.sh` **code rather than a comment**. These block only the
|
||||
variable, because storing the secret early is harmless and is the plan's own order.
|
||||
|
||||
**Measured in the tree at the time of writing: all three arm-blocking checks FAIL** (no
|
||||
`CONFORMANCE_RUNNER_IMAGE` in `conformance/versions.env`; `chainsaw-job.yaml` is still the
|
||||
skeleton with an empty `workspace` emptyDir and the stock chainsaw image; `ci/conformance.sh`
|
||||
names the trigger kubeconfig only in its TODO comment). So on a live run today the script
|
||||
would provision the secret and then **refuse to arm `CLUSTER_AVAILABLE`**, naming each
|
||||
missing item — which is the intended behaviour, not a defect. §3 items 1, 2 and 4 are the
|
||||
work that unblocks it.
|
||||
|
||||
*Both writes are verified the way `set-ci-token.sh` learned to.* `curl -sS` exits 0 on an
|
||||
HTTP error, so status alone proves nothing: each write checks `%{http_code}` **and** reads
|
||||
the object back. For the secret that is presence-by-name only (Gitea never returns secret
|
||||
values) and the script says so rather than implying the content was checked; for the variable
|
||||
it asserts the value reads back as exactly `true`.
|
||||
|
||||
*Noted, not fixed at the time (since fixed on this branch).* `infra/gitea/README.md`'s
|
||||
detector table still named `GITEA_SETTINGS_TOKEN` — the illegal name PR #15 replaced with
|
||||
`SETTINGS_API_TOKEN`. The second commit on this branch corrects the README's table and its
|
||||
closing "remaining step" line, and records what `ci/gitea-settings.sh` actually does
|
||||
(`TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"` — the old name survives only as a
|
||||
local-run environment variable, never as CI wiring, because the secret that would feed it
|
||||
cannot be created at all). Documentation only, no behaviour change. Still NOT fixed, because
|
||||
it is outside this PR's diff: `ci/gitea-settings.sh:45` instructs the reader to create a repo
|
||||
secret `GITEA_SETTINGS_TOKEN`, which is impossible.
|
||||
|
||||
*Rework, 2026-09-29.* This branch originally appended ~99 lines to the tail of
|
||||
`DECISIONS-LOG.md`. That is the hunk every concurrent branch collided on and the sole reason
|
||||
PR #19 was `mergeable: false`. The branch was rebuilt on today's `main` with the
|
||||
`DECISIONS-LOG.md` change dropped entirely, and this file is that handoff moved into
|
||||
`decisions/`. The changes to `infra/gitea/**` are byte-identical to the original three
|
||||
commits; the commit that contained only the log append was dropped and its substance folded
|
||||
in below.
|
||||
|
||||
## What I verified
|
||||
|
||||
*Of the script, before it stores anything:* it PROVES the credential narrow — `auth whoami`
|
||||
must return `system:serviceaccount:conformance:conformance-trigger`; a real `get jobs`
|
||||
round-trip must succeed; then an `auth can-i` matrix asserts BOTH halves — granted
|
||||
(`create/get jobs`, `get pods`, `get pods/log`) and **denied** (`delete jobs`, `create
|
||||
pods/exec`, `get secrets`, `create jobs` in `default`, `get nodes`,
|
||||
`'*' '*' --all-namespaces`). A mismatch aborts before anything is stored, and the message
|
||||
says explicitly **not** to widen `conformance/incluster/rbac.yaml` to make it pass. That is a
|
||||
property of the code as read, not of an execution.
|
||||
|
||||
*Of PR #19's CI, measured rather than assumed — and it contradicts the brief.* The brief for
|
||||
this work stated this PR's CI would FAIL `approval-policy` because `infra/gitea/**` is
|
||||
security-relevant. Measured on the PR head commit's `/statuses`:
|
||||
|
||||
gate / gate (pull_request) success Successful in 1m34s
|
||||
gate / conformance (pull_request) pending Blocked by required conditions
|
||||
|
||||
`.gitea/workflows/` on `main` contains only `gate.yml` and the dormant `merge-gate.yaml`.
|
||||
**No path-scoped approval policy exists**, so nothing machine-enforces "security-relevant
|
||||
paths need a human"; that rule rests entirely on `required_approvals: 1` plus a reviewer
|
||||
actually reading the diff. Another instance of the CLAUDE.md §2b failure mode — a control
|
||||
believed to exist because a brief said so. If the policy is intended, it is missing and needs
|
||||
its own change. And per this repo's own rule, that green `gate` tick is **not** evidence this
|
||||
script works: the gate never executes `infra/gitea/**`, and the L2/L3 stage skipped.
|
||||
|
||||
*Gate on the reworked branch (pinned toolbox, from the agent worktree, `DEV_KUBECONFIG=/dev/null`
|
||||
plus the `ENGINE` bind-mount shim per CLAUDE.md §3):* `MERGE GATE: UNDETERMINED`, **rc 3**.
|
||||
Diff base `origin/main @ c59fe9f923c4` ... this branch's head; touched requirement resolved as
|
||||
`R-ENV-01`. `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden` and `l1` all
|
||||
green; the gate decision names exactly one stage that did not run — `gitea-settings`
|
||||
(`SETTINGS_API_TOKEN` is a credential this agent does not hold, unwired repo-wide). The
|
||||
`conformance` stage also reported SKIPPED (`CLUSTER_AVAILABLE!=true`, Phase 0). The new
|
||||
`decisions-log` stage — merged to `main` as PR #24 after this rework began — validated this very
|
||||
file: "archive frozen, 2 entries well-formed", with `DECISIONS-LOG.md` unchanged and matching
|
||||
`decisions/ARCHIVE.sha256`. rc 3 is UNDETERMINED, not green, and is not reported as green here.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
**The script has never executed.** Not one line of it has run against Gitea or against the
|
||||
cluster. Only `bash -n` was run (clean); `shellcheck` is not on the host and there is no
|
||||
shellcheck stage in the gate.
|
||||
|
||||
- The Gitea variables API shape (POST-creates / PUT-updates, body field `value` not `data`)
|
||||
is from **documentation, not measurement**. The script tries POST then falls back to PUT,
|
||||
prints the response body on failure, and treats the read-back as the authority.
|
||||
- The Kyverno readiness field (`.status.ready` vs. a Ready condition) is handled both ways
|
||||
and treats "neither present" as a block rather than assuming ready — but which form this
|
||||
cluster actually reports is **unmeasured**.
|
||||
- Whether the kubeconfig's embedded server URL is reachable from the runner pod **cannot be
|
||||
known from here** — hence `--server`, and a hard refusal on a loopback address.
|
||||
- No credential was minted, no Actions secret was created, `vars.CLUSTER_AVAILABLE` was not
|
||||
set, and no `can-i` matrix was ever run against a real token. The narrowness argument above
|
||||
is read from the RBAC and the script, not observed.
|
||||
- The gate proves only its L0/L1 offline subset. L2/L3 conformance did not run, so nothing
|
||||
here is evidence about runtime behaviour.
|
||||
@@ -1,123 +0,0 @@
|
||||
---
|
||||
date: 2026-09-29
|
||||
branch: fix/crafty-oauth2-proxy-signin
|
||||
title: Crafty "won't sign in after Keycloak" — the gate was fine, Crafty's own login was the wall
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Reported as *"crafty utilizing the keycloak oauth2-proxy does not sign in to the service after
|
||||
signing into keycloak; there is no way to set the service up without credentials."* The sidecar's
|
||||
own access log settles which layer refused, and it is not the one the report names:
|
||||
|
||||
GET / -> 302 to Keycloak
|
||||
[AuthSuccess] ... groups:[platform-admins] <- oauth2-proxy did its job
|
||||
GET /oauth2/callback -> 302
|
||||
GET /panel/dashboard -> 302
|
||||
GET /login?next=/panel/dashboard -> 200 <- CRAFTY's own login page
|
||||
POST /api/v2/auth/login/ -> 401 <- CRAFTY refused
|
||||
|
||||
**The Keycloak gate was never broken.** Crafty 4 has no OIDC and cannot consume the sidecar's
|
||||
identity headers, so it presents its own local login to an already-authenticated user. That second
|
||||
factor is intended. What was not intended — and what made the service genuinely impossible to set
|
||||
up — is that **nothing documented where that password comes from**. Crafty generates it **once, on
|
||||
first boot only** (users table empty → `DatabaseBuilder.default_settings()`) into
|
||||
`/crafty/app/config/default-creds.txt`, mode `0600`, on the PVC's `config` subPath. Never in git; a
|
||||
fresh PVC generates a new one and the old is gone with the volume. `platform/crafty/README.md`
|
||||
asserted the double login was "expected, not a bug" and then left the reader with no way forward —
|
||||
the assertion was true and useless in the same sentence.
|
||||
|
||||
So this branch:
|
||||
|
||||
- Adds `runbooks/crafty-admin-credential.md` — tell the two auth layers apart from the sidecar log,
|
||||
retrieve and rotate the bootstrap credential, and reset a lost one via argon2. Registered in the
|
||||
runbooks `configMapGenerator`/`kustomization.yaml` and listed in `runbooks/README.md`, so the URL
|
||||
actually resolves rather than the file merely existing.
|
||||
- Documents the credential's provenance and the two auth layers in `platform/crafty/README.md`.
|
||||
- **Fixes a second, real defect found while proving the first.** Both oauth2-proxy instances emitted
|
||||
oauth2-proxy's **default** cookie names (`_oauth2_proxy`, `_oauth2_proxy_csrf`) scoped to
|
||||
`.home.keithjackson.dev`, while holding **different** cookie-secrets. One cookie pair, two
|
||||
proxies: whichever authenticated last clobbered the other's session; the loser logged `No valid
|
||||
authentication in request`, bounced to Keycloak, and — SSO session still live — was redirected
|
||||
straight back, clobbering it again. **That mutual eviction presents exactly as the reported
|
||||
symptom**, intermittently, and would have been blamed on Keycloak indefinitely. Fixed on the
|
||||
crafty side only (renaming the shared proxy's cookie would sign out every other gated app): a
|
||||
unique `--cookie-name=_oauth2_proxy_crafty` plus a **host-scoped** `--cookie-domain`.
|
||||
`--whitelist-domain` stays zone-wide — it governs post-login *redirect* targets, not cookie scope.
|
||||
|
||||
**Considered and rejected: seeding `default.json` from a SOPS Secret.** Crafty does ship the hook —
|
||||
`find_default_password()` reads `app/config/default.json` (`username`, `password`, min 8 chars) and
|
||||
`delete_default_json` controls its removal — so an initContainer could make a rebuilt Crafty's admin
|
||||
password deterministic and git-managed. Rejected: the image entrypoint pulls its default config
|
||||
**only when `app/config` is empty** (`if [ ! "$(ls -A --ignore=.gitkeep ./app/config)" ]; then cp -r
|
||||
./app/config_original/* ./app/config/`). Writing `default.json` there pre-boot makes the directory
|
||||
non-empty and **suppresses the entire config bootstrap** — no `config.json`, no `credits.json`.
|
||||
Doing it safely means replicating the entrypoint's copy inside the initContainer, i.e. co-owning
|
||||
image-internal bootstrap logic that a 4.11 bump can change silently, to replace a one-command
|
||||
retrieval. Not worth it. If it is ever revisited, the `ls -A` guard is the landmine to design around.
|
||||
|
||||
## What I verified
|
||||
|
||||
- **Which layer refused**, from the sidecar's own access log: `[AuthSuccess] ...
|
||||
groups:[platform-admins]` followed by Crafty's `/login` page and `POST /api/v2/auth/login/ → 401`.
|
||||
oauth2-proxy admitted the user; Crafty rejected the local credential.
|
||||
- **The existing bootstrap credential is still usable**, so no DB surgery is needed to get in:
|
||||
`users.last_update` **equals** `users.created` for `admin` (`09/05/2026, 08:10:04`) — the password
|
||||
has never been changed — and admin's `last_login` is 09/14 from `127.0.0.1`, the API login that
|
||||
created the "homelab" server. `default-creds.txt` is therefore live.
|
||||
- **The two cookie-secrets differ** — compared the two live Secrets (key presence/values compared
|
||||
in-cluster, not printed), so neither proxy can decrypt the other's cookie; and both instances were
|
||||
emitting the default cookie names on the shared `.home.keithjackson.dev` scope.
|
||||
- **The 3-strikes cooldown cannot fire behind this sidecar.** Crafty *records* failed attempts under
|
||||
`get_remote_ip()` (honours `X-Forwarded-For` — the live tracker key is the chain
|
||||
`"192.168.2.132, 10.0.0.123"`, with 1 attempt: the reported 401) but *checks* the cooldown under
|
||||
`self.request.remote_ip`, always `127.0.0.1` here. The halves key differently, so
|
||||
`get_recent_attempts()` always reads empty and `max_login_attempts: 3` never fires. Good
|
||||
(**repeated guesses cannot lock the operator out**) and bad (that throttle is not a control we
|
||||
have; the `platform-admins` gate is the only thing limiting who reaches the form). I had written
|
||||
the opposite into the README from the config value alone before measuring it — the config key
|
||||
existing is not the same claim as the code path running.
|
||||
- **`set_passwd` is unreachable in k8s**: it calls `getpass.getpass()` and needs a TTY; the container
|
||||
runs `main.py -d -i`. Recovery must go through the DB.
|
||||
- **argon2 is installed only in `/crafty/.venv/bin/python`** — the system `python3` in that container
|
||||
cannot import it, so the obvious reset one-liner fails.
|
||||
- **`dev.sh` does not forward stdin.** A `kubectl exec -i` through it hangs forever (provoked, not
|
||||
assumed). So the runbook's reset generates its password **in-pod** into a `0600` file — Crafty's own
|
||||
`default-creds.txt` idiom — rather than passing a secret through argv, stdin or shell history.
|
||||
- **Gate:** toolbox `./ci/gate.sh` — see the next section for the exit code. Green stages:
|
||||
`spec-immutability`, `lint`, `l0`, `render-golden`, `l1`.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **No browser round-trip.** The sign-in fix is **UNVERIFIED end-to-end**. The cookie change is
|
||||
reasoned from measured facts (default cookie names on both, shared domain, differing secrets) and
|
||||
renders correctly, but nobody has signed in through it. The human test is in
|
||||
`platform/crafty/README.md`.
|
||||
- **Nothing was applied.** No `kubectl apply/edit/patch`, no `flux reconcile`. The running pod is
|
||||
still the pre-fix spec (`crafty-64f7b6d78-pkpgp`, 2/2, 15d, 0 restarts — unchanged by this
|
||||
session's read-only probes). **Merging is what deploys, and it will restart the pod** —
|
||||
`strategy: Recreate` on an RWO volume, so the "homelab" Minecraft server stops and restarts. Pick
|
||||
the moment.
|
||||
- **Gate rc 3, not 0.** Green: `spec-immutability`, `lint`, `l0`, `render-golden`, `l1`. **Skipped:**
|
||||
`gitea-settings` (no `SETTINGS_API_TOKEN`) and **L2/L3 conformance** (`CLUSTER_AVAILABLE` unset) —
|
||||
touched requirements `R-DNS-01 R-NET-01 R-OBS-04 R-PLAT-02 R-STATE-01` are **not certified**.
|
||||
Nothing failed; nothing runtime was proven either. **rc 3 is UNDETERMINED, not green.**
|
||||
- **The admin password was not rotated**, and the bootstrap credential is still the standing admin
|
||||
credential. The runbook says to rotate on first sign-in; that is an operator action and is still
|
||||
open.
|
||||
- **The cookie collision's symptom was never reproduced in a browser.** This is the part of the
|
||||
change with no observed failure to point at — the sidecar trace above shows a *working* session,
|
||||
because that browser had not visited another gated app first. It is a latent-bug fix.
|
||||
|
||||
### Follow-ups
|
||||
|
||||
1. **Sign in and rotate** — `runbooks/crafty-admin-credential.md`, then confirm `last_update` moves
|
||||
off `created`. Until then the panel's admin password is a value generated on 2026-09-05 that has
|
||||
sat unchanged and undocumented.
|
||||
2. **Decide whether the inert brute-force cooldown matters.** Fixing it is upstream (Crafty keys the
|
||||
two halves differently); the alternative is to accept the `platform-admins` gate as the only
|
||||
throttle and say so.
|
||||
3. **Any further per-app reverse-proxy sidecar needs a unique `--cookie-name` too.** The collision is
|
||||
a property of the pattern, not of Crafty. Worth an L0 check asserting that every oauth2-proxy
|
||||
instance in the tree declares a distinct cookie name — the failure is silent and intermittent,
|
||||
which is exactly the shape this repo keeps getting burned by.
|
||||
+3
-69
@@ -102,19 +102,7 @@ Behaviour, proven by test rather than assumed:
|
||||
|---|---|---|
|
||||
| live matches the declaration | 0 | green |
|
||||
| any key drifted, or no runner online | 1 | **blocks the merge** |
|
||||
| no `SETTINGS_API_TOKEN`, or Gitea unreachable | 3 | skips **loudly**; never a pass, never a block |
|
||||
|
||||
> **The secret is `SETTINGS_API_TOKEN`.** An earlier draft of this table named
|
||||
> `GITEA_SETTINGS_TOKEN`, which **cannot exist as a Gitea secret at all** — Gitea reserves the
|
||||
> `GITEA_` prefix and returns HTTP 400 `invalid secret name`. That is the whole 2026-09-29
|
||||
> inert-detector story, so the wrong name surviving here is worth correcting rather than
|
||||
> reading past. Measured in `ci/gitea-settings.sh`:
|
||||
>
|
||||
> TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
|
||||
>
|
||||
> so `GITEA_SETTINGS_TOKEN` is still honoured — but **only as an environment variable for a
|
||||
> LOCAL run**. It can never be the CI wiring, because the secret that would feed it cannot be
|
||||
> created. In CI, only `SETTINGS_API_TOKEN` exists.
|
||||
| no `GITEA_SETTINGS_TOKEN`, or Gitea unreachable | 3 | skips **loudly**; never a pass, never a block |
|
||||
|
||||
The skip is deliberately *not* a block: an unwired credential blocking every PR is the
|
||||
lock-yourself-out failure this repo keeps hitting. Wire the token with `set-ci-token.sh`, confirm
|
||||
@@ -124,59 +112,6 @@ the stage goes **green on a real PR**, and only then set `REQUIRE_GITEA_SETTINGS
|
||||
**Changing the declaration is how you change policy.** If the live state is what you now want,
|
||||
edit this JSON in the same commit — deliberately, in review — rather than letting the two drift.
|
||||
|
||||
## `conformance-trigger-kubeconfig.sh` — the narrow L2/L3 trigger credential
|
||||
|
||||
Implements `docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 6 (= §5 step 7): mint a
|
||||
ServiceAccount token for `conformance/conformance-trigger`, wrap **only that token** in a
|
||||
kubeconfig, store it as the Actions secret `CONFORMANCE_TRIGGER_KUBECONFIG`, and — separately,
|
||||
and only on request — set `vars.CLUSTER_AVAILABLE=true`.
|
||||
|
||||
./infra/gitea/conformance-trigger-kubeconfig.sh --dry-run # measure only; writes nothing
|
||||
./infra/gitea/conformance-trigger-kubeconfig.sh # mint + store the secret
|
||||
./infra/gitea/conformance-trigger-kubeconfig.sh --arm-cluster-available # + the variable
|
||||
|
||||
Never a cluster-admin kubeconfig, never a copy of the operator's own. The token is never
|
||||
printed: it goes from the TokenRequest API into a 0600 file, into the request **body file**
|
||||
(`--data @file`, so not even into `ps`), and nowhere else. The script refuses to run under
|
||||
`set -x`.
|
||||
|
||||
**Its preconditions are split in two, and the split is the point.**
|
||||
|
||||
| Class | Checks | Blocks |
|
||||
|---|---|---|
|
||||
| **mint-blocking** (cluster) | SA/Role/RoleBinding exist; ClusterPolicy `conformance-trigger-job-shape` is present, `Enforce`, `failurePolicy: Fail`, Ready | the **mint** — nothing is created |
|
||||
| **arm-blocking** (repo) | digest-pinned `CONFORMANCE_RUNNER_IMAGE`; `chainsaw-job.yaml` has an init-container and the admitted image; `ci/conformance.sh` names the trigger kubeconfig in **code, not a comment** | only `CLUSTER_AVAILABLE` |
|
||||
|
||||
Why mint-blocking: Kubernetes puts **no `use` check on a Job's `serviceAccountName`**, so a
|
||||
token that can `create jobs` in `conformance` can run arbitrary code as the privileged
|
||||
`conformance-runner` SA. RBAC cannot close that; the Kyverno policy is the boundary. Minting
|
||||
while it is absent or in `Audit` hands CI the privileged identity, so the script will not do it.
|
||||
|
||||
Why arm-blocking is separate: storing the secret before the runner image / Job template / CI
|
||||
trigger flow exist is harmless (the job is gated off), and is the plan's intended order. So
|
||||
running this today provisions the secret and then **refuses to arm the variable**, naming the
|
||||
missing item.
|
||||
|
||||
**`CLUSTER_AVAILABLE` is not a "turn conformance on" switch.** Measured 2026-09-29: set with no
|
||||
trigger path, `run.sh` scaffold-skipped all 54 tests at rc 0 and `ci/conformance.sh` printed
|
||||
"L2/L3 conformance passed" — a fake green on an enforcing required check. `ci/conformance.sh`
|
||||
now blocks that case, so arming early does not enable L2/L3; it only breaks every PR. Hence:
|
||||
`--arm-cluster-available` plus all preconditions green plus a typed confirmation on a TTY, or
|
||||
nothing happens.
|
||||
|
||||
Both writes are verified the way `set-ci-token.sh` learned to: check `%{http_code}` **and**
|
||||
read the object back. For the secret that proves presence by name only (Gitea never returns
|
||||
secret values); for the variable it proves the value is exactly `true`, which is what
|
||||
`gate.yml` compares against.
|
||||
|
||||
The token **expires** — it is a TokenRequest token, and the script decodes the JWT `exp` claim
|
||||
to report the real expiry rather than trusting `--duration` (the API server caps it). Re-run to
|
||||
rotate. After expiry the trigger gets 401 and the stage fails loudly rather than skipping.
|
||||
|
||||
`--server` exists because the embedded API-server URL must be reachable **from the CI runner
|
||||
pod** — the VIP — which is not necessarily the address your workstation uses. The script
|
||||
refuses a loopback server outright.
|
||||
|
||||
### Why these scripts exist at all, and what is still missing
|
||||
|
||||
Gitea's repo settings live in **its own database**, not in anything Flux reconciles or the
|
||||
@@ -189,9 +124,8 @@ These scripts encode the desired state and can re-assert it; `ci/gitea-settings.
|
||||
**detector** that turns divergence from a silent, invisible failure into a loud one. It closes the
|
||||
root cause of the 2026-09-29 session.
|
||||
|
||||
It is only as good as its wiring: **until `SETTINGS_API_TOKEN` exists it skips**, and a skipped
|
||||
detector detects nothing. (That secret was stored and verified in PR #15; this sentence predates
|
||||
it. The name matters — see the note under the behaviour table above.)
|
||||
It is only as good as its wiring: **until `GITEA_SETTINGS_TOKEN` exists it skips**, and a skipped
|
||||
detector detects nothing. That is the remaining step.
|
||||
|
||||
## Historical sequence (kept for context)
|
||||
|
||||
|
||||
@@ -1,575 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# infra/gitea/conformance-trigger-kubeconfig.sh
|
||||
#
|
||||
# OPERATOR script for docs/plans/l2-l3-conformance-ci-wiring.md §3 item 6 (= §5 step 7):
|
||||
# provision the NARROW CI trigger credential for L2/L3 conformance, and — only when the
|
||||
# rest of the trigger path actually exists — arm `vars.CLUSTER_AVAILABLE`.
|
||||
#
|
||||
# What it does, in order:
|
||||
# 1. measures the preconditions (cluster-side AND repo-side) instead of assuming them
|
||||
# 2. mints a bounded-lifetime ServiceAccount token for `conformance-trigger` in ns
|
||||
# `conformance` (kubectl create token -> TokenRequest API)
|
||||
# 3. assembles a kubeconfig containing ONLY that token + the cluster CA + the API server
|
||||
# 4. PROVES the kubeconfig is the narrow identity (auth whoami + an allow/deny matrix)
|
||||
# 5. stores it as the Gitea Actions repo secret CONFORMANCE_TRIGGER_KUBECONFIG, and
|
||||
# VERIFIES the store by reading the secret list back
|
||||
# 6. sets vars.CLUSTER_AVAILABLE=true ONLY under --arm-cluster-available, only with every
|
||||
# precondition green, and only after a typed confirmation. Otherwise it REFUSES.
|
||||
#
|
||||
# NOTHING SECRET IS EVER PRINTED. The token exists in a shell variable and in a 0600 file
|
||||
# under a 0700 temp dir that is removed on exit; it reaches the Gitea API in a request-body
|
||||
# FILE (`--data @file`), never in argv, never on stdout. This script prints names, byte
|
||||
# lengths, HTTP status codes, the token's expiry timestamp, and yes/no authorization answers.
|
||||
#
|
||||
# ----------------------------------------------------------------------------------------
|
||||
# THREE HARD-WON LESSONS THIS SCRIPT IS BUILT AROUND
|
||||
# ----------------------------------------------------------------------------------------
|
||||
# 1. GITEA RESERVES THE `GITEA_` PREFIX FOR SECRET NAMES.
|
||||
# `PUT .../actions/secrets/GITEA_ANYTHING` returns HTTP 400 {"message":"invalid secret
|
||||
# name"}. That is why set-ci-token.sh stores SETTINGS_API_TOKEN and not
|
||||
# GITEA_SETTINGS_TOKEN. `CONFORMANCE_TRIGGER_KUBECONFIG` is legal, and is the exact name
|
||||
# .gitea/workflows/{gate.yml,merge-gate.yaml} already reference — do not "tidy" it.
|
||||
#
|
||||
# 2. `curl -sS` EXITS 0 ON AN HTTP ERROR. The first version of set-ci-token.sh sent its PUT
|
||||
# to /dev/null and printed "stored" unconditionally; the secret never existed and nobody
|
||||
# knew for hours. So every write here captures `-w '%{http_code}'`, checks it, AND reads
|
||||
# the object back afterwards. A store is not claimed unless the read-back shows it.
|
||||
#
|
||||
# 3. `CLUSTER_AVAILABLE` IS A LOADED GUN — SEE THE 2026-09-29 CORRECTION IN THE PLAN.
|
||||
# `conformance/run.sh` exits 0 in scaffold mode with all 54 tests `<skipped>` when it
|
||||
# cannot reach a cluster. Measured on 2026-09-29: with CLUSTER_AVAILABLE=true and no
|
||||
# trigger path, ci/conformance.sh printed "[ ok ] L2/L3 conformance passed" at rc 0 — a
|
||||
# FAKE GREEN on an enforcing required status check. ci/conformance.sh now BLOCKS that
|
||||
# case (exit 1), so today the variable does not enable L2/L3: it only changes which way
|
||||
# the stage fails. Arming it before §3 items 1-4 land therefore buys nothing and breaks
|
||||
# every PR. This script refuses to do it.
|
||||
#
|
||||
# ----------------------------------------------------------------------------------------
|
||||
# WHY THE PRECONDITIONS ARE SPLIT INTO TWO CLASSES (deliberate, and load-bearing)
|
||||
# ----------------------------------------------------------------------------------------
|
||||
# MINT-BLOCKING (cluster-side). Kubernetes places NO `use` check on a Job's
|
||||
# `serviceAccountName`, so a token with plain `create jobs` in ns `conformance` can create a
|
||||
# Job running as the PRIVILEGED `conformance-runner` SA with an arbitrary command. The
|
||||
# trigger Role is only the necessary half; the sufficient half is the Kyverno ClusterPolicy
|
||||
# `conformance-trigger-job-shape` (platform/kyverno/admission/), which must be present AND
|
||||
# Enforce AND Ready. Minting this token while that policy is absent or in Audit hands CI a
|
||||
# privileged identity. So those checks block the MINT itself — nothing is created if they
|
||||
# fail. (See conformance/incluster/rbac.yaml's trigger header and plan §4.)
|
||||
#
|
||||
# ARM-BLOCKING (repo-side). The runner image, the Job template's init-container, and
|
||||
# ci/conformance.sh's in-cluster trigger flow (plan §3 items 1, 2, 4) are what make a real
|
||||
# L2/L3 run possible. Storing the kubeconfig secret without them is harmless — the
|
||||
# conformance job is gated on the variable and never starts — and the plan explicitly lands
|
||||
# the secret before the switch. So these checks block ONLY the variable.
|
||||
#
|
||||
# Net effect: you can safely run this script today to provision the secret. It will then
|
||||
# REFUSE to arm the variable, and tell you exactly which repo-side item is missing.
|
||||
#
|
||||
# ----------------------------------------------------------------------------------------
|
||||
# USAGE
|
||||
# ----------------------------------------------------------------------------------------
|
||||
# ./infra/gitea/conformance-trigger-kubeconfig.sh --dry-run
|
||||
# Read-only. Measures every precondition, mints nothing, writes nothing.
|
||||
# ./infra/gitea/conformance-trigger-kubeconfig.sh [--server URL]
|
||||
# Mint + store the secret. Does NOT touch CLUSTER_AVAILABLE.
|
||||
# ./infra/gitea/conformance-trigger-kubeconfig.sh --arm-cluster-available
|
||||
# The above, then set vars.CLUSTER_AVAILABLE=true after a typed confirmation —
|
||||
# refused unless every precondition is green.
|
||||
#
|
||||
# Options:
|
||||
# --dry-run measure only; no TokenRequest, no PUT/POST
|
||||
# --arm-cluster-available additionally set vars.CLUSTER_AVAILABLE=true (guarded)
|
||||
# --server URL API server URL to embed (default: the operator context's own)
|
||||
# --context NAME kube context to read the CA/server from (default: k3s-prod)
|
||||
# --duration D requested token lifetime (default: 720h = 30d)
|
||||
# -h | --help this text
|
||||
#
|
||||
# ON --server: the embedded server URL must be reachable FROM THE CI RUNNER POD, which is
|
||||
# not necessarily the address your workstation uses. The runner has broad egress and reaches
|
||||
# the kube-API VIP (per the plan: 192.168.1.49:6443). If your operator kubeconfig points at
|
||||
# a workstation-only name or a localhost tunnel, pass --server explicitly; the script warns
|
||||
# on a loopback server but cannot know your topology for you.
|
||||
#
|
||||
# ON TOKEN LIFETIME: this is a TokenRequest token and it EXPIRES. The API server may cap the
|
||||
# requested duration, so the script decodes the JWT `exp` claim and reports the ACTUAL
|
||||
# expiry rather than trusting --duration. Re-run this script to rotate. After expiry the
|
||||
# trigger gets 401 and the conformance stage FAILS (loudly) rather than skipping — which is
|
||||
# the correct direction, but put the expiry date in your calendar.
|
||||
#
|
||||
# DELIBERATELY NOT DONE: SOPS-sealing the token into secrets/conformance-trigger.sops.yaml
|
||||
# (plan §3 item 6's "for reproducibility/rotation"). A bearer token with a 30-day life
|
||||
# cannot usefully survive the branch -> PR -> review -> merge flow (CLAUDE.md §6), and
|
||||
# committing one buys re-issuance we get for free by re-running this script against the
|
||||
# live SA. If the operator still wants it sealed, that is a separate reviewed change.
|
||||
#
|
||||
# FORBIDDEN HERE, ON PURPOSE: no cluster-admin kubeconfig, no copy of the operator's own
|
||||
# kubeconfig, no widening of conformance/incluster/rbac.yaml, no mutating kubectl beyond the
|
||||
# TokenRequest itself, no `kubectl apply`.
|
||||
set -euo pipefail
|
||||
|
||||
# xtrace would print the token. Refuse rather than leak.
|
||||
case "$-" in
|
||||
*x*) echo "FATAL: refusing to run under 'set -x' — xtrace would print the token" >&2; exit 1 ;;
|
||||
esac
|
||||
umask 077
|
||||
|
||||
HOST=gitea.home.keithjackson.dev
|
||||
OWNER=Home-Network
|
||||
REPO=home-network-k3s-impl
|
||||
# Legal secret name: NOT prefixed GITEA_ (lesson 1), and the name the workflows already use.
|
||||
SECRET=CONFORMANCE_TRIGGER_KUBECONFIG
|
||||
VARIABLE=CLUSTER_AVAILABLE
|
||||
NS=conformance
|
||||
SA=conformance-trigger
|
||||
RUNNER_SA=conformance-runner
|
||||
POLICY=conformance-trigger-job-shape
|
||||
# The image prefix the admission policy admits. Kept in sync with
|
||||
# platform/kyverno/admission/conformance-trigger-job-shape.yaml by the P2 check below.
|
||||
RUNNER_IMAGE_PREFIX='gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:'
|
||||
|
||||
KCTX=k3s-prod
|
||||
DURATION=720h
|
||||
DRY_RUN=false
|
||||
ARM=false
|
||||
SERVER_OVERRIDE=""
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||
|
||||
step() { printf '\n==> %s\n' "$*"; }
|
||||
info() { printf ' %s\n' "$*"; }
|
||||
warn() { printf ' WARN: %s\n' "$*" >&2; }
|
||||
die() { printf 'FATAL: %s\n' "$*" >&2; exit 1; }
|
||||
|
||||
usage() {
|
||||
cat <<'USAGE'
|
||||
conformance-trigger-kubeconfig.sh — provision the narrow L2/L3 conformance CI trigger credential.
|
||||
|
||||
--dry-run measure every precondition; mint nothing, write nothing (read-only)
|
||||
--arm-cluster-available additionally set vars.CLUSTER_AVAILABLE=true (guarded + confirmed)
|
||||
--server URL API server URL to embed (default: the operator context's own;
|
||||
must be reachable FROM THE CI RUNNER POD — the kube-API VIP)
|
||||
--context NAME kube context to read the CA/server from (default: k3s-prod)
|
||||
--duration D requested token lifetime (default: 720h)
|
||||
-h | --help this text
|
||||
|
||||
Read the header of this file before running it: the CLUSTER_AVAILABLE guard, the split between
|
||||
mint-blocking and arm-blocking preconditions, and why nothing secret is ever printed.
|
||||
USAGE
|
||||
}
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--dry-run) DRY_RUN=true ;;
|
||||
--arm-cluster-available) ARM=true ;;
|
||||
--server) SERVER_OVERRIDE="${2:?--server needs a URL}"; shift ;;
|
||||
--context) KCTX="${2:?--context needs a name}"; shift ;;
|
||||
--duration) DURATION="${2:?--duration needs a value}"; shift ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*) die "unknown argument: $1 (try --help)" ;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
if $DRY_RUN && $ARM; then die "--dry-run and --arm-cluster-available are contradictory"; fi
|
||||
|
||||
for t in kubectl jq curl git base64; do
|
||||
command -v "$t" >/dev/null || die "missing required tool: $t"
|
||||
done
|
||||
|
||||
WORK="$(mktemp -d)"
|
||||
KC="${WORK}/conformance-trigger.kubeconfig"
|
||||
cleanup() { rm -rf "$WORK"; }
|
||||
trap cleanup EXIT
|
||||
trap 'cleanup; exit 130' INT TERM
|
||||
|
||||
# ========================================================================================
|
||||
# 0. Gitea credentials — from the credential helper git already has for this host, exactly
|
||||
# as branch-protection.sh / set-ci-token.sh / require-gate.sh do. Never in argv: curl
|
||||
# reads them from a --config file descriptor.
|
||||
# ========================================================================================
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"
|
||||
P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
unset creds
|
||||
[ -n "$U" ] && [ -n "$P" ] || die "no stored credential for ${HOST} (git credential fill returned nothing)"
|
||||
basic() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
||||
API="https://${HOST}/api/v1/repos/${OWNER}/${REPO}"
|
||||
|
||||
# ========================================================================================
|
||||
# 1. PRECONDITIONS — measured, never assumed (CLAUDE.md §2b: query the thing itself).
|
||||
# Two independent tallies, per the split documented in the header.
|
||||
# ========================================================================================
|
||||
MINT_FAIL=0
|
||||
ARM_FAIL=0
|
||||
pass() { printf ' [ ok ] %s\n' "$*"; }
|
||||
fail_mint() { printf ' [MINT-BLOCK] %s\n' "$*" >&2; MINT_FAIL=$((MINT_FAIL + 1)); }
|
||||
fail_arm() { printf ' [ARM-BLOCK] %s\n' "$*" >&2; ARM_FAIL=$((ARM_FAIL + 1)); }
|
||||
|
||||
step "preconditions — cluster side (these block the MINT: without them the token IS privileged)"
|
||||
|
||||
kubectl --context "$KCTX" get --raw /version >/dev/null 2>&1 \
|
||||
|| die "cannot reach the API server via context '${KCTX}' — nothing below can be measured"
|
||||
pass "API server reachable via context ${KCTX}"
|
||||
|
||||
# C1: the trigger identity exists (created by Flux from conformance/incluster/rbac.yaml via
|
||||
# clusters/base/conformance.yaml). We print object names only, never contents.
|
||||
if kubectl --context "$KCTX" -n "$NS" get serviceaccount "$SA" -o name >/dev/null 2>&1; then
|
||||
pass "ServiceAccount ${NS}/${SA} exists"
|
||||
else
|
||||
fail_mint "ServiceAccount ${NS}/${SA} is ABSENT — is the 'conformance' Flux Kustomization reconciled?"
|
||||
fi
|
||||
for kind in role rolebinding; do
|
||||
if kubectl --context "$KCTX" -n "$NS" get "$kind" "$SA" -o name >/dev/null 2>&1; then
|
||||
pass "${kind} ${NS}/${SA} exists"
|
||||
else
|
||||
fail_mint "${kind} ${NS}/${SA} is ABSENT — the token would authenticate but authorize nothing"
|
||||
fi
|
||||
done
|
||||
|
||||
# C2: THE SECURITY BOUNDARY. Present, Enforce, and Ready. Anything else and we do not mint.
|
||||
# `validationFailureAction: Audit` is not a boundary; an unready policy is not a boundary.
|
||||
if pol="$(kubectl --context "$KCTX" get clusterpolicy "$POLICY" -o json 2>/dev/null)"; then
|
||||
action="$(jq -r '.spec.validationFailureAction // .spec.rules[0].validate.failureAction // "UNSET"' <<<"$pol")"
|
||||
failpol="$(jq -r '.spec.failurePolicy // "UNSET"' <<<"$pol")"
|
||||
# Kyverno moved readiness from .status.ready to a Ready condition; accept either, and
|
||||
# treat "neither key present" as UNMEASURABLE -> block (fail closed, do not assume ready).
|
||||
ready="$(jq -r '
|
||||
if (.status.ready != null) then (.status.ready | tostring)
|
||||
elif (.status.conditions != null) then
|
||||
((.status.conditions[] | select(.type == "Ready") | .status) // "UNKNOWN")
|
||||
else "UNKNOWN" end' <<<"$pol")"
|
||||
info "ClusterPolicy ${POLICY}: validationFailureAction=${action} failurePolicy=${failpol} ready=${ready}"
|
||||
[ "$action" = "Enforce" ] || fail_mint "ClusterPolicy ${POLICY} is '${action}', not Enforce — the trigger would be unbounded"
|
||||
[ "$failpol" = "Fail" ] || fail_mint "ClusterPolicy ${POLICY} failurePolicy='${failpol}', not Fail — a Kyverno outage would admit arbitrary Jobs"
|
||||
case "$ready" in
|
||||
true|True) pass "ClusterPolicy ${POLICY} reports Ready" ;;
|
||||
*) fail_mint "ClusterPolicy ${POLICY} does not report Ready (got '${ready}') — refusing to assume it is loaded" ;;
|
||||
esac
|
||||
else
|
||||
fail_mint "ClusterPolicy ${POLICY} is ABSENT — plan §4's boundary does not exist; a token with 'create jobs' here can run arbitrary code as ${RUNNER_SA}"
|
||||
fi
|
||||
|
||||
step "preconditions — repo side (these block ONLY arming ${VARIABLE}; the secret may be stored first)"
|
||||
|
||||
# R1: the runner image is built and digest-pinned. Without it the Job cannot run run.sh at
|
||||
# all (the stock chainsaw image has no bash/python3/kubectl/git) — plan §3 item 1.
|
||||
VERSIONS="${REPO_ROOT}/conformance/versions.env"
|
||||
if grep -Eq '^[[:space:]]*CONFORMANCE_RUNNER_IMAGE=.*@sha256:[0-9a-f]{64}' "$VERSIONS" 2>/dev/null; then
|
||||
pass "CONFORMANCE_RUNNER_IMAGE is digest-pinned in conformance/versions.env"
|
||||
else
|
||||
fail_arm "conformance/versions.env has no digest-pinned CONFORMANCE_RUNNER_IMAGE (plan §3 item 1) — the Job cannot run ./run.sh"
|
||||
fi
|
||||
|
||||
# R2: the Job template is the real thing, not the skeleton: the admitted image AND an
|
||||
# init-container that populates the empty `workspace` volume with the suite at the PR SHA
|
||||
# (plan §3 item 2). A template whose image the policy will not admit is a guaranteed denial.
|
||||
JOBTPL="${REPO_ROOT}/conformance/incluster/chainsaw-job.yaml"
|
||||
if grep -q 'initContainers' "$JOBTPL" 2>/dev/null; then
|
||||
pass "chainsaw-job.yaml declares an initContainers block (workspace population)"
|
||||
else
|
||||
fail_arm "chainsaw-job.yaml has no initContainers — 'workspace' is an empty emptyDir and the suite is never delivered (plan §3 item 2)"
|
||||
fi
|
||||
if grep -q "$RUNNER_IMAGE_PREFIX" "$JOBTPL" 2>/dev/null; then
|
||||
pass "chainsaw-job.yaml uses the digest-pinned image the admission policy admits"
|
||||
else
|
||||
fail_arm "chainsaw-job.yaml does not use ${RUNNER_IMAGE_PREFIX}… — every triggered Job would be DENIED by ${POLICY}"
|
||||
fi
|
||||
|
||||
# R3: ci/conformance.sh actually drives the in-cluster Job. Today it mentions
|
||||
# CONFORMANCE_TRIGGER_KUBECONFIG only in a TODO comment and otherwise runs run.sh LOCALLY in
|
||||
# the CI job — where there is no credential at all (the runner sets
|
||||
# automountServiceAccountToken: false by design), so it can only scaffold-skip. Hence the
|
||||
# check must ignore comment lines: a mention in prose is not an implementation.
|
||||
#
|
||||
# ONE awk, NOT `grep -v … | grep -q …`: under `set -o pipefail` a successful `grep -q` exits
|
||||
# early, the upstream grep takes EPIPE, and the pipeline reports non-zero — which would read
|
||||
# here as "unwired" exactly when the seam IS wired. A single process cannot have that bug.
|
||||
CONF_STAGE="${REPO_ROOT}/ci/conformance.sh"
|
||||
if awk '!/^[[:space:]]*#/ && /CONFORMANCE_TRIGGER_KUBECONFIG/ { found = 1 } END { exit !found }' \
|
||||
"$CONF_STAGE" 2>/dev/null; then
|
||||
pass "ci/conformance.sh references CONFORMANCE_TRIGGER_KUBECONFIG in code (in-cluster trigger path wired)"
|
||||
else
|
||||
fail_arm "ci/conformance.sh mentions CONFORMANCE_TRIGGER_KUBECONFIG only in comments — the trigger path is UNIMPLEMENTED (plan §3 item 4); arming ${VARIABLE} now only changes which way the stage fails"
|
||||
fi
|
||||
|
||||
info ""
|
||||
info "precondition tally: mint-blocking failures=${MINT_FAIL} arm-blocking failures=${ARM_FAIL}"
|
||||
|
||||
if [ "$MINT_FAIL" -gt 0 ]; then
|
||||
cat >&2 <<EOF
|
||||
|
||||
REFUSING TO MINT. ${MINT_FAIL} cluster-side precondition(s) failed.
|
||||
|
||||
Minting a token for ${NS}/${SA} while the admission boundary is absent/Audit/unready hands CI
|
||||
an identity that can create a Job running as ${RUNNER_SA} (nodes/patch, pods/exec,
|
||||
namespaces/patch) with an arbitrary command. That is precisely the escalation
|
||||
docs/plans/l2-l3-conformance-ci-wiring.md §4 exists to close, and RBAC cannot close it.
|
||||
|
||||
Fix the cluster side first (reconcile platform/kyverno/admission/ and conformance/incluster/,
|
||||
confirm the ClusterPolicy is Enforce + Ready), then re-run. Nothing was created.
|
||||
EOF
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if $ARM && [ "$ARM_FAIL" -gt 0 ]; then
|
||||
cat >&2 <<EOF
|
||||
|
||||
REFUSING TO ARM ${VARIABLE}. ${ARM_FAIL} repo-side precondition(s) failed.
|
||||
|
||||
Measured 2026-09-29: with ${VARIABLE}=true and no working trigger path, conformance/run.sh
|
||||
scaffold-skips all 54 tests at rc 0 and ci/conformance.sh used to print
|
||||
"[ ok ] L2/L3 conformance passed" — a fake green on an enforcing required status check. The
|
||||
guard now in ci/conformance.sh turns that into exit 1 instead, so arming today does not
|
||||
enable L2/L3: it breaks every PR while proving nothing.
|
||||
|
||||
Land plan §3 items 1, 2 and 4 first. Re-run WITHOUT --arm-cluster-available to provision the
|
||||
secret in the meantime — that part is safe and is step 6's intended order.
|
||||
EOF
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if $DRY_RUN; then
|
||||
step "DRY RUN — stopping here. Nothing was minted, nothing was written."
|
||||
info "secret that WOULD be written: ${SECRET} (repo ${OWNER}/${REPO})"
|
||||
info "variable that WOULD be set: none — ${VARIABLE} is never touched in --dry-run"
|
||||
# Read-only visibility into what is already there. Gitea never returns secret VALUES.
|
||||
names="$(basic "${API}/actions/secrets" | jq -rc '[.. | objects | select(has("name")) | .name] | unique')" || true
|
||||
info "existing Actions secret names: ${names:-<unreadable>}"
|
||||
cur="$(basic -o "${WORK}/var.json" -w '%{http_code}' "${API}/actions/variables/${VARIABLE}")" || true
|
||||
if [ "$cur" = 200 ]; then
|
||||
info "existing ${VARIABLE} = $(jq -r '.data // "<no data field>"' "${WORK}/var.json")"
|
||||
else
|
||||
info "existing ${VARIABLE}: not set (HTTP ${cur})"
|
||||
fi
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ========================================================================================
|
||||
# 2. MINT the ServiceAccount token (TokenRequest). This is the only write this script makes
|
||||
# against the cluster, and it creates no durable object.
|
||||
# ========================================================================================
|
||||
step "minting a ServiceAccount token for ${NS}/${SA} (requested duration ${DURATION})"
|
||||
TOKEN="$(kubectl --context "$KCTX" -n "$NS" create token "$SA" --duration="$DURATION")" \
|
||||
|| die "kubectl create token failed for ${NS}/${SA}"
|
||||
[ -n "$TOKEN" ] || die "kubectl create token returned an empty token"
|
||||
info "got a ${#TOKEN}-character token (value NOT shown)"
|
||||
|
||||
# Report the ACTUAL expiry from the JWT rather than trusting --duration: the API server caps
|
||||
# TokenRequest lifetimes (--service-account-max-token-expiration) and silently shortens.
|
||||
jwt_exp() {
|
||||
local payload pad b64
|
||||
payload="$(cut -d. -f2 <<<"$1")"
|
||||
pad=$(( (4 - ${#payload} % 4) % 4 ))
|
||||
b64="$(tr '_-' '/+' <<<"$payload")$(printf '%*s' "$pad" '' | tr ' ' '=')"
|
||||
printf '%s' "$b64" | base64 -d 2>/dev/null | jq -r '.exp // empty' 2>/dev/null || true
|
||||
}
|
||||
exp="$(jwt_exp "$TOKEN")"
|
||||
if [ -n "$exp" ]; then
|
||||
info "token expires: $(date -u -d "@${exp}" '+%Y-%m-%dT%H:%M:%SZ' 2>/dev/null || echo "epoch ${exp}") — re-run this script to rotate"
|
||||
else
|
||||
warn "could not decode the token's exp claim; lifetime is UNVERIFIED (requested ${DURATION})"
|
||||
fi
|
||||
|
||||
# ========================================================================================
|
||||
# 3. ASSEMBLE the kubeconfig — SA token + cluster CA + server. Nothing else.
|
||||
#
|
||||
# The CA and the server URL are read from the operator's context, but the operator's
|
||||
# client certificate/key are NEVER copied: only .cluster fields are extracted. `config
|
||||
# view --raw` output is held in a variable and discarded, never printed.
|
||||
# Built with jq (not `kubectl config set-credentials --token=…`) so the token never
|
||||
# appears in argv where `ps` can see it.
|
||||
# ========================================================================================
|
||||
step "assembling the kubeconfig (SA token only — never cluster-admin, never the operator's)"
|
||||
raw="$(kubectl --context "$KCTX" config view --raw --minify -o json)" \
|
||||
|| die "could not read context '${KCTX}' from the operator kubeconfig"
|
||||
SERVER="$(jq -r '.clusters[0].cluster.server // empty' <<<"$raw")"
|
||||
CADATA="$(jq -r '.clusters[0].cluster["certificate-authority-data"] // empty' <<<"$raw")"
|
||||
CAFILE="$(jq -r '.clusters[0].cluster["certificate-authority"] // empty' <<<"$raw")"
|
||||
INSECURE="$(jq -r '.clusters[0].cluster["insecure-skip-tls-verify"] // false | tostring' <<<"$raw")"
|
||||
unset raw
|
||||
|
||||
[ -n "$SERVER" ] || die "context '${KCTX}' has no cluster.server"
|
||||
if [ -n "$SERVER_OVERRIDE" ]; then
|
||||
info "overriding server ${SERVER} -> ${SERVER_OVERRIDE}"
|
||||
SERVER="$SERVER_OVERRIDE"
|
||||
fi
|
||||
case "$SERVER" in
|
||||
*localhost*|*127.0.0.1*|*://0.0.0.0*)
|
||||
die "server '${SERVER}' is loopback — the CI runner pod cannot reach it. Pass --server with the address the runner uses (the kube-API VIP)." ;;
|
||||
esac
|
||||
if [ -z "$CADATA" ] && [ -n "$CAFILE" ]; then
|
||||
[ -r "$CAFILE" ] || die "certificate-authority file '${CAFILE}' is not readable"
|
||||
CADATA="$(base64 -w0 <"$CAFILE")"
|
||||
fi
|
||||
[ -n "$CADATA" ] || die "no cluster CA available for '${KCTX}' (insecure-skip-tls-verify=${INSECURE}); refusing to emit a kubeconfig that trusts anything"
|
||||
[ "$INSECURE" = "false" ] || die "context '${KCTX}' sets insecure-skip-tls-verify; refusing to propagate that into CI"
|
||||
info "server: ${SERVER}"
|
||||
info "CA: ${#CADATA} base64 characters embedded (public certificate, not a secret)"
|
||||
|
||||
jq -n --arg srv "$SERVER" --arg ca "$CADATA" --arg tok "$TOKEN" --arg ns "$NS" --arg sa "$SA" '
|
||||
{
|
||||
apiVersion: "v1",
|
||||
kind: "Config",
|
||||
clusters: [ { name: "conformance", cluster: { server: $srv, "certificate-authority-data": $ca } } ],
|
||||
users: [ { name: $sa, user: { token: $tok } } ],
|
||||
contexts: [ { name: "conformance", context: { cluster: "conformance", user: $sa, namespace: $ns } } ],
|
||||
"current-context": "conformance"
|
||||
}' >"$KC"
|
||||
chmod 600 "$KC"
|
||||
info "kubeconfig written to the temp dir: $(wc -c <"$KC") bytes (content NOT shown)"
|
||||
|
||||
# Prove kubectl can PARSE the file we generated (JSON is valid YAML, but assert it rather
|
||||
# than assume it) — this prints only the server we already printed.
|
||||
parsed="$(kubectl --kubeconfig "$KC" config view -o jsonpath='{.clusters[0].cluster.server}')" \
|
||||
|| die "kubectl cannot parse the generated kubeconfig"
|
||||
[ "$parsed" = "$SERVER" ] || die "generated kubeconfig parses to server '${parsed}', expected '${SERVER}'"
|
||||
pass "kubectl parses the generated kubeconfig"
|
||||
|
||||
# ========================================================================================
|
||||
# 4. PROVE it is the narrow identity. Two independent proofs:
|
||||
# a) who the token authenticates AS (SelfSubjectReview)
|
||||
# b) an allow/deny matrix (SelfSubjectAccessReview) — both halves matter: "can create
|
||||
# jobs" proves it works, "cannot get secrets / delete jobs / exec" proves it is not
|
||||
# something wider that happens to include the trigger rights.
|
||||
# All read-only. `auth can-i` answers from the API server's own authorizer.
|
||||
# ========================================================================================
|
||||
step "verifying the credential is the NARROW trigger identity"
|
||||
EXPECT_USER="system:serviceaccount:${NS}:${SA}"
|
||||
if who="$(kubectl --kubeconfig "$KC" auth whoami -o jsonpath='{.status.userInfo.username}' 2>/dev/null)"; then
|
||||
[ "$who" = "$EXPECT_USER" ] || die "token authenticates as '${who}', expected '${EXPECT_USER}' — WRONG CREDENTIAL, nothing stored"
|
||||
pass "authenticates as ${who}"
|
||||
else
|
||||
warn "'kubectl auth whoami' unavailable (older kubectl / SelfSubjectReview denied) — identity UNVERIFIED by this check; the matrix below still constrains it"
|
||||
fi
|
||||
|
||||
# A real authorization round-trip, not just a review: `list jobs` is granted by the Role.
|
||||
kubectl --kubeconfig "$KC" -n "$NS" get jobs -o name >/dev/null \
|
||||
|| die "the minted credential cannot list Jobs in ns ${NS} — the RoleBinding is not effective; nothing stored"
|
||||
pass "can list Jobs in ns ${NS} (end-to-end authentication + authorization)"
|
||||
|
||||
cani() { # cani <expected yes|no> <verb> <resource...>
|
||||
local want="$1"; shift
|
||||
local got
|
||||
got="$(kubectl --kubeconfig "$KC" auth can-i "$@" 2>/dev/null || true)"
|
||||
case "$got" in
|
||||
yes|no) : ;;
|
||||
*) die "auth can-i '$*' returned '${got:-<nothing>}' — cannot measure the identity's rights; refusing to store an unverified credential" ;;
|
||||
esac
|
||||
if [ "$got" = "$want" ]; then
|
||||
pass "can-i $* -> ${got} (expected)"
|
||||
else
|
||||
die "can-i $* -> ${got}, expected ${want}. This credential is NOT the narrow trigger identity described in conformance/incluster/rbac.yaml. Nothing stored. Do NOT widen that Role to make this pass — investigate what it is actually bound to."
|
||||
fi
|
||||
}
|
||||
# Granted by the trigger Role (create/get/list/watch jobs; get/list/watch pods + pods/log).
|
||||
cani yes create jobs.batch -n "$NS"
|
||||
cani yes get pods -n "$NS"
|
||||
cani yes get pods/log -n "$NS"
|
||||
# Explicitly NOT granted — each is a specific escape route the Role's comments call out.
|
||||
cani no delete jobs.batch -n "$NS" # a trigger that can delete can hide a failed run
|
||||
cani no create pods/exec -n "$NS" # no execution channel into the runner pod
|
||||
cani no get secrets -n "$NS"
|
||||
cani no create jobs.batch -n default # namespaced Role: no reach outside ns conformance
|
||||
cani no get nodes # the RUNNER has nodes/patch; the trigger must not
|
||||
cani no '*' '*' --all-namespaces # not cluster-admin, stated as a measurement
|
||||
|
||||
# ========================================================================================
|
||||
# 5. STORE as the Gitea Actions repo secret, then VERIFY BY READING IT BACK (lesson 2).
|
||||
# The body goes in a 0600 file so the token is not in argv.
|
||||
# ========================================================================================
|
||||
step "storing the kubeconfig as repo Actions secret ${SECRET}"
|
||||
jq -n --rawfile d "$KC" '{data:$d}' >"${WORK}/body.json"
|
||||
chmod 600 "${WORK}/body.json"
|
||||
code="$(basic -o "${WORK}/put.out" -w '%{http_code}' -X PUT \
|
||||
-H 'Content-Type: application/json' --data @"${WORK}/body.json" \
|
||||
"${API}/actions/secrets/${SECRET}")"
|
||||
unset TOKEN
|
||||
rm -f "${WORK}/body.json" "$KC"
|
||||
if [ "$code" != 201 ] && [ "$code" != 204 ]; then
|
||||
# Safe to echo: the response body is Gitea's error message, not our payload. HTTP 400
|
||||
# "invalid secret name" here means the name hit the reserved GITEA_ prefix (lesson 1).
|
||||
die "PUT ${SECRET} failed: HTTP ${code} -- $(head -c 300 "${WORK}/put.out" 2>/dev/null)"
|
||||
fi
|
||||
info "PUT returned HTTP ${code}"
|
||||
|
||||
# curl -sS exits 0 on an HTTP error, so the status check above is necessary but NOT
|
||||
# sufficient: read the object back. Gitea never returns secret VALUES, so this proves
|
||||
# PRESENCE BY NAME only — say so rather than implying the content was checked.
|
||||
names="$(basic "${API}/actions/secrets" | jq -rc '[.. | objects | select(has("name")) | .name] | unique')"
|
||||
grep -q "$SECRET" <<<"$names" \
|
||||
|| die "PUT reported HTTP ${code} but ${SECRET} is NOT in ${names} — refusing to claim success"
|
||||
pass "stored and VERIFIED PRESENT by read-back: ${names}"
|
||||
info "(read-back proves the secret EXISTS; Gitea never returns secret values, so its CONTENT"
|
||||
info " is proven only by the first CI run that uses it — see the NOTE below.)"
|
||||
info ""
|
||||
info "FOR WHOEVER WRITES plan §3 item 4: this secret holds the kubeconfig CONTENT, not a path."
|
||||
info "The plan's sketch writes 'kubectl --kubeconfig=\$CONFORMANCE_TRIGGER_KUBECONFIG', which"
|
||||
info "would treat it as a filename. ci/conformance.sh must write the env var into a 0600 temp"
|
||||
info "file (umask 077, trap-removed) and pass THAT path — an Actions secret is a string, so a"
|
||||
info "kubeconfig cannot reach CI any other way."
|
||||
|
||||
# ========================================================================================
|
||||
# 6. vars.CLUSTER_AVAILABLE — the loaded gun. Only here, only on request, only confirmed.
|
||||
# ========================================================================================
|
||||
if ! $ARM; then
|
||||
cat <<EOF
|
||||
|
||||
==> ${VARIABLE} NOT TOUCHED (no --arm-cluster-available). This is the safe default.
|
||||
|
||||
The secret is in place; the conformance job stays gated off until the variable is set. That
|
||||
is plan §5's order: land items 1-6, prove them, and only then flip step 7's switch.
|
||||
EOF
|
||||
else
|
||||
step "arming vars.${VARIABLE}=true — every precondition above is green"
|
||||
cat <<EOF
|
||||
This makes .gitea/workflows/gate.yml's conformance job RUN on every PR. If the
|
||||
in-cluster trigger path misbehaves, ci/conformance.sh will BLOCK (exit 1) rather than
|
||||
fake a green — so the failure mode is loud PRs, not silent false confidence. Undo with:
|
||||
curl -X DELETE .../repos/${OWNER}/${REPO}/actions/variables/${VARIABLE}
|
||||
EOF
|
||||
[ -t 0 ] || die "refusing to arm ${VARIABLE} non-interactively — this needs a human confirmation"
|
||||
printf ' Type exactly "ARM %s" to proceed: ' "$VARIABLE"
|
||||
read -r confirm
|
||||
[ "$confirm" = "ARM ${VARIABLE}" ] || die "not confirmed (got '${confirm}') — ${VARIABLE} unchanged. The secret IS stored."
|
||||
|
||||
# Gitea's variables API: POST creates, PUT updates, and the body field is `value` — NOT
|
||||
# `data` as in the secrets API above. Try create, fall back to update. The value "true" is
|
||||
# not a secret, so response bodies are printed here to make a failure diagnosable.
|
||||
vcode="$(basic -o "${WORK}/var.out" -w '%{http_code}' -X POST \
|
||||
-H 'Content-Type: application/json' --data '{"value":"true"}' \
|
||||
"${API}/actions/variables/${VARIABLE}")"
|
||||
if [ "$vcode" != 201 ] && [ "$vcode" != 204 ]; then
|
||||
info "POST returned HTTP ${vcode} ($(head -c 200 "${WORK}/var.out" 2>/dev/null)); trying PUT (variable may already exist)"
|
||||
vcode="$(basic -o "${WORK}/var.out" -w '%{http_code}' -X PUT \
|
||||
-H 'Content-Type: application/json' \
|
||||
--data "$(jq -n --arg n "$VARIABLE" '{name:$n,value:"true"}')" \
|
||||
"${API}/actions/variables/${VARIABLE}")"
|
||||
fi
|
||||
[ "$vcode" = 201 ] || [ "$vcode" = 204 ] \
|
||||
|| die "could not set ${VARIABLE}: HTTP ${vcode} -- $(head -c 300 "${WORK}/var.out" 2>/dev/null)"
|
||||
info "write returned HTTP ${vcode}"
|
||||
|
||||
# Unlike a secret, a variable's value IS readable — so verify the VALUE, not just presence.
|
||||
# gate.yml compares `vars.CLUSTER_AVAILABLE == 'true'`, so it must be exactly "true".
|
||||
gcode="$(basic -o "${WORK}/varget.json" -w '%{http_code}' "${API}/actions/variables/${VARIABLE}")"
|
||||
[ "$gcode" = 200 ] || die "stored HTTP ${vcode} but GET ${VARIABLE} returned HTTP ${gcode} — refusing to claim success"
|
||||
val="$(jq -r '.data // .value // empty' "${WORK}/varget.json")"
|
||||
[ "$val" = "true" ] || die "${VARIABLE} reads back as '${val}', not 'true' — gate.yml compares against 'true' exactly"
|
||||
pass "${VARIABLE}=${val} stored and VERIFIED by read-back"
|
||||
fi
|
||||
|
||||
cat <<EOF
|
||||
|
||||
NEXT — and none of it is proven until you watch it happen:
|
||||
1. Open a THROWAWAY PR. Read the conformance job's LOG, not its tick: a skipped Gitea job
|
||||
reports success ("Has been skipped"), and a green tick is not evidence anything ran.
|
||||
2. Confirm a Job appeared in ns ${NS}, ran the touched --requirements subset, and that its
|
||||
artifacts came back. Then break an L3 test deliberately and confirm the gate BLOCKS.
|
||||
3. Only after seeing BOTH directions should the conformance check be added to branch
|
||||
protection (infra/gitea/require-gate.sh's status_check_contexts).
|
||||
4. Diary the token expiry printed above. Re-run this script to rotate; CI fails loudly
|
||||
(401 on the trigger) rather than skipping when it lapses.
|
||||
|
||||
ROLLBACK: DELETE the ${VARIABLE} variable (instant SKIP, gate reverts to L0/L1-only) and/or
|
||||
DELETE the ${SECRET} secret. The token itself is revoked by deleting the ${NS}/${SA}
|
||||
ServiceAccount, which Flux will recreate with a fresh identity.
|
||||
EOF
|
||||
@@ -49,49 +49,9 @@ Crafty's UI is gated by a **dedicated oauth2-proxy sidecar in the Crafty pod** (
|
||||
`crafty.home.keithjackson.dev`). The `crafty` Service targets the sidecar, never Crafty's `:8443`, so
|
||||
the gate is **in the data path** (fail-closed: no valid `platform-admins` session ⇒ no upstream).
|
||||
|
||||
#### Cookie isolation (why the session used to vanish)
|
||||
|
||||
This sidecar and the **shared** forward-auth oauth2-proxy are independent instances with **different
|
||||
`cookie-secret` values** (measured in-cluster — neither can decrypt the other's cookie). Both used to
|
||||
emit oauth2-proxy's *default* cookie names (`_oauth2_proxy`, `_oauth2_proxy_csrf`) scoped to
|
||||
`.home.keithjackson.dev`, so the browser held **one** cookie pair for **both**: whichever proxy
|
||||
authenticated last clobbered the other's session, and the loser logged
|
||||
`No valid authentication in request`, bounced to Keycloak, and — because the Keycloak SSO session was
|
||||
still live — was redirected straight back, clobbering the cookie again. That mutual eviction presents
|
||||
exactly as *"I signed in to Keycloak but I'm still not signed in."*
|
||||
|
||||
Fixed on this side only (renaming the shared proxy's cookie would sign out every other gated app):
|
||||
`--cookie-name=_oauth2_proxy_crafty` and a **host-scoped** `--cookie-domain=crafty.home.keithjackson.dev`.
|
||||
`--whitelist-domain` stays zone-wide — it governs post-login *redirect* targets, not cookie scope.
|
||||
|
||||
#### Crafty's own local login — this is where setup actually blocks
|
||||
|
||||
Crafty has **no OIDC**, so after oauth2-proxy admits the user they still meet Crafty's own
|
||||
username/password page. That second factor is intended. What was **not** intended — and what made the
|
||||
service impossible to set up — is that **nothing documented where that password comes from**.
|
||||
|
||||
Crafty generates it **once, on first boot only**, when the users table is empty
|
||||
(`main.py` `FRESH_INSTALL` → `DatabaseBuilder.default_settings()`), and writes it to
|
||||
**`/crafty/app/config/default-creds.txt`** (mode `0600`, on the `config` subPath of the PVC). It is
|
||||
**never** in git and **cannot** be recovered from git: a rebuild onto a fresh PVC generates a *new*
|
||||
random password. Submitting anything else yields `POST /api/v2/auth/login/` → **401**, which is exactly
|
||||
what a browser sees with no hint as to why.
|
||||
|
||||
**→ Retrieval, rotation, and lost-password recovery: [`../../runbooks/crafty-admin-credential.md`](../../runbooks/crafty-admin-credential.md).**
|
||||
|
||||
Two traps that runbook covers:
|
||||
|
||||
- Crafty's `set_passwd` console command is **unreachable** in Kubernetes — it calls
|
||||
`getpass.getpass()` and so needs a TTY, while the container runs `main.py -d -i` (daemon).
|
||||
Recovery therefore has to go through the DB with Crafty's own argon2 hasher.
|
||||
- Crafty's own 3-strikes brute-force cooldown is **inert behind this sidecar**, verified live. Crafty
|
||||
*records* failed attempts under `get_remote_ip()` (which honours `X-Forwarded-For` — the live
|
||||
tracker key is the chain `"192.168.2.132, 10.0.0.123"`) but *checks* for a cooldown under
|
||||
`self.request.remote_ip`, which behind the sidecar is always `127.0.0.1`. The two halves key
|
||||
differently, so `get_recent_attempts()` always reads an empty list and `max_login_attempts: 3`
|
||||
never fires. Good news for the operator (**repeated guesses cannot lock you out**), but it means
|
||||
the only thing actually limiting who can reach the login form is the oauth2-proxy
|
||||
`platform-admins` gate — worth knowing before anyone treats that cooldown as a control.
|
||||
> **Note:** Crafty ALSO has its own local admin login (it has no OIDC). After oauth2-proxy admits the
|
||||
> user, they still see Crafty's own login page — a second, defense-in-depth factor. The operator logs
|
||||
> in with Crafty's local admin credentials (set on first boot). This is expected, not a bug.
|
||||
|
||||
### Keycloak client
|
||||
|
||||
|
||||
@@ -150,27 +150,7 @@ spec:
|
||||
# Single provider => 302 straight to Keycloak, no oauth2-proxy interstitial button.
|
||||
- --skip-provider-button=true
|
||||
- --redirect-url=https://crafty.${BASE_DOMAIN}/oauth2/callback
|
||||
# ── COOKIE ISOLATION (bug fix 2026-09-29) ────────────────────────────────────────────
|
||||
# This sidecar and the SHARED forward-auth oauth2-proxy (platform/oauth2-proxy) are
|
||||
# INDEPENDENT instances holding DIFFERENT cookie-secrets (verified in-cluster: the two
|
||||
# `cookie-secret` values differ, so neither can decrypt the other's cookie). Both used
|
||||
# to emit oauth2-proxy's DEFAULT cookie names (`_oauth2_proxy` and `_oauth2_proxy_csrf`)
|
||||
# scoped to `.${BASE_DOMAIN}`, so the browser kept ONE pair of cookies for BOTH proxies:
|
||||
# whichever authenticated last overwrote the other's session, and the loser then logged
|
||||
# "No valid authentication in request" and bounced to Keycloak -- which, thanks to the
|
||||
# live Keycloak SSO session, silently redirected back and overwrote the cookie again.
|
||||
# That mutual eviction presents EXACTLY as the reported symptom: "I signed in to
|
||||
# Keycloak but I am still not signed in to the app."
|
||||
# Two independent fixes, both applied here (only this side changes -- renaming the
|
||||
# SHARED proxy's cookie would sign every other gated app out):
|
||||
# 1. --cookie-name: a unique name, so the two sessions cannot alias at all.
|
||||
# 2. --cookie-domain: HOST-scoped. A per-app reverse proxy has no reason to broadcast
|
||||
# its session to every other host in the zone; the shared proxy keeps the wide
|
||||
# scope because cross-host forward-auth is precisely its job.
|
||||
# --whitelist-domain deliberately stays zone-wide: it governs permitted post-login
|
||||
# REDIRECT targets, not cookie scope.
|
||||
- --cookie-name=_oauth2_proxy_crafty
|
||||
- --cookie-domain=crafty.${BASE_DOMAIN}
|
||||
- --cookie-domain=.${BASE_DOMAIN}
|
||||
- --whitelist-domain=.${BASE_DOMAIN}
|
||||
- --cookie-secure=true
|
||||
- --pass-access-token=true
|
||||
|
||||
@@ -115,12 +115,6 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
|
||||
plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
|
||||
token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
|
||||
Includes why a changed Secret does not restart anything, and why an expired token is harmless.
|
||||
- **`crafty-admin-credential`** — you signed into Keycloak and Crafty *still* asks for a password.
|
||||
Expected: Crafty 4 has no OIDC, so its own local login sits behind the oauth2-proxy gate as a second
|
||||
factor. The page tells the two layers apart from the sidecar log, and explains that the bootstrap
|
||||
password exists **only** on the PVC (`default-creds.txt`, generated once on first boot, never in git).
|
||||
Includes the argon2 reset for a lost password, why the 3-strikes lockout cannot fire behind the
|
||||
sidecar, and why deleting `crafty.sqlite` to force a new password destroys every server definition.
|
||||
|
||||
### Virtualisation
|
||||
- **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.
|
||||
|
||||
@@ -1,179 +0,0 @@
|
||||
# crafty-admin-credential — Crafty's own login rejects you after Keycloak sign-in
|
||||
|
||||
**Not alert-driven.** This is an operator procedure, linked from
|
||||
[`platform/crafty/README.md`](../platform/crafty/README.md). It covers getting into the Crafty panel the
|
||||
first time, rotating that credential, and recovering when it is lost.
|
||||
|
||||
> **Everything below prints or sets a LIVE credential.** Do not paste the output into a PR, an issue, a
|
||||
> chat, or an AI agent transcript. Read it, use it, rotate it. (Repo rule §6: secrets never enter a
|
||||
> transcript.)
|
||||
|
||||
## What this means here
|
||||
|
||||
`crafty.home.keithjackson.dev` sits behind **two** independent authentications, and only the first is
|
||||
Keycloak's:
|
||||
|
||||
1. the **oauth2-proxy sidecar** in the Crafty pod — Keycloak `homelab` realm, group `platform-admins`;
|
||||
2. **Crafty's own local username/password login** — Crafty 4 has no OIDC and cannot consume the
|
||||
sidecar's identity headers, so it shows its own login page to an already-authenticated user.
|
||||
|
||||
So "I signed into Keycloak and I'm still not signed in" is the **expected** shape of this service, and
|
||||
the real question is only ever *which* of the two rejected you. They fail differently:
|
||||
|
||||
| Where you are | What you see | Which layer |
|
||||
|---|---|---|
|
||||
| Bounced to Keycloak repeatedly, never reach Crafty | sidecar log: `No valid authentication in request` | oauth2-proxy |
|
||||
| Authenticated but `403` | not in `platform-admins` | oauth2-proxy |
|
||||
| **Crafty's own login page, credentials refused** | `POST /api/v2/auth/login/` → **401** | **Crafty local** |
|
||||
|
||||
The third row is the one that blocks setup, because **nothing in the UI tells you where that password
|
||||
comes from**. Crafty generates it **once, on first boot only** — when the users table is empty
|
||||
(`main.py` `FRESH_INSTALL` → `DatabaseBuilder.default_settings()`) — and writes it to
|
||||
`/crafty/app/config/default-creds.txt` (mode `0600`) on the `config` subPath of the `crafty-data` PVC.
|
||||
It is **never in git** and **cannot be recovered from git**: a rebuild onto a fresh PVC generates a new
|
||||
random password, and the old one is gone with the old volume.
|
||||
|
||||
Two things that are *not* the problem, both verified live:
|
||||
|
||||
- **Repeated wrong guesses cannot lock you out.** Crafty *records* failed attempts under
|
||||
`get_remote_ip()` (honours `X-Forwarded-For`) but *checks* the 3-strikes cooldown under
|
||||
`self.request.remote_ip`, which behind the sidecar is always `127.0.0.1`. The halves key
|
||||
differently, so `max_login_attempts: 3` never fires. (It also means that throttle is not a control
|
||||
you have — the `platform-admins` gate is what limits who reaches the form at all.)
|
||||
- **Crafty's `set_passwd` console command is unreachable here.** It calls `getpass.getpass()` and needs
|
||||
a TTY; the container runs `main.py -d -i` as a daemon. Recovery goes through the DB instead.
|
||||
|
||||
## First commands
|
||||
|
||||
`kubectl` does not exist on the workstation — the toolchain lives only in the pinned dev container, and
|
||||
`dev.sh` picks a current-context **per invocation**, so always name the context.
|
||||
|
||||
**1. Is the pod healthy, and is the password still the generated one?**
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c '
|
||||
kubectl --context k3s-prod -n crafty get pods -l app.kubernetes.io/name=crafty
|
||||
kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- python3 -c "
|
||||
import sqlite3
|
||||
c=sqlite3.connect(\"file:/crafty/app/config/db/crafty.sqlite?mode=ro\",uri=True)
|
||||
for r in c.execute(\"SELECT username,created,last_update,last_login FROM users WHERE user_id=1\"):
|
||||
print(\"created :\",r[1]); print(\"last_update:\",r[2]); print(\"last_login :\",r[3])
|
||||
"'
|
||||
```
|
||||
|
||||
Read `last_update` against `created`: **equal ⇒ the password has never been changed**, so
|
||||
`default-creds.txt` is still live. Different ⇒ someone rotated it and that file is stale — skip to
|
||||
*Reset*.
|
||||
|
||||
**2. Read the bootstrap credential** (prints a live secret — see the warning at the top):
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- cat /crafty/app/config/default-creds.txt'
|
||||
```
|
||||
|
||||
The username is `admin`. Sign in at `https://crafty.home.keithjackson.dev` (Keycloak first, then this).
|
||||
|
||||
**3. Rotate it immediately**, in the UI: user menu → your profile → change password. That is the only
|
||||
supported path that keeps Crafty's own bookkeeping consistent. Then confirm it took — re-run command 1
|
||||
and check `last_update` has moved away from `created`.
|
||||
|
||||
## Likely causes, most likely first
|
||||
|
||||
1. **You never had the password.** By far the most common, and the reason this runbook exists. The
|
||||
credential was only ever on the PVC. → commands 1–3 above.
|
||||
2. **The password was rotated and not recorded.** `last_update` ≠ `created` while nobody knows the
|
||||
value. `default-creds.txt` is now misleading, not authoritative. → *Reset* below.
|
||||
3. **The PVC was rebuilt.** A fresh volume means a fresh random password *and* an empty panel — no
|
||||
servers, no worlds (`platform/crafty/README.md`, "the servers themselves are NOT in git"). Read
|
||||
`default-creds.txt` from the new pod; do not expect the old credential to work.
|
||||
4. **You are actually stuck at the oauth2-proxy layer, not Crafty's.** Check which layer is talking
|
||||
before touching any password:
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n crafty logs deploy/crafty -c oauth2-proxy --tail=200 | grep -v kube-probe'
|
||||
```
|
||||
`[AuthSuccess] ... groups:[platform-admins]` followed by a `200` on `/login` means oauth2-proxy did
|
||||
its job and the 401 is Crafty's. Repeated `No valid authentication in request` with no `AuthSuccess`
|
||||
means the session cookie is not surviving — see *cookie isolation* in
|
||||
`platform/crafty/README.md`; this sidecar must keep its own `--cookie-name` distinct from the shared
|
||||
proxy's, or the two evict each other.
|
||||
5. **`platform-admins` membership lapsed.** A `403` rather than a login page. Fix in Keycloak, not here.
|
||||
|
||||
## Reset when the password is unknown
|
||||
|
||||
Crafty stores an **argon2** hash in `users.password`. This writes a fresh random password, invalidates
|
||||
existing sessions, and drops the new value into a `0600` file **inside the pod** — deliberately
|
||||
mirroring Crafty's own `default-creds.txt` idiom so the secret never crosses argv, stdin or shell
|
||||
history. (`dev.sh` does **not** forward stdin — a `kubectl exec -i` here hangs forever, which is why the
|
||||
password is generated in-pod rather than passed in.)
|
||||
|
||||
Two details that will otherwise cost you the afternoon: argon2 is installed **only** in Crafty's venv,
|
||||
so it must be `/crafty/.venv/bin/python`, not the system `python3`; and this is a **mutating operator
|
||||
action**, not an agent action.
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- /crafty/.venv/bin/python -c "
|
||||
import os, secrets, sqlite3, string, datetime
|
||||
from argon2 import PasswordHasher
|
||||
pw = \"\".join(secrets.choice(string.ascii_letters + string.digits) for _ in range(24))
|
||||
now = datetime.datetime.now()
|
||||
c = sqlite3.connect(\"/crafty/app/config/db/crafty.sqlite\", timeout=30)
|
||||
cur = c.execute(
|
||||
\"UPDATE users SET password=?, last_update=?, valid_tokens_from=? WHERE username=?\",
|
||||
(PasswordHasher().hash(pw), str(now),
|
||||
str(datetime.datetime.now(datetime.timezone.utc)), \"admin\"),
|
||||
)
|
||||
c.commit()
|
||||
print(\"rows updated:\", cur.rowcount)
|
||||
out = \"/crafty/app/config/reset-creds.txt\"
|
||||
open(out, \"w\").write(pw + chr(10))
|
||||
os.chmod(out, 0o600)
|
||||
print(\"new password written to\", out)
|
||||
"'
|
||||
```
|
||||
|
||||
Expect `rows updated: 1`. Then read it, sign in, rotate in the UI, and delete the file:
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- cat /crafty/app/config/reset-creds.txt'
|
||||
# after signing in and rotating:
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- rm -f /crafty/app/config/reset-creds.txt'
|
||||
```
|
||||
|
||||
`rows updated: 0` means the username is not `admin` — list the accounts (names only, never the
|
||||
`password` column) and retry against the right one:
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- python3 -c "
|
||||
import sqlite3
|
||||
c=sqlite3.connect(\"file:/crafty/app/config/db/crafty.sqlite?mode=ro\",uri=True)
|
||||
print([r[0] for r in c.execute(\"SELECT username FROM users\")])
|
||||
"'
|
||||
```
|
||||
|
||||
`database is locked` means Crafty held the write lock; the `timeout=30` usually rides it out, otherwise
|
||||
retry. The panel picks the new hash up on the next login attempt — no restart needed, and a restart
|
||||
costs a Minecraft-server stop (`strategy: Recreate` on an RWO volume), so do not restart reflexively.
|
||||
|
||||
## When to escalate or stop
|
||||
|
||||
- **Stop** as soon as you can sign in, and **rotate immediately** — do not leave the generated
|
||||
bootstrap password in place as the standing admin credential, and do not leave `reset-creds.txt` on
|
||||
the PVC.
|
||||
- **Do not `kubectl edit`/`patch`/`apply`** the Deployment to work around a login problem. Flux
|
||||
reconciles `main`; hand-applied changes are drift. Manifest changes go through a branch and a PR.
|
||||
- **Do not delete `crafty.sqlite`** to force a "fresh install" and a new generated password. It holds
|
||||
every server definition — jar, ports, heap, `auto_start` — and losing it means recreating each server
|
||||
by hand, re-accepting the Mojang EULA, and re-`PATCH`ing `auto_start`. The worlds under
|
||||
`/crafty/servers/<uuid>/` are not in git and not recreatable. The reset above is always cheaper.
|
||||
- **Escalate to the Keycloak side** (`runbooks/keycloak-down.md`,
|
||||
`runbooks/keycloak-client-secrets.md`) if there is no `AuthSuccess` in the sidecar log at all — that
|
||||
is an IdP or client-secret problem and no Crafty password will help.
|
||||
- **If `default-creds.txt` is absent on a pod whose DB says the password was never changed**, stop and
|
||||
hand off: that combination should not occur, and it suggests the `config` subPath is not the volume
|
||||
you think it is. Record what you measured rather than guessing.
|
||||
@@ -60,9 +60,6 @@ configMapGenerator:
|
||||
- cnpg-exporter=cnpg-exporter.md
|
||||
- cnpg-primary-spread=cnpg-primary-spread.md
|
||||
- cnpg-wal-growth=cnpg-wal-growth.md
|
||||
# Operator procedure, not alert-linked (Crafty's local login is a second factor behind
|
||||
# the Keycloak gate and its bootstrap password lives only on the PVC).
|
||||
- crafty-admin-credential=crafty-admin-credential.md
|
||||
- denied-flows=denied-flows.md
|
||||
- flux-reconcile=flux-reconcile.md
|
||||
- flux-suspended=flux-suspended.md
|
||||
|
||||
Reference in new issue
Block a user