home-network-k3s-impl/docs/plans/l2-l3-conformance-ci-wiring.md

L2/L3 conformance CI wiring — implementation plan

Status: PLAN — deferred, awaiting operator decision on the trigger-credential model. No cluster changes made. Author: WS-GITOSS agent, 2026-08-31 (Hand-off 2). Scope: wire the L2 (behavioural) + L3 (fault-injection) conformance layers into the LIVE Gitea Actions merge gate, running them as an in-cluster Chainsaw Job so CI holds no broad kubeconfig (docs/14 "Where each layer runs", ADR-0011, R-SEC-02).

This companions the landed Hand-off 1 (commit 27bc024: make -C ci gate parity + the comment-only R-ID scan fix). The conformance job in .gitea/workflows/gate.yml is still an honest if: false placeholder — this plan is what turns it on safely.

---

1. Why this is a written plan, not a landed change

The intended seam is real and mostly scaffolded, but finishing it autonomously would mean handing a shared, live cluster a new privileged identity reachable from CI, plus building several missing functional pieces. The hand-off's rule is explicit: if wiring real cluster access from CI is too risky/ambiguous to finish autonomously, STOP and write a concrete plan — do NOT hand CI a broad kubeconfig. The blocker is a genuine privilege-escalation design question (§4), not just plumbing, and the cluster is being changed concurrently by sibling agents. So the credential model must be decided by the operator before anything is applied.

---

2. Current state (what exists today)

Piece	File	State
Live gate workflow	.gitea/workflows/gate.yml	LIVE. gate job runs make -C ci gate. conformance job is if: false placeholder.
Dormant full blueprint	.gitea/workflows/merge-gate.yaml	Shows the intended shape: conformance job if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}, secrets.CONFORMANCE_TRIGGER_KUBECONFIG, make -C ci conformance, then a merge-gate decision job (scripts/decide.sh). Manual-only (workflow_dispatch).
CI conformance stage	ci/conformance.sh	SKIPS (exit 3) unless CLUSTER_AVAILABLE=true. When "on", it currently runs conformance/run.sh --layer l2,l3 locally in the CI job and collects _artifacts/. Its own header carries TODO(Phase 1): drive the in-cluster Chainsaw Job under the narrow CONFORMANCE_TRIGGER_KUBECONFIG — i.e. the trigger path is not implemented.
Conformance runner	conformance/run.sh + conformance/lib/, conformance/tests/	Works locally/in-cluster. --layer l2,l3 [--requirements ...], writes _artifacts/report.junit.xml + coverage/. Needs bash + python3 + chainsaw (+ kubectl for conf_cluster_reachable).
In-cluster Job skeleton	conformance/incluster/chainsaw-job.yaml	Skeleton: generateName: conformance-run-, ns conformance, serviceAccountName: conformance-runner, image ghcr.io/kyverno/chainsaw:v0.2.12, command: ["./run.sh"], args default --layer all --strict-coverage. workspace is an empty emptyDir with no init-container to populate it, and results is an emptyDir.
Scoped runner RBAC	conformance/incluster/rbac.yaml	ns conformance + SA conformance-runner + a privileged ClusterRole (namespaces create/delete/patch, pods+pods/exec, jobs create/delete, nodes get/list/patch for R-STATE-02 cordon, pods/log). Least-privilege, per-test-justified. Not reconciled by Flux (no kustomization references conformance/incluster/).
Runner deployment	apps/wave-1/gitea-actions.yaml	act_runner + DinD (Docker executor). automountServiceAccountToken: false — the runner and its job containers hold NO SA token. Broad egress allowed (- {}), so it can reach the kube-API VIP 192.168.1.49:6443, but has no credential.
Secrets encryption	.sops.yaml (repo root)	age recipient age175l8m…mfzg; encrypts data/stringData of *.sops.yaml under `platform

Cluster reality: the "Phase 0 / no cluster" language in these files is stale — the cluster is LIVE (and it is now PROD; dev was decommissioned 2026-09-12).

CORRECTION (2026-09-29): CLUSTER_AVAILABLE is NOT "the only remaining gate", and setting it today is actively DANGEROUS. Measured: with CLUSTER_AVAILABLE=true and no reachable cluster, conformance/run.sh exits 0 in scaffold mode having marked all 54 tests <skipped>, and ci/conformance.sh then printed [ ok ] L2/L3 conformance passed for touched subset — rc 0, a fake green, on a merge gate that is now an enforcing required status check. The CI runner has no cluster credential at all (automountServiceAccountToken: false, by design), so the "local run" this stage attempts can never succeed; it can only skip and, before the guard added on 2026-09-29, lie about it.

ci/conformance.sh now BLOCKS (exit 1) when a cluster is declared but every test skipped. Do not set CLUSTER_AVAILABLE=true until §3's trigger path actually exists — the variable does not enable L2/L3, it only changes which way this stage fails.

---

3. Gaps to close (functional)

1. Runner image. ghcr.io/kyverno/chainsaw is a minimal image; run.sh needs bash + python3 + chainsaw + kubectl, and the repo-clone init needs git. The Job cannot run ./run.sh on the stock chainsaw image. → Build a pinned conformance-runner image (chainsaw v0.2.12 + kubectl v1.36.4 + git + python3 + bash), push to the Gitea container registry, pin its digest in conformance/versions.env (CONFORMANCE_RUNNER_IMAGE). Keep CHAINSAW_VERSION as the single source of truth the image is built from.
2. Repo at the commit-under-test. The Job must run the suite at the PR head SHA. → Add an init-container (git clone --depth 1 of the impl repo, then git checkout <SHA>) writing into the shared workspace volume, using a read-only Gitea deploy token. SHA + the touched --requirements are injected by the CI trigger step as Job env/args.
3. Results collection without giving CI exec/cp. The Job writes _artifacts/ to the results volume; CI holds only trigger/read rights (§4). → Job's last step prints a delimited base64 tar of _artifacts/ to stdout (=== ARTIFACTS-TGZ-BASE64 ===…); CI decodes it from kubectl logs. No pods/exec, no kubectl cp, no results PVC exposure required. (A results PVC + a read-only sidecar is the durable alternative if log size becomes a problem.)
4. ci/conformance.sh rewrite. Replace the local run.sh call with: render the Job manifest (inject image digest, SHA, --requirements <touched>), kubectl --kubeconfig=$CONFORMANCE_TRIGGER_KUBECONFIG -n conformance create -f -, kubectl wait --for=condition=complete/failed, stream logs, decode artifacts into results/junit + reports/coverage, map Job success→0 / failure→1 / no-cred→3 (skip). Keep the touched-subset selection it already computes.
5. Flux wiring. Reconcile conformance/incluster/ (ns + conformance-runner SA/RBAC + the new trigger SA/Role + the admission policy of §4) via a non-gating Kustomization (wait:false, dependsOn: platform-config), mirroring apps-vms — a conformance identity must never gate the app tier.
6. Gitea provisioning (operator, out-of-GitOps). Set repo/org vars.CLUSTER_AVAILABLE=true and secrets.CONFORMANCE_TRIGGER_KUBECONFIG via the Gitea API (PUT /repos/{owner}/{repo}/actions/{variables,secrets}/{name}). The kubeconfig embeds the trigger SA's token only. SOPS-seal the SA token in-repo (secrets/conformance-trigger.sops.yaml) for reproducibility/rotation; the Gitea secret is derived from it.

---

4. The security crux — trigger credential must NOT grant conformance-runner's power

Problem. Kubernetes places no use check on a Pod/Job's serviceAccountName at creation. So any principal that can create jobs in namespace conformance can create a Job with serviceAccountName: conformance-runner and an arbitrary command — i.e. run arbitrary code with the privileged runner SA (nodes/patch, pods/exec, jobs, namespaces/patch). A CI-held token with plain create jobs therefore is the privileged identity. That breaks R-SEC-02 ("CI holds no broad cluster access") even though the token's RBAC "looks" narrow.

Recommended model — narrow trigger SA + an admission constraint (uses the existing Kyverno L0 stack):

• SA conformance-trigger (ns conformance), Role: create on jobs; get,list,watch on jobs; get on pods and pods/log. Nothing else. Bound only in the conformance ns.
• A Kyverno ClusterPolicy (validate, enforce, fail-closed) that, for Jobs created by system:serviceaccount:conformance:conformance-trigger, ADMITS only Jobs that match the committed shape: image == the pinned CONFORMANCE_RUNNER_IMAGE digest, serviceAccountName == conformance-runner, command == ["./run.sh"], args matching ^--(layer|requirements|strict-coverage), securityContext non-root/no-privilege/no host namespaces, and no extra volumes beyond workspace/results. Any other Job from that SA is DENIED. This is the real security boundary: it bounds CI to "run exactly the committed conformance suite, nothing else." (Equivalent: a k8s ValidatingAdmissionPolicy bound to that SA.)
• The chainsaw-conf-* per-test namespaces the runner creates are already flagged in rbac.yaml as needing exclusion from any cluster-wide disallow-privileged/host-namespaces policy for R-NET-06 — carry that exception through, documented and audited.

Alternative model — GitOps-native, zero CI cluster credential (more work, strongest): CI does not reach the cluster at all. A ConformanceRun marker (a CR or a commit to a watched path) is reconciled by an in-cluster controller/Flux that creates the Job from the in-repo template; the Job publishes results back as a Gitea commit-status / a fetchable artifact that CI polls. Removes the trigger credential entirely, at the cost of a controller and a trigger/ordering mechanism that must complete before the merge decision. Recommended only if even the constrained trigger SA of the primary model is deemed unacceptable.

Do NOT: give CI a cluster-admin/broad kubeconfig; give the trigger SA unconstrained create jobs; mount an SA token into the act_runner pod; or reuse conformance-runner as the trigger identity.

---

5. Concrete step sequence

Land these as separate reviewable PRs, gate-green at each step. Steps 1–5 are in-repo/GitOps and safe to land incrementally without turning L2/L3 on; only step 7 (the operator setting the Gitea var) actually enables it.

1. Runner image. conformance/incluster/Containerfile (chainsaw+kubectl+git+python3+bash); build+push to the Gitea registry; pin CONFORMANCE_RUNNER_IMAGE=<registry>/conformance-runner@sha256:… in conformance/versions.env. Add a note tying it to CHAINSAW_VERSION.
2. Job manifest. Rewrite chainsaw-job.yaml: swap to CONFORMANCE_RUNNER_IMAGE; add the git-clone init-container (SHA + repo URL + read token env); parametrise args (CI injects --layer l2,l3 --requirements <touched>); keep restartPolicy: Never, backoffLimit: 0, non-root/seccomp/readonly-rootfs; add sane CPU/mem requests+limits (heavy L3 etcd load — see run.sh HEAVY set + config/chainsaw-config.yaml).
3. Trigger SA/RBAC + admission policy. Add conformance-trigger SA + Role/RoleBinding to rbac.yaml; add the Kyverno ClusterPolicy (new file under platform/kyverno/ or conformance/incluster/) constraining that SA's Jobs (§4). Add an L0 Kyverno test fixture for the policy (admit the good Job, deny a tampered one) so make -C ci l0 covers it.
4. Flux Kustomization. New clusters/<cluster>/conformance.yaml (or an apps-conformance entry) — wait:false, dependsOn: platform-config, prune:true, pointing at conformance/incluster/. No postBuild.substituteFrom unless needed. Verify it reconciles Ready with the SA/RBAC/policy present and NO Job running (Jobs are created only on trigger).
5. ci/conformance.sh rewrite. Implement the trigger→wait→collect→map flow of §3.4 behind CONFORMANCE_TRIGGER_KUBECONFIG; keep exit 3 when the cred is absent (honest skip). Update ci/README.md. make -C ci gate still SKIPS L2/L3 locally (no cred) — unchanged for humans.
6. Workflow. Replace the if: false placeholder in gate.yml's conformance job with the blueprint shape: if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}, needs: [gate], install the conformance toolchain, env: CONFORMANCE_TRIGGER_KUBECONFIG: ${{ secrets.CONFORMANCE_TRIGGER_KUBECONFIG }}, run: make -C ci conformance. Decide: either keep the single-gate model (aggregate gate.sh already SKIPS conformance and reports UNVERIFIED) and add conformance as a second required check, or adopt the blueprint's separate merge-gate decision job (scripts/decide.sh). Keep the job honest: skipped when the var is off, never faked green (HARD RULE 4). Add the conformance check to branch protection only after step 7 proves it green.
7. Enable (operator). Provision the SOPS-sealed trigger SA token; set the Gitea secrets.CONFORMANCE_TRIGGER_KUBECONFIG and vars.CLUSTER_AVAILABLE=true. First run on a throwaway PR; confirm the Job is created, runs the touched subset, results are collected, and the gate blocks on a real L2/L3 failure and passes on green. Then make it required.

---

6. Validation & rollback

• Dry-run the admission policy before granting the trigger SA: kubectl auth can-i matrix for conformance-trigger; attempt a tampered Job (different command) as that SA and confirm Kyverno DENIES it; attempt the committed Job and confirm ADMIT.
• Fail-closed check: temporarily break an L3 test, confirm the Job fails and the gate BLOCKS (mirrors Hand-off 1's proof that gate.sh exit 1 fails the job).
• Rollback: unset vars.CLUSTER_AVAILABLE (instant SKIP, gate reverts to L0/L1-only) and/or suspend the Flux Kustomization; delete the trigger SA token from Gitea secrets. No app-tier impact (conformance is a non-gating tier).

---

7. Open decisions for the operator

1. Trigger model: constrained trigger SA + Kyverno admission policy (§4 primary) vs. GitOps-native zero-credential (§4 alternative). This plan recommends the primary.
2. Runner image host: Gitea container registry (in-network, matches the Flux source loop) vs. ghcr.io. Recommend Gitea registry, digest-pinned.
3. Gate topology: conformance as a second required check on the single-job gate.yml vs. restoring the blueprint's multi-job merge-gate.yaml with the decide.sh decision job.
4. Results transport: base64-tar-over-logs (simple, no extra rights) vs. a results PVC + read-only sidecar (durable). Recommend logs first; revisit if artifacts grow.
5. Which requirements go live first: the R-NET/R-TLS/R-DNS/R-STATE tests already green against the live cluster (per DECISIONS-LOG) are the natural first --requirements subset to certify through the wired path before widening.