From d1a547fd15209afd2239ca173079273a097cee88 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Mon, 31 Aug 2026 09:01:18 -0500 Subject: [PATCH] docs(plan): L2/L3 conformance CI wiring plan; gate.yml placeholder -> plan pointer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit HAND-OFF 2 (L2/L3 conformance wiring) — DEFERRED with a concrete plan, per the hand-off's escape hatch (too risky to wire live cluster access from CI autonomously; do NOT hand CI a broad kubeconfig). The intended seam (in-cluster Chainsaw Job under its own scoped SA; CI holds only a narrow trigger credential) is mostly scaffolded but not safe to finish autonomously: - ci/conformance.sh still runs run.sh locally rather than triggering the in-cluster Job. - conformance/incluster/chainsaw-job.yaml has no repo-clone init-container and the stock chainsaw image lacks bash/python3/kubectl/git that run.sh needs. - No trigger SA/RBAC/credential exists, and the core blocker is a privilege-escalation design question: any `create jobs` credential in the conformance namespace can create a Job running as the privileged conformance-runner SA. This needs an admission constraint (recommended: narrow trigger SA + a Kyverno policy pinning the admissible Job shape), or a GitOps-native zero-credential trigger — an operator decision. - conformance/incluster/ isn't wired into Flux; Gitea vars/secrets aren't provisioned. docs/plans/l2-l3-conformance-ci-wiring.md captures current state, the security analysis, the functional gaps, a file-by-file step sequence, validation/rollback, and the open decisions. gate.yml's conformance placeholder now points to it (still if:false, honest). Co-Authored-By: Claude Opus 4.8 --- .gitea/workflows/gate.yml | 19 +-- docs/plans/l2-l3-conformance-ci-wiring.md | 185 ++++++++++++++++++++++ 2 files changed, 195 insertions(+), 9 deletions(-) create mode 100644 docs/plans/l2-l3-conformance-ci-wiring.md diff --git a/.gitea/workflows/gate.yml b/.gitea/workflows/gate.yml index d4354ae..0764382 100644 --- a/.gitea/workflows/gate.yml +++ b/.gitea/workflows/gate.yml @@ -84,15 +84,16 @@ jobs: # Gitea-compatible @v3 action or Gitea's own artifact backend if retention is wanted. # ─────────────────────────────────────────────────────────────────────────── - # PLACEHOLDER — NOT WIRED (STOP-and-report item, WS-GITOSS Phase 6). - # L2/L3 conformance (Chainsaw behavioural + fault-injection) needs a LIVE cluster. - # Turning this on is a design decision with two open sub-decisions: - # 1. Toolchain: build+push a pinned CI image to the Gitea container registry and - # reference it as the job image, vs. install-in-job from ci/versions.env (slower). - # 2. Cluster creds: how the runner reaches the cluster — the intended model is an - # in-cluster Chainsaw Job driven by its OWN scoped ServiceAccount (CI holds only a - # narrow trigger credential; the runner mounts NO SA token today). That SA + RBAC - # + the trigger secret do not exist yet. + # PLACEHOLDER — NOT WIRED (deferred; concrete plan written, awaiting operator decision). + # L2/L3 conformance (Chainsaw behavioural + fault-injection) needs a LIVE cluster and is + # gated behind vars.CLUSTER_AVAILABLE + a narrow secrets.CONFORMANCE_TRIGGER_KUBECONFIG. + # The intended model: an in-cluster Chainsaw Job (conformance/incluster/) under its OWN + # scoped ServiceAccount — CI holds NO broad kubeconfig, only a trigger credential. The + # blocker is a real privilege-escalation design question (a `create jobs` credential can + # create a Job running as the privileged runner SA), plus a runner image + repo-clone + + # results-collection that don't exist yet. Full analysis + step sequence: + # docs/plans/l2-l3-conformance-ci-wiring.md + # The blueprint job shape is retained dormant in merge-gate.yaml. # Kept `if: false` so it never runs (shows skipped, never fakes green — HARD RULE 4). # ─────────────────────────────────────────────────────────────────────────── conformance: diff --git a/docs/plans/l2-l3-conformance-ci-wiring.md b/docs/plans/l2-l3-conformance-ci-wiring.md new file mode 100644 index 0000000..8378ae9 --- /dev/null +++ b/docs/plans/l2-l3-conformance-ci-wiring.md @@ -0,0 +1,185 @@ +# L2/L3 conformance CI wiring — implementation plan + +**Status:** PLAN — deferred, awaiting operator decision on the trigger-credential model. No +cluster changes made. +**Author:** WS-GITOSS agent, 2026-08-31 (Hand-off 2). +**Scope:** wire the L2 (behavioural) + L3 (fault-injection) conformance layers into the LIVE +Gitea Actions merge gate, running them as an **in-cluster Chainsaw Job** so CI holds **no broad +kubeconfig** (docs/14 "Where each layer runs", ADR-0011, R-SEC-02). + +> This companions the landed Hand-off 1 (commit `27bc024`: `make -C ci gate` parity + the +> comment-only R-ID scan fix). The `conformance` job in `.gitea/workflows/gate.yml` is still an +> honest `if: false` placeholder — this plan is what turns it on **safely**. + +--- + +## 1. Why this is a written plan, not a landed change + +The intended seam is real and mostly scaffolded, but finishing it autonomously would mean +handing a shared, **live** cluster a new privileged identity reachable from CI, plus building +several missing functional pieces. The hand-off's rule is explicit: *if wiring real cluster +access from CI is too risky/ambiguous to finish autonomously, STOP and write a concrete plan — +do NOT hand CI a broad kubeconfig.* The blocker is a genuine **privilege-escalation** design +question (§4), not just plumbing, and the cluster is being changed concurrently by sibling +agents. So the credential model must be decided by the operator before anything is applied. + +--- + +## 2. Current state (what exists today) + +| Piece | File | State | +|---|---|---| +| Live gate workflow | `.gitea/workflows/gate.yml` | LIVE. `gate` job runs `make -C ci gate`. `conformance` job is `if: false` placeholder. | +| Dormant full blueprint | `.gitea/workflows/merge-gate.yaml` | Shows the intended shape: `conformance` job `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}`, `secrets.CONFORMANCE_TRIGGER_KUBECONFIG`, `make -C ci conformance`, then a `merge-gate` decision job (`scripts/decide.sh`). Manual-only (`workflow_dispatch`). | +| CI conformance stage | `ci/conformance.sh` | SKIPS (exit 3) unless `CLUSTER_AVAILABLE=true`. When "on", it currently runs `conformance/run.sh --layer l2,l3` **locally in the CI job** and collects `_artifacts/`. Its own header carries `TODO(Phase 1): drive the in-cluster Chainsaw Job under the narrow CONFORMANCE_TRIGGER_KUBECONFIG` — i.e. the trigger path is **not implemented**. | +| Conformance runner | `conformance/run.sh` + `conformance/lib/`, `conformance/tests/` | Works locally/in-cluster. `--layer l2,l3 [--requirements ...]`, writes `_artifacts/report.junit.xml` + `coverage/`. Needs bash + python3 + chainsaw (+ kubectl for `conf_cluster_reachable`). | +| In-cluster Job skeleton | `conformance/incluster/chainsaw-job.yaml` | Skeleton: `generateName: conformance-run-`, ns `conformance`, `serviceAccountName: conformance-runner`, image `ghcr.io/kyverno/chainsaw:v0.2.12`, `command: ["./run.sh"]`, args default `--layer all --strict-coverage`. **`workspace` is an empty `emptyDir` with no init-container to populate it**, and `results` is an `emptyDir`. | +| Scoped runner RBAC | `conformance/incluster/rbac.yaml` | ns `conformance` + SA `conformance-runner` + a **privileged** ClusterRole (namespaces create/delete/patch, pods+pods/exec, jobs create/delete, **nodes get/list/patch** for R-STATE-02 cordon, pods/log). Least-privilege, per-test-justified. **Not reconciled by Flux** (no kustomization references `conformance/incluster/`). | +| Runner deployment | `apps/wave-1/gitea-actions.yaml` | act_runner + DinD (Docker executor). **`automountServiceAccountToken: false`** — the runner and its job containers hold NO SA token. Broad egress allowed (`- {}`), so it can *reach* the kube-API VIP `192.168.1.49:6443`, but has no credential. | +| Secrets encryption | `.sops.yaml` (repo root) | age recipient `age175l8m…mfzg`; encrypts `data`/`stringData` of `*.sops.yaml` under `platform|apps|catalog/**` and `secrets/**`. Private key is the in-cluster `sops-age` Secret Flux uses. | + +**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the dev cluster +is LIVE (Phase 6). So `CLUSTER_AVAILABLE` is the only remaining gate, exactly as the blueprint says. + +--- + +## 3. Gaps to close (functional) + +1. **Runner image.** `ghcr.io/kyverno/chainsaw` is a minimal image; `run.sh` needs **bash + + python3 + chainsaw + kubectl**, and the repo-clone init needs **git**. The Job cannot run + `./run.sh` on the stock chainsaw image. → Build a pinned `conformance-runner` image (chainsaw + `v0.2.12` + kubectl `v1.36.4` + git + python3 + bash), push to the Gitea container registry, + pin its digest in `conformance/versions.env` (`CONFORMANCE_RUNNER_IMAGE`). Keep + `CHAINSAW_VERSION` as the single source of truth the image is built from. +2. **Repo at the commit-under-test.** The Job must run the suite at the PR head SHA. → Add an + **init-container** (`git clone --depth 1` of the impl repo, then `git checkout `) writing + into the shared `workspace` volume, using a **read-only** Gitea deploy token. SHA + the touched + `--requirements` are injected by the CI trigger step as Job env/args. +3. **Results collection without giving CI exec/cp.** The Job writes `_artifacts/` to the `results` + volume; CI holds only trigger/read rights (§4). → Job's last step prints a delimited + base64 tar of `_artifacts/` to stdout (`=== ARTIFACTS-TGZ-BASE64 ===`…); CI decodes it from + `kubectl logs`. No `pods/exec`, no `kubectl cp`, no results PVC exposure required. (A results + PVC + a read-only sidecar is the durable alternative if log size becomes a problem.) +4. **`ci/conformance.sh` rewrite.** Replace the local `run.sh` call with: render the Job manifest + (inject image digest, SHA, `--requirements `), `kubectl --kubeconfig=$CONFORMANCE_TRIGGER_KUBECONFIG + -n conformance create -f -`, `kubectl wait --for=condition=complete/failed`, stream logs, + decode artifacts into `results/junit` + `reports/coverage`, map Job success→0 / failure→1 / + no-cred→3 (skip). Keep the touched-subset selection it already computes. +5. **Flux wiring.** Reconcile `conformance/incluster/` (ns + `conformance-runner` SA/RBAC + the + new trigger SA/Role + the admission policy of §4) via a **non-gating** Kustomization + (`wait:false`, `dependsOn: platform-config`), mirroring `apps-vms` — a conformance identity + must never gate the app tier. +6. **Gitea provisioning (operator, out-of-GitOps).** Set repo/org `vars.CLUSTER_AVAILABLE=true` + and `secrets.CONFORMANCE_TRIGGER_KUBECONFIG` via the Gitea API + (`PUT /repos/{owner}/{repo}/actions/{variables,secrets}/{name}`). The kubeconfig embeds the + trigger SA's token only. SOPS-seal the SA token in-repo (`secrets/conformance-trigger.sops.yaml`) + for reproducibility/rotation; the Gitea secret is derived from it. + +--- + +## 4. The security crux — trigger credential must NOT grant `conformance-runner`'s power + +**Problem.** Kubernetes places no `use` check on a Pod/Job's `serviceAccountName` at creation. +So **any principal that can `create jobs` in namespace `conformance` can create a Job with +`serviceAccountName: conformance-runner` and an arbitrary `command`** — i.e. run arbitrary code +with the privileged runner SA (nodes/patch, pods/exec, jobs, namespaces/patch). A CI-held token +with plain `create jobs` therefore *is* the privileged identity. That breaks R-SEC-02 ("CI holds +no broad cluster access") even though the token's RBAC "looks" narrow. + +**Recommended model — narrow trigger SA + an admission constraint (uses the existing Kyverno L0 +stack):** + +- SA `conformance-trigger` (ns `conformance`), Role: `create` on `jobs`; `get,list,watch` on + `jobs`; `get` on `pods` and `pods/log`. Nothing else. Bound only in the `conformance` ns. +- A **Kyverno ClusterPolicy (validate, enforce, fail-closed)** that, *for Jobs created by + `system:serviceaccount:conformance:conformance-trigger`*, ADMITS only Jobs that match the + committed shape: image == the pinned `CONFORMANCE_RUNNER_IMAGE` digest, + `serviceAccountName == conformance-runner`, `command == ["./run.sh"]`, `args` matching + `^--(layer|requirements|strict-coverage)`, `securityContext` non-root/no-privilege/no host + namespaces, and no extra volumes beyond `workspace`/`results`. Any other Job from that SA is + DENIED. This is the real security boundary: it bounds CI to "run exactly the committed + conformance suite, nothing else." (Equivalent: a k8s ValidatingAdmissionPolicy bound to that SA.) +- The `chainsaw-conf-*` per-test namespaces the runner creates are already flagged in + `rbac.yaml` as needing exclusion from any cluster-wide disallow-privileged/host-namespaces + policy for R-NET-06 — carry that exception through, documented and audited. + +**Alternative model — GitOps-native, zero CI cluster credential (more work, strongest):** CI does +not reach the cluster at all. A `ConformanceRun` marker (a CR or a commit to a watched path) is +reconciled by an in-cluster controller/Flux that creates the Job from the in-repo template; the +Job publishes results back as a Gitea commit-status / a fetchable artifact that CI polls. Removes +the trigger credential entirely, at the cost of a controller and a trigger/ordering mechanism that +must complete before the merge decision. Recommended only if even the constrained trigger SA of +the primary model is deemed unacceptable. + +**Do NOT:** give CI a `cluster-admin`/broad kubeconfig; give the trigger SA unconstrained +`create jobs`; mount an SA token into the act_runner pod; or reuse `conformance-runner` as the +trigger identity. + +--- + +## 5. Concrete step sequence + +Land these as separate reviewable PRs, gate-green at each step. Steps 1–5 are in-repo/GitOps and +safe to land incrementally *without* turning L2/L3 on; only step 7 (the operator setting the Gitea +var) actually enables it. + +1. **Runner image.** `conformance/incluster/Containerfile` (chainsaw+kubectl+git+python3+bash); + build+push to the Gitea registry; pin `CONFORMANCE_RUNNER_IMAGE=/conformance-runner@sha256:…` + in `conformance/versions.env`. Add a note tying it to `CHAINSAW_VERSION`. +2. **Job manifest.** Rewrite `chainsaw-job.yaml`: swap to `CONFORMANCE_RUNNER_IMAGE`; add the + git-clone init-container (SHA + repo URL + read token env); parametrise `args` (CI injects + `--layer l2,l3 --requirements `); keep `restartPolicy: Never`, `backoffLimit: 0`, + non-root/seccomp/readonly-rootfs; add sane CPU/mem requests+limits (heavy L3 etcd load — + see `run.sh` HEAVY set + `config/chainsaw-config.yaml`). +3. **Trigger SA/RBAC + admission policy.** Add `conformance-trigger` SA + Role/RoleBinding to + `rbac.yaml`; add the Kyverno ClusterPolicy (new file under `platform/kyverno/` or + `conformance/incluster/`) constraining that SA's Jobs (§4). Add an L0 Kyverno test fixture for + the policy (admit the good Job, deny a tampered one) so `make -C ci l0` covers it. +4. **Flux Kustomization.** New `clusters//conformance.yaml` (or an `apps-conformance` + entry) — `wait:false`, `dependsOn: platform-config`, `prune:true`, pointing at + `conformance/incluster/`. No `postBuild.substituteFrom` unless needed. Verify it reconciles + Ready with the SA/RBAC/policy present and NO Job running (Jobs are created only on trigger). +5. **`ci/conformance.sh` rewrite.** Implement the trigger→wait→collect→map flow of §3.4 behind + `CONFORMANCE_TRIGGER_KUBECONFIG`; keep exit 3 when the cred is absent (honest skip). Update + `ci/README.md`. `make -C ci gate` still SKIPS L2/L3 locally (no cred) — unchanged for humans. +6. **Workflow.** Replace the `if: false` placeholder in `gate.yml`'s `conformance` job with the + blueprint shape: `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}`, `needs: [gate]`, install the + `conformance` toolchain, `env: CONFORMANCE_TRIGGER_KUBECONFIG: ${{ secrets.CONFORMANCE_TRIGGER_KUBECONFIG }}`, + `run: make -C ci conformance`. Decide: either keep the single-`gate` model (aggregate `gate.sh` + already SKIPS conformance and reports UNVERIFIED) and add conformance as a **second required + check**, or adopt the blueprint's separate `merge-gate` decision job (`scripts/decide.sh`). + Keep the job honest: skipped when the var is off, never faked green (HARD RULE 4). Add the + conformance check to branch protection only after step 7 proves it green. +7. **Enable (operator).** Provision the SOPS-sealed trigger SA token; set the Gitea + `secrets.CONFORMANCE_TRIGGER_KUBECONFIG` and `vars.CLUSTER_AVAILABLE=true`. First run on a + throwaway PR; confirm the Job is created, runs the touched subset, results are collected, and + the gate blocks on a real L2/L3 failure and passes on green. Then make it required. + +--- + +## 6. Validation & rollback + +- **Dry-run the admission policy** before granting the trigger SA: `kubectl auth can-i` matrix for + `conformance-trigger`; attempt a *tampered* Job (different command) as that SA and confirm + Kyverno DENIES it; attempt the committed Job and confirm ADMIT. +- **Fail-closed check:** temporarily break an L3 test, confirm the Job fails and the gate BLOCKS + (mirrors Hand-off 1's proof that `gate.sh` exit 1 fails the job). +- **Rollback:** unset `vars.CLUSTER_AVAILABLE` (instant SKIP, gate reverts to L0/L1-only) and/or + suspend the Flux Kustomization; delete the trigger SA token from Gitea secrets. No app-tier + impact (conformance is a non-gating tier). + +--- + +## 7. Open decisions for the operator + +1. **Trigger model:** constrained trigger SA + Kyverno admission policy (§4 primary) vs. + GitOps-native zero-credential (§4 alternative). This plan recommends the primary. +2. **Runner image host:** Gitea container registry (in-network, matches the Flux source loop) vs. + ghcr.io. Recommend Gitea registry, digest-pinned. +3. **Gate topology:** conformance as a second required check on the single-job `gate.yml` vs. + restoring the blueprint's multi-job `merge-gate.yaml` with the `decide.sh` decision job. +4. **Results transport:** base64-tar-over-logs (simple, no extra rights) vs. a results PVC + + read-only sidecar (durable). Recommend logs first; revisit if artifacts grow. +5. **Which requirements go live first:** the R-NET/R-TLS/R-DNS/R-STATE tests already green against + the live cluster (per DECISIONS-LOG) are the natural first `--requirements` subset to certify + through the wired path before widening.