jakman85andClaude Opus 5
8eb5baffa4
Implements the credential model the operator chose for wiring L2/L3 into CI: a narrow
trigger SA bounded by a fail-closed Kyverno admission policy, rather than the
zero-credential GitOps-native alternative (plan section 4, primary model).
THE ESCALATION THIS CLOSES. Kubernetes places NO `use` check on a Job's
serviceAccountName. Any principal that can `create jobs` in ns conformance can create a Job
running as conformance-runner -- nodes/patch, pods/exec, namespaces/patch -- with an
arbitrary command. A CI token with plain `create jobs` IS that privileged identity no matter
how narrow its own Role looks. RBAC cannot fix this; only constraining the SHAPE of what the
trigger may create can.
Adds:
* SA/Role/RoleBinding conformance-trigger -- create+get+list+watch jobs, get pods and
pods/log, nothing else. Namespaced. No delete (a trigger that can delete can hide a
failed run), no exec, no secrets. Results come back as a tarball on stdout read via
pods/log, so no execution channel into the runner pod is ever granted.
* ClusterPolicy conformance-trigger-job-shape -- Enforce, failurePolicy: Fail. It does NOT
use ${KYVERNO_VALIDATION_ACTION}: a security boundary an overlay can downgrade to Audit
is not a boundary. It constrains ONLY conformance-trigger, so a mistake here cannot wedge
the rest of the cluster.
* Flux Kustomization `conformance` -> ./conformance/incluster, wait:false and non-gating
like apps-vms, so a test identity can never stall the platform or app tiers.
* A kyverno test with seven cases, six of them real escalation paths.
VERIFIED by `kyverno apply` (pass: 10, fail: 6, error: 0), not assumed:
evil-command DENIED at /spec/template/spec/containers/0/command/ <- THE attack
evil-image DENIED at .../image/
tag-not-digest DENIED at .../image/ (mutable tag is republishable)
host-namespaces DENIED at /spec/template/spec/hostPID/
extra-volume DENIED by the volumes foreach
injected-args DENIED by the args foreach
good-run ADMITTED
ALSO FIXES A FAIL-OPEN IN THE GATE ITSELF. `kyverno test` does NOT fail when a policy cannot
LOAD: every row becomes "Skip / Invalid Policy" and it still prints "N tests passed" and
exits 0. Measured while writing this policy -- a schema error (subjects in a separate
match.all entry from resources) produced exactly that, and ci/l0.sh would have certified a
security policy that does not load as green. This is not specific to this policy: a typo in
restrict-host-namespaces or secret-provenance would silently disable it with the gate still
green. ci/l0.sh now scans for "Invalid Policy" and fails. Proven: broken policy -> l0 rc=1,
restored -> rc=0.
IMAGE PINNING, HONEST LIMIT: the plan specifies pinning the exact runner image digest. That
image does not exist yet, so this enforces the enforceable half -- the image must be under
the conformance repository path AND digest-pinned, never a mutable tag. Residual surface:
anyone who can PUSH to that path could publish a different digest. Tighten to the literal
digest when the image is built; it is a one-line change to the pattern, reviewed in a PR,
which is where a change to a security boundary belongs.
NOT YET WIRED, deliberately: no runner image, no trigger kubeconfig, no CLUSTER_AVAILABLE.
The identities here are inert until those exist, and PR #12 makes CLUSTER_AVAILABLE=true
block rather than fake a green in the meantime.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
242 lines
12 KiB
YAML
242 lines
12 KiB
YAML
# =============================================================================
|
|
# Scoped test ServiceAccount + RBAC for the in-cluster Chainsaw conformance Job
|
|
# (ADR-0011; docs/14 "Where each layer runs"; R-SEC-02 least-privilege).
|
|
#
|
|
# SKELETON / PLACEHOLDER — RBAC is deliberately LEFT EMPTY here and authored
|
|
# least-privilege as each fault-injection capability actually lands (T-3.0+). The
|
|
# rule is: grant nothing until a committed test needs it, then grant exactly that
|
|
# verb/resource, with a comment naming the test that requires it. This SA is a
|
|
# PRIVILEGED TEST IDENTITY — it is its own audit item (R-SEC-02): it MUST NOT be
|
|
# reused by workloads, and every widening is reviewed.
|
|
#
|
|
# CI holds NO cluster credentials. CI only triggers the Job (T-6.1b via Gitea
|
|
# Actions); the Job runs under THIS ServiceAccount, so cluster access stays
|
|
# in-cluster and scoped (docs/14, R-SEC-02).
|
|
#
|
|
# Deployment: reconciled by Flux into the `conformance` namespace once the cluster
|
|
# exists (Phase 1+). Not applied in Phase 0 (no cluster).
|
|
# =============================================================================
|
|
apiVersion: v1
|
|
kind: Namespace
|
|
metadata:
|
|
name: conformance
|
|
labels:
|
|
app.kubernetes.io/part-of: conformance
|
|
# The test namespace itself is exempt from nothing: the platform default-deny
|
|
# baseline still applies; the Job opens only the egress it declares.
|
|
pod-security.kubernetes.io/enforce: restricted
|
|
---
|
|
apiVersion: v1
|
|
kind: ServiceAccount
|
|
metadata:
|
|
name: conformance-runner
|
|
namespace: conformance
|
|
labels:
|
|
app.kubernetes.io/part-of: conformance
|
|
annotations:
|
|
conformance.home.k3s/audit: >-
|
|
PRIVILEGED TEST IDENTITY. Scoped least-privilege for fault-injection only.
|
|
Never reuse for workloads. Widen only alongside the test that needs it.
|
|
---
|
|
# ClusterRole — the union of what BEHAVIOR + FAULT tests need. Every rule is
|
|
# commented with the test/requirement that justifies it. Start empty; uncomment /
|
|
# add rules as the corresponding L2/L3 tests are delivered (do NOT pre-grant).
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: ClusterRole
|
|
metadata:
|
|
name: conformance-runner
|
|
labels:
|
|
app.kubernetes.io/part-of: conformance
|
|
rules:
|
|
# ---- Hermetic fixtures (all L2/L3): create/delete throwaway test namespaces + fixtures
|
|
- apiGroups: [""]
|
|
resources: [namespaces]
|
|
# `patch` is required by R-NET-06 (vpn-killswitch): the test relabels its own
|
|
# ephemeral namespace to PodSecurity enforce=privileged so it can launch the
|
|
# node-pinned privileged fault Job. Scoped to the per-test namespace only.
|
|
verbs: [create, delete, get, list, watch, patch]
|
|
- apiGroups: [""]
|
|
resources: [pods, services, configmaps, secrets, persistentvolumeclaims]
|
|
# LOAD-BEARING `get` ON CONFIGMAPS, BEYOND FIXTURE CREATION (R-ENV-02): the environment-
|
|
# agnostic L2 tests (dns-exposure, dns-internal-resolution, gateway-ingress-reachable,
|
|
# oidc-sso-grafana, trusted-tls) resolve the cluster's own BASE_DOMAIN / GATEWAY_VIP at run
|
|
# time from `flux-system/cluster-vars` — the same ConfigMap Flux substitutes from — via
|
|
# chainsaw `x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars')` bindings.
|
|
# If this rule is ever narrowed to the test's own namespace, those bindings fail and the
|
|
# tests error out. Narrow it only alongside a namespace-scoped read of that ConfigMap.
|
|
verbs: [create, delete, get, list, watch, patch]
|
|
- apiGroups: [""]
|
|
resources: [pods/exec]
|
|
verbs: [create] # probe exec for behavior assertions
|
|
- apiGroups: [""]
|
|
resources: [pods/log]
|
|
verbs: [get] # read fault/probe Job logs on failure
|
|
# ---- Storage-tier precondition gate (READ-ONLY). Every storage test self-skips on the
|
|
# absence of the `longhorn` StorageClass rather than failing: data-survival (R-STATE-01),
|
|
# node-loss (R-STATE-02), dr-restore (R-STATE-03), vm-data-survival / vm-lifecycle /
|
|
# vm-default-deny (R-PLAT-03). Without this read the gate cannot tell "absent" from
|
|
# "forbidden" and would skip for the wrong reason.
|
|
- apiGroups: [storage.k8s.io]
|
|
resources: [storageclasses]
|
|
verbs: [get, list]
|
|
# ---- Controller-readiness preconditions (READ-ONLY). dr-restore (R-STATE-03) skips cleanly
|
|
# unless the Velero Deployment and its node-agent DaemonSet (kopia File System Backup, the
|
|
# volume-DATA path) are actually ready — a backup path that is not up is an absent
|
|
# precondition, not a failure. No write verbs: tests never scale or patch a platform
|
|
# controller.
|
|
- apiGroups: [apps]
|
|
resources: [deployments, daemonsets]
|
|
verbs: [get, list, watch]
|
|
# ---- Probe / fault Jobs (R-TLS-01/03 tls-probe; R-NET-06 tunnel stop/start)
|
|
# `create` cannot be constrained by resourceNames, and the fault Job names are
|
|
# fixed per-test (tunnel-stop/tunnel-start/tunnel-start-finally); scoped by the
|
|
# in-namespace blast radius of the ephemeral test namespace.
|
|
- apiGroups: [batch]
|
|
resources: [jobs]
|
|
verbs: [create, delete, get, list, watch, patch]
|
|
#
|
|
# ---- R-NET-01/02 (default-deny, deny-undeclared): apply/patch/delete allow policies
|
|
# - apiGroups: [cilium.io]
|
|
# resources: [ciliumnetworkpolicies]
|
|
# verbs: [create, delete, get, list, patch]
|
|
#
|
|
# ---- R-NET-06 (VPN kill-switch, ADR-0015): the fault is a HOST systemd action
|
|
# (stop/start `vpn-egress-tunnel` on the egress-gateway node), NOT scaling an
|
|
# in-cluster gluetun Deployment (superseded design). It is driven by a privileged,
|
|
# hostPID, node-pinned Job (nsenter -t 1 -- systemctl ...). That power comes from
|
|
# PodSecurity (the test namespace relabeled to enforce=privileged) + the pod's
|
|
# securityContext, NOT from RBAC — the fault Job pod needs NO Kubernetes API
|
|
# rights. The runner only needs `jobs` create/delete + `namespaces` patch above.
|
|
# PREREQUISITE: if a cluster-wide Kyverno disallow-privileged / disallow-host-
|
|
# namespaces policy is enforced, the `chainsaw-conf-*` test namespaces MUST be in
|
|
# its exclusion set (documented, audited exception — R-SEC-02).
|
|
#
|
|
# ---- R-STATE-02 (node loss, tests/l3-fault-injection/node-loss): the fault is
|
|
# `kubectl cordon` + label of ONE node, reverted by `kubectl uncordon` + label removal
|
|
# in the test's `finally`. All three (cordon/uncordon/label) are a `patch` on the node
|
|
# object; `get`/`list` locate the victim + the finally cleanup selector. The test evicts
|
|
# only its OWN writer pod via `delete pod` (already granted above) — it deliberately does
|
|
# NOT `kubectl drain`, so no eviction rights and no reach over unrelated pods are needed.
|
|
- apiGroups: [""]
|
|
resources: [nodes]
|
|
verbs: [get, list, patch] # cordon/uncordon/label = patch spec.unschedulable + metadata.labels
|
|
#
|
|
# ---- R-UPD-02 (bad-update auto-rollback): inject a bad image tag via automation
|
|
# - apiGroups: [image.toolkit.fluxcd.io]
|
|
# resources: [imagepolicies, imagerepositories]
|
|
# verbs: [get, patch]
|
|
#
|
|
# ---- R-STATE-03 (DR restore, tests/l3-fault-injection/dr-restore): the drill creates a
|
|
# Velero Backup of its OWN scratch namespace, destroys that namespace + its PV, creates a
|
|
# Restore into a fresh namespace, and (in `finally`) issues a DeleteBackupRequest so the
|
|
# objects it wrote are removed from the off-cluster bucket again. `create` cannot be
|
|
# constrained by resourceNames; the blast radius is bounded by the fact that every object
|
|
# the test creates or deletes is named `<its-own-chainsaw-namespace>-dr-*`.
|
|
- apiGroups: [velero.io]
|
|
resources: [backups, restores, deletebackuprequests]
|
|
verbs: [create, delete, get, list, watch]
|
|
# Read-only evidence: the BSL is the off-cluster target whose Available phase + endpoint the
|
|
# test asserts; schedules prove backups are SCHEDULED (not one-off); podvolumebackups /
|
|
# podvolumerestores prove the kopia snapshot of the volume DATA reached/returned from the
|
|
# remote repository, and are dumped on failure for diagnosis.
|
|
- apiGroups: [velero.io]
|
|
resources: [backupstoragelocations, schedules, podvolumebackups, podvolumerestores]
|
|
verbs: [get, list, watch]
|
|
# R-STATE-03 needs `delete` on the PV, not merely read: the proof that the restore came from
|
|
# OFF-cluster is that no in-cluster copy of the data survived. The test asserts its own PV is
|
|
# gone, and force-deletes it if a Retain policy or a stuck finalizer left it behind. It only
|
|
# ever names the PV bound to the PVC it created itself.
|
|
- apiGroups: [""]
|
|
resources: [persistentvolumes]
|
|
verbs: [get, list, delete]
|
|
#
|
|
# ---- R-GITOPS-03 (drift): mutate a reconciled resource + observe Flux revert
|
|
# - apiGroups: [kustomize.toolkit.fluxcd.io]
|
|
# resources: [kustomizations]
|
|
# verbs: [get, list, watch] # observe only; do NOT grant reconcile suspension
|
|
#
|
|
# ---- R-PLAT-03 (VM lifecycle): create/start/stop/destroy VMs
|
|
# - apiGroups: [kubevirt.io]
|
|
# resources: [virtualmachines, virtualmachineinstances]
|
|
# verbs: [create, delete, get, list, patch]
|
|
---
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: ClusterRoleBinding
|
|
metadata:
|
|
name: conformance-runner
|
|
labels:
|
|
app.kubernetes.io/part-of: conformance
|
|
roleRef:
|
|
apiGroup: rbac.authorization.k8s.io
|
|
kind: ClusterRole
|
|
name: conformance-runner
|
|
subjects:
|
|
- kind: ServiceAccount
|
|
name: conformance-runner
|
|
namespace: conformance
|
|
---
|
|
# =============================================================================
|
|
# THE TRIGGER IDENTITY — deliberately NOT conformance-runner.
|
|
#
|
|
# THE ESCALATION THIS EXISTS TO CLOSE (docs/plans/l2-l3-conformance-ci-wiring.md §4):
|
|
# Kubernetes places NO `use` check on a Pod/Job's `serviceAccountName`. So any principal
|
|
# that can `create jobs` in this namespace can create a Job with
|
|
# `serviceAccountName: conformance-runner` and an ARBITRARY command — i.e. run arbitrary
|
|
# code with the runner's privileges (nodes/patch, pods/exec, namespaces/patch, jobs).
|
|
# A CI-held token with plain `create jobs` therefore IS the privileged identity, no matter
|
|
# how narrow its own RBAC looks. That would break R-SEC-02 ("CI holds no broad cluster
|
|
# access") while appearing to satisfy it.
|
|
#
|
|
# RBAC ALONE CANNOT FIX THIS. The real boundary is the admission policy
|
|
# platform/kyverno/admission/conformance-trigger-job-shape.yaml, which admits Jobs from
|
|
# THIS ServiceAccount only when they match the committed conformance shape. This Role is
|
|
# the necessary half; that policy is the sufficient half. Neither works alone — do not
|
|
# grant this Role without that policy reconciled and ENFORCING.
|
|
#
|
|
# Namespaced Role, bound only in `conformance`. No ClusterRole, deliberately.
|
|
# =============================================================================
|
|
apiVersion: v1
|
|
kind: ServiceAccount
|
|
metadata:
|
|
name: conformance-trigger
|
|
namespace: conformance
|
|
labels:
|
|
app.kubernetes.io/part-of: conformance
|
|
---
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: Role
|
|
metadata:
|
|
name: conformance-trigger
|
|
namespace: conformance
|
|
labels:
|
|
app.kubernetes.io/part-of: conformance
|
|
rules:
|
|
# Create the run, and watch it to completion.
|
|
- apiGroups: [batch]
|
|
resources: [jobs]
|
|
verbs: [create, get, list, watch]
|
|
# Read the Job's pod and its log — this is how results come back. Deliberately NOT
|
|
# pods/exec and NOT `kubectl cp`: the run tars its artifacts to stdout and CI decodes
|
|
# them from the log, so no execution channel into the runner pod is ever granted.
|
|
- apiGroups: [""]
|
|
resources: [pods, pods/log]
|
|
verbs: [get, list, watch]
|
|
# NOTHING ELSE. In particular NO delete on jobs (a trigger that can delete can hide a
|
|
# failed run), no secrets, no configmaps, no exec, no create on pods.
|
|
---
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: RoleBinding
|
|
metadata:
|
|
name: conformance-trigger
|
|
namespace: conformance
|
|
labels:
|
|
app.kubernetes.io/part-of: conformance
|
|
roleRef:
|
|
apiGroup: rbac.authorization.k8s.io
|
|
kind: Role
|
|
name: conformance-trigger
|
|
subjects:
|
|
- kind: ServiceAccount
|
|
name: conformance-trigger
|
|
namespace: conformance
|