jakman85andClaude Opus 5
f178d85c71
Prod's UniFi VM has no disk. `DataVolume unifi/unifi-rootdisk` stuck in
ImportScheduled for hours, because Kyverno denied CDI's own generated pod:
validate.kyverno.svc-fail denied Pod/unifi/importer-prime-<uid>
require-auth-mode-label (R-IDP-02)
require-egress-class-label (R-NET-05)
Both policies match any Pod in an `app-namespace: "true"` namespace with no
exclusion, and CDI generates `importer-*` itself — it cannot carry repo labels.
A COLD-START defect, not a prod quirk: dev's identical import Succeeded 9 days
ago, before the policies enforced, so dev has never exercised it and would hit it
on any rebuild. Fourth bug of this shape in two days.
MECHANISM: a MUTATE ClusterPolicy that ADDS
`egress-class: internet` + `auth-mode: none` (add-if-absent anchor) to Pods on
CREATE in app namespaces WHEN the requesting subject is ServiceAccount
`cdi/cdi-sa`. Both validate policies are untouched — no exclude, no exception,
`background: true` preserved on them.
Why mutate beats exempting: the requirement stays literally true ("every pod in an
app namespace declares both labels") and the values are HONEST — the importer does
pull over the default WAN path, and `internet` is the least-reach class (it does
not grant the VPN path); a transient worker pod with no UI genuinely is
`auth-mode: none`.
THE MATCH IS IDENTITY, NOT A NAME. `subjects: ServiceAccount cdi/cdi-sa` is the
AdmissionReview userInfo, taken from the real prod denial
(`system:serviceaccount:cdi:cdi-sa`, name importer-prime-…, op CREATE). A name
prefix is attacker-choosable; a requesting ServiceAccount is not — and the test
proves it, with a fixture `importer-prime-forged-by-an-app` that copies both the
name prefix AND CDI's labels and is still DENIED. Covers every CDI worker kind
(importer/uploader/cloner) without enumerating names that change per release.
Confirmed on prod that cdi-sa is the only cdi-namespace SA that can create pods.
REJECTED, and the first one is a trap worth recording:
* PolicyException — the CRD EXISTS (v2/v2beta1, from the pre-applied bundle) but
all controllers run `--enablePolicyException=false` (verified live on prod:
admission, background and reports controllers all carry the flag). An exception
would apply cleanly through Flux and be SILENTLY IGNORED. Enabling the gate is a
cluster-wide capability change for no gain over the mutate.
* `exclude` scoped to cdi-sa — sound identity, worse consequences: any userInfo
match forces `background: false`, which would drop background scanning and
policy-report coverage of R-NET-05/R-IDP-02 for ALL EXISTING PODS to fix one pod
class. It also leaves the importer with no labels at all — the unlabelled
fall-through that require-egress-class-label's own header calls a fail-open leak.
* name-prefix / CDI-label matching — attacker-choosable, and proven walkable.
TWO-WAY PROOF, non-vacuous by construction — same resource, different requester,
opposite outcomes (verified by me, not just reported):
userinfo cdi-sa -> mutate pass, require-egress-class pass, require-auth-mode pass
userinfo ordinary -> mutate SKIP, require-egress-class FAIL, require-auth-mode FAIL
Kyverno L0 went 17 -> 29 tests passed, 0 failed. The mutate policy is deliberately
kept OUT of the existing tests/kyverno-test.yaml: that suite declares no userinfo,
and without userinfo the offline engine does not discriminate on subjects — it
mutates everything, which would have silently turned its negative rows into passes.
platform-config/kustomization.yaml is the live apply path for these ClusterPolicies;
without it the policy never reaches a cluster.
RESIDUAL TRADE-OFF, reversible by deleting one file: anything created by cdi-sa in
an app namespace now has these labels supplied rather than being rejected for not
declaring them — including a pod that is not a CDI worker. That identity already
holds `pods create/delete` cluster-wide via ClusterRole `cdi`, so it is not a new
capability; what changes is that for that one identity the declaration is assumed
rather than demanded. restrict-host-namespaces, the namespace default-deny baseline
and the CiliumNetworkPolicies all still apply.
FIRST resource-mutating policy on this cluster: kyverno-resource-mutating-webhook-cfg
holds 0 webhooks today; with autoUpdateWebhooks=true Kyverno registers
mutate.kyverno.svc-fail for Pods within seconds of this applying. Prod's
forceFailurePolicyIgnore=false makes it failurePolicy: Fail — same posture as the
validating webhook already in that path.
CDI retries importer creation on a loop, so the stuck import should proceed once
this lands. The detached prime-be3f9d1c-… 40 GiB scratch PVC is separate operator
cleanup; untouched, as is the DataVolume.
VERIFIED: full CI gate PASS. Nothing applied to any cluster.
Refs R-IDP-02, R-NET-05, R-PLAT-01.
Co-Authored-By: Claude Opus 5 <[email protected]>
92 lines
3.5 KiB
YAML
92 lines
3.5 KiB
YAML
# L0 policy-as-code suite for the CDI worker-pod label mutation — runs in CI WITHOUT a cluster
|
|
# (discovered recursively by the gate's `kyverno test platform/kyverno`; ci/l0.sh).
|
|
#
|
|
# WHY A SEPARATE SUITE from ../kyverno-test.yaml: a `kyverno test` manifest carries ONE
|
|
# `userinfo`, and the whole point here is that the SAME pods get different outcomes for
|
|
# different requesters. Two Test documents, two identities, one file:
|
|
# 1. cdi-controller-worker-pods-admitted (requester = cdi/cdi-sa) -> mutated, ADMITTED
|
|
# 2. cdi-worker-exemption-is-narrow (requester = ordinary user) -> not mutated, DENIED
|
|
# Keeping the mutate policy OUT of ../kyverno-test.yaml is deliberate: that suite declares no
|
|
# userinfo, and with no userinfo the offline engine does not discriminate on `subjects`, which
|
|
# would silently turn its negative (result: fail) rows into passes.
|
|
apiVersion: cli.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: cdi-controller-worker-pods-admitted
|
|
# Namespace labels for offline namespaceSelector resolution (shared with the main suite).
|
|
variables: ../values.yaml
|
|
userinfo: userinfo-cdi-sa.yaml
|
|
policies:
|
|
- ../../policies/label-cdi-worker-pods.yaml
|
|
- ../../policies/require-egress-class-label.yaml
|
|
- ../../policies/require-auth-mode-label.yaml
|
|
resources:
|
|
- ../resources/namespaces.yaml
|
|
- resources/cdi-worker-pods.yaml
|
|
results:
|
|
# ---- the mutation fires for the CDI controller and adds exactly the two labels ----
|
|
- policy: label-cdi-worker-pods
|
|
rule: add-worker-pod-labels
|
|
kind: Pod
|
|
resources:
|
|
- importer-prime-be3f9d1c-c3ce-455a-a450-3d9e95c7baa2
|
|
patchedResources: patched/importer-mutated.yaml
|
|
result: pass
|
|
# ---- R-NET-05 + R-IDP-02: the mutated CDI worker pod now SATISFIES both policies, so the
|
|
# import CDI could not start is admitted. (The engine validates the mutated resource.)
|
|
- policy: require-egress-class-label
|
|
rule: require-egress-class
|
|
kind: Pod
|
|
resources:
|
|
- importer-prime-be3f9d1c-c3ce-455a-a450-3d9e95c7baa2
|
|
result: pass
|
|
- policy: require-auth-mode-label
|
|
rule: require-auth-mode
|
|
kind: Pod
|
|
resources:
|
|
- importer-prime-be3f9d1c-c3ce-455a-a450-3d9e95c7baa2
|
|
result: pass
|
|
---
|
|
apiVersion: cli.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: cdi-worker-exemption-is-narrow
|
|
variables: ../values.yaml
|
|
userinfo: userinfo-ordinary.yaml
|
|
policies:
|
|
- ../../policies/label-cdi-worker-pods.yaml
|
|
- ../../policies/require-egress-class-label.yaml
|
|
- ../../policies/require-auth-mode-label.yaml
|
|
resources:
|
|
- ../resources/namespaces.yaml
|
|
- resources/cdi-worker-pods.yaml
|
|
results:
|
|
# ---- the mutation does NOT fire for an ordinary requester, not even for a pod that forges
|
|
# the importer's name and CDI's labels ----
|
|
- policy: label-cdi-worker-pods
|
|
rule: add-worker-pod-labels
|
|
kind: Pod
|
|
resources:
|
|
- importer-prime-be3f9d1c-c3ce-455a-a450-3d9e95c7baa2
|
|
- importer-prime-forged-by-an-app
|
|
- plain-app-pod-no-labels
|
|
result: skip
|
|
# ---- so every unlabelled pod in the SAME app namespace is still DENIED (R-NET-05) ----
|
|
- policy: require-egress-class-label
|
|
rule: require-egress-class
|
|
kind: Pod
|
|
resources:
|
|
- importer-prime-be3f9d1c-c3ce-455a-a450-3d9e95c7baa2
|
|
- importer-prime-forged-by-an-app
|
|
- plain-app-pod-no-labels
|
|
result: fail
|
|
# ---- ... and by R-IDP-02 ----
|
|
- policy: require-auth-mode-label
|
|
rule: require-auth-mode
|
|
kind: Pod
|
|
resources:
|
|
- importer-prime-be3f9d1c-c3ce-455a-a450-3d9e95c7baa2
|
|
- importer-prime-forged-by-an-app
|
|
- plain-app-pod-no-labels
|
|
result: fail
|