From b7d6ba3d5f092cec66a9bc26a005ed4a2add03d1 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 6 Sep 2026 08:41:05 -0500 Subject: [PATCH] fix(kyverno): unblock CDI imports by SATISFYING the label requirement, not exempting it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Prod's UniFi VM has no disk. `DataVolume unifi/unifi-rootdisk` stuck in ImportScheduled for hours, because Kyverno denied CDI's own generated pod: validate.kyverno.svc-fail denied Pod/unifi/importer-prime- require-auth-mode-label (R-IDP-02) require-egress-class-label (R-NET-05) Both policies match any Pod in an `app-namespace: "true"` namespace with no exclusion, and CDI generates `importer-*` itself — it cannot carry repo labels. A COLD-START defect, not a prod quirk: dev's identical import Succeeded 9 days ago, before the policies enforced, so dev has never exercised it and would hit it on any rebuild. Fourth bug of this shape in two days. MECHANISM: a MUTATE ClusterPolicy that ADDS `egress-class: internet` + `auth-mode: none` (add-if-absent anchor) to Pods on CREATE in app namespaces WHEN the requesting subject is ServiceAccount `cdi/cdi-sa`. Both validate policies are untouched — no exclude, no exception, `background: true` preserved on them. Why mutate beats exempting: the requirement stays literally true ("every pod in an app namespace declares both labels") and the values are HONEST — the importer does pull over the default WAN path, and `internet` is the least-reach class (it does not grant the VPN path); a transient worker pod with no UI genuinely is `auth-mode: none`. THE MATCH IS IDENTITY, NOT A NAME. `subjects: ServiceAccount cdi/cdi-sa` is the AdmissionReview userInfo, taken from the real prod denial (`system:serviceaccount:cdi:cdi-sa`, name importer-prime-…, op CREATE). A name prefix is attacker-choosable; a requesting ServiceAccount is not — and the test proves it, with a fixture `importer-prime-forged-by-an-app` that copies both the name prefix AND CDI's labels and is still DENIED. Covers every CDI worker kind (importer/uploader/cloner) without enumerating names that change per release. Confirmed on prod that cdi-sa is the only cdi-namespace SA that can create pods. REJECTED, and the first one is a trap worth recording: * PolicyException — the CRD EXISTS (v2/v2beta1, from the pre-applied bundle) but all controllers run `--enablePolicyException=false` (verified live on prod: admission, background and reports controllers all carry the flag). An exception would apply cleanly through Flux and be SILENTLY IGNORED. Enabling the gate is a cluster-wide capability change for no gain over the mutate. * `exclude` scoped to cdi-sa — sound identity, worse consequences: any userInfo match forces `background: false`, which would drop background scanning and policy-report coverage of R-NET-05/R-IDP-02 for ALL EXISTING PODS to fix one pod class. It also leaves the importer with no labels at all — the unlabelled fall-through that require-egress-class-label's own header calls a fail-open leak. * name-prefix / CDI-label matching — attacker-choosable, and proven walkable. TWO-WAY PROOF, non-vacuous by construction — same resource, different requester, opposite outcomes (verified by me, not just reported): userinfo cdi-sa -> mutate pass, require-egress-class pass, require-auth-mode pass userinfo ordinary -> mutate SKIP, require-egress-class FAIL, require-auth-mode FAIL Kyverno L0 went 17 -> 29 tests passed, 0 failed. The mutate policy is deliberately kept OUT of the existing tests/kyverno-test.yaml: that suite declares no userinfo, and without userinfo the offline engine does not discriminate on subjects — it mutates everything, which would have silently turned its negative rows into passes. platform-config/kustomization.yaml is the live apply path for these ClusterPolicies; without it the policy never reaches a cluster. RESIDUAL TRADE-OFF, reversible by deleting one file: anything created by cdi-sa in an app namespace now has these labels supplied rather than being rejected for not declaring them — including a pod that is not a CDI worker. That identity already holds `pods create/delete` cluster-wide via ClusterRole `cdi`, so it is not a new capability; what changes is that for that one identity the declaration is assumed rather than demanded. restrict-host-namespaces, the namespace default-deny baseline and the CiliumNetworkPolicies all still apply. FIRST resource-mutating policy on this cluster: kyverno-resource-mutating-webhook-cfg holds 0 webhooks today; with autoUpdateWebhooks=true Kyverno registers mutate.kyverno.svc-fail for Pods within seconds of this applying. Prod's forceFailurePolicyIgnore=false makes it failurePolicy: Fail — same posture as the validating webhook already in that path. CDI retries importer creation on a loop, so the stuck import should proceed once this lands. The detached prime-be3f9d1c-… 40 GiB scratch PVC is separate operator cleanup; untouched, as is the DataVolume. VERIFIED: full CI gate PASS. Nothing applied to any cluster. Refs R-IDP-02, R-NET-05, R-PLAT-01. Co-Authored-By: Claude Opus 5 --- platform-config/kustomization.yaml | 8 ++ platform/kyverno/README.md | 23 ++- platform/kyverno/admission/kustomization.yaml | 4 + platform/kyverno/labels.md | 11 ++ .../policies/label-cdi-worker-pods.yaml | 136 ++++++++++++++++++ .../controller-worker-pods/kyverno-test.yaml | 91 ++++++++++++ .../patched/importer-mutated.yaml | 16 +++ .../resources/cdi-worker-pods.yaml | 53 +++++++ .../userinfo-cdi-sa.yaml | 21 +++ .../userinfo-ordinary.yaml | 16 +++ 10 files changed, 377 insertions(+), 2 deletions(-) create mode 100644 platform/kyverno/policies/label-cdi-worker-pods.yaml create mode 100644 platform/kyverno/tests/controller-worker-pods/kyverno-test.yaml create mode 100644 platform/kyverno/tests/controller-worker-pods/patched/importer-mutated.yaml create mode 100644 platform/kyverno/tests/controller-worker-pods/resources/cdi-worker-pods.yaml create mode 100644 platform/kyverno/tests/controller-worker-pods/userinfo-cdi-sa.yaml create mode 100644 platform/kyverno/tests/controller-worker-pods/userinfo-ordinary.yaml diff --git a/platform-config/kustomization.yaml b/platform-config/kustomization.yaml index cc4a893..4fb3962 100644 --- a/platform-config/kustomization.yaml +++ b/platform-config/kustomization.yaml @@ -25,6 +25,14 @@ resources: - ../platform/kyverno/policies/restrict-host-namespaces.yaml - ../platform/kyverno/policies/require-egress-class-label.yaml - ../platform/kyverno/policies/require-auth-mode-label.yaml + # COLD-START FIX (2026-09-06, prod): a MUTATE policy that must travel WITH the two label + # policies above. CDI generates its own short-lived worker pods (importer-*, uploader-*, + # cloner-*) inside app namespaces; nothing in git can label them, so both policies DENIED them + # and `DataVolume unifi/unifi-rootdisk` sat in ImportScheduled with no disk. This policy adds + # the two labels to pods created by ServiceAccount cdi/cdi-sa, so the requirement is SATISFIED + # rather than bypassed. Applying it here (not later) matters: if it lands after the validating + # policies, the next import is blocked in the gap. + - ../platform/kyverno/policies/label-cdi-worker-pods.yaml - ../platform/kyverno/admission/secret-provenance.yaml # cilium: L2 LB IP pool + announcement policy (cilium.io CRDs already registered by Cilium) - cilium/l2-ip-pool.yaml diff --git a/platform/kyverno/README.md b/platform/kyverno/README.md index 388d61f..7afbeaa 100644 --- a/platform/kyverno/README.md +++ b/platform/kyverno/README.md @@ -22,14 +22,19 @@ platform/kyverno/ │ ├── disallow-plaintext-secrets.yaml # R-SEC-01 (CI-only: sops block required) │ ├── restrict-host-namespaces.yaml # R-SEC-02 (+R-NET-01 bypass guard) │ ├── require-egress-class-label.yaml # R-NET-05 -│ └── require-auth-mode-label.yaml # R-IDP-02 +│ ├── require-auth-mode-label.yaml # R-IDP-02 +│ └── label-cdi-worker-pods.yaml # R-NET-05+R-IDP-02 (mutate: CDI's own worker pods) ├── servicedefinition/ # ServiceDefinition shape validation (kyverno-json) │ ├── validate-servicedefinition.yaml # R-GITOPS-02 │ ├── payloads/ (valid-* / invalid-*) # pass & fail fixtures │ └── README.md ├── tests/ # `kyverno test` suite — runnable L0 tests NOW, no cluster │ ├── kyverno-test.yaml # policies × fixtures × expected results -│ └── resources/ (good-* / bad-* fixtures + generated golden) +│ ├── resources/ (good-* / bad-* fixtures + generated golden) +│ └── controller-worker-pods/ # the CDI-worker mutation, proven BOTH ways +│ ├── kyverno-test.yaml # 2 Tests: cdi-sa requester vs ordinary requester +│ ├── userinfo-cdi-sa.yaml / userinfo-ordinary.yaml +│ └── resources/ + patched/ # real + FORGED importer pods; expected patch ├── admission/ # in-cluster enforcing bundle (scaffolding, T-2.7) │ ├── namespace.yaml · helmrelease.yaml · kustomization.yaml │ ├── secret-provenance.yaml # R-SEC-01 admission variant (provenance) @@ -47,6 +52,7 @@ platform/kyverno/ | `restrict-host-namespaces` | `disallow-host-namespaces`, `disallow-privileged-containers` | **R-SEC-02** (+R-NET-01) | docs/07 least reach | App pods can't use hostNetwork/PID/IPC or privileged (would bypass default-deny) | | `require-egress-class-label` | `require-egress-class` | **R-NET-05** | docs/03, docs/13 | App pods must declare `egress-class` ∈ {internet,vpn} | | `require-auth-mode-label` | `require-auth-mode` | **R-IDP-02** | docs/13, docs/15 | App pods must declare `auth-mode` ∈ {oidc,forward-auth,native,none} | +| `label-cdi-worker-pods` (mutate) | `add-worker-pod-labels` | **R-NET-05**, **R-IDP-02** | docs/03, docs/13 | CDI's **controller-generated** worker pods (`importer-*`/`uploader-*`/`cloner-*`, created by SA `cdi/cdi-sa`) get `egress-class: internet` + `auth-mode: none` **added**, so they satisfy the two policies above instead of being exempted from them. Matched on the **generating ServiceAccount**, never a pod-name prefix | | `validate-servicedefinition` (kyverno-json) | shape / dependencies / storage | **R-GITOPS-02** | docs/13 (+03,15) | ServiceDefinition is schema/invariant valid before render; public⇒not `none` auth | | `admission/secret-provenance` | `secrets-must-be-flux-managed` | **R-SEC-01** | docs/07 | **Admission-only** variant: app-ns Secrets must come from Flux/SOPS (see below) | @@ -90,6 +96,19 @@ self-contained — no cluster, no external state (reproducibility contract, R-TE - **Label keys are a proposed contract** (`labels.md`) shared with WS-NET / WS-SEC / WS-GITOPS. Values are spec-fixed; keys need owner sign-off (coordination-protocol §2). +- **Controller-generated pods are a cold-start hazard.** A policy that matches *any* Pod in an + app namespace also matches pods that no manifest in git authors. Prod hit this on 2026-09-06: + `require-auth-mode-label` + `require-egress-class-label` denied CDI's own `importer-prime-*` + pod, so `DataVolume unifi/unifi-rootdisk` stuck in `ImportScheduled` and the UniFi VM had no + disk. Fix: `policies/label-cdi-worker-pods.yaml` **mutates** those pods to carry truthful + labels (matched on the generating ServiceAccount `cdi/cdi-sa`), rather than excluding them. + `virt-launcher-*` pods need nothing — they inherit the VM pod-template labels. KubeVirt + hotplug (`hp-volume-*`) and export (`virt-export-*`) pods do **not** inherit them and are + unused here; extend the same policy if either is adopted. +- **`PolicyException` is NOT usable in this install**: the Kyverno admission/background/reports + controllers all run `--enablePolicyException=false` (chart default, verified on prod + 2026-09-06). The CRD exists (from the pre-applied CRD bundle) so a `PolicyException` would + APPLY cleanly and be silently ignored — a policy hole that looks like a fix and isn't. - **Secrets CI vs admission divergence** is deliberate and documented in `admission/README.md` and `policies/disallow-plaintext-secrets.yaml`. - The ServiceDefinition JSON Schema (WS-GITOPS T-0.3) is the complementary structural diff --git a/platform/kyverno/admission/kustomization.yaml b/platform/kyverno/admission/kustomization.yaml index c771fb7..4b20352 100644 --- a/platform/kyverno/admission/kustomization.yaml +++ b/platform/kyverno/admission/kustomization.yaml @@ -13,6 +13,10 @@ resources: - ../policies/restrict-host-namespaces.yaml - ../policies/require-egress-class-label.yaml - ../policies/require-auth-mode-label.yaml + # mutate: gives CDI's OWN generated worker pods (importer-*/uploader-*/cloner-*) the two + # labels the policies above require -- they are controller-generated, so nothing in git can + # label them. Ships with them, never separately. + - ../policies/label-cdi-worker-pods.yaml # --- secrets: the admission variant REPLACES the CI-only sops-block policy --- # (../policies/disallow-plaintext-secrets.yaml is CI-only; see secret-provenance.yaml) - secret-provenance.yaml diff --git a/platform/kyverno/labels.md b/platform/kyverno/labels.md index 5798c04..60bfcc6 100644 --- a/platform/kyverno/labels.md +++ b/platform/kyverno/labels.md @@ -15,6 +15,17 @@ Shared contract with WS-NET (egress rendering), WS-SEC (auth), and WS-GITOPS (re | `homelab.keithjackson.dev/baseline` | NetworkPolicy | `default-deny` | (convention) identifies the baseline NP | `require-default-deny-baseline` (generate), `validate-default-deny-shape` | | `homelab.keithjackson.dev/secret-managed-by` | Secret | `controller` | (convention) exempts controller-generated Secrets | `disallow-plaintext-secrets` (exclude), admission `secret-provenance` | +## Controller-generated pods (who sets the labels when git cannot) + +`egress-class` / `auth-mode` are normally projected onto pod templates by the renderer (below). +Some pods are generated by a **controller at runtime** and therefore have no manifest in git to +label — CDI's `importer-*` / `uploader-*` / `cloner-*` worker pods are the live example: both +label policies denied CDI's importer on prod (2026-09-06) and the UniFi VM got no disk. For those, +`policies/label-cdi-worker-pods.yaml` **adds** the labels at admission (`+(…)` add-if-absent), +matched on the **generating ServiceAccount** (`cdi/cdi-sa`) — never on a pod-name prefix or on +CDI's own labels, both of which any pod author in the namespace can copy. `virt-launcher-*` needs +nothing: KubeVirt propagates the VM pod-template labels onto it (verified on dev). + ## Rendering expectation (for WS-GITOPS) The ServiceDefinition (`docs/13`) declares intent — `egress:` and `auth.mode:` — at the diff --git a/platform/kyverno/policies/label-cdi-worker-pods.yaml b/platform/kyverno/policies/label-cdi-worker-pods.yaml new file mode 100644 index 0000000..709e5ca --- /dev/null +++ b/platform/kyverno/policies/label-cdi-worker-pods.yaml @@ -0,0 +1,136 @@ +# Serves: R-NET-05 (egress is a declared property) + R-IDP-02 (auth mode is declared) — +# this policy does NOT weaken either requirement. It makes the one class of pod that +# CANNOT carry the labels from git SATISFY them, instead of being exempted from them. +# +# THE DEFECT IT FIXES (observed on LIVE prod, 2026-09-05/06): +# `DataVolume unifi/unifi-rootdisk` sat in ImportScheduled for hours holding a detached +# 40Gi `prime-…` scratch PVC. `kubectl -n cdi logs deploy/cdi-deployment` showed: +# admission webhook "validate.kyverno.svc-fail" denied the request: resource +# Pod/unifi/importer-prime- was blocked due to the following policies +# require-auth-mode-label / require-egress-class-label +# CDI generates its own short-lived worker pods (importer-*, uploader-*, cloner-*/ +# clone-source-*) from inside the cdi-deployment controller. Those pods are not in git, +# so nothing can put the platform labels on them, and both label policies match ANY Pod +# in a namespace labelled homelab.keithjackson.dev/app-namespace=true. Result: the VM +# never gets a disk. This is a COLD-START defect, not a prod quirk — dev's identical +# import Succeeded before the policies were enforcing, so dev has never exercised it and +# would hit it on any rebuild. +# +# WHY MUTATE (and not an exception/exclusion): +# The honest reading of R-NET-05/R-IDP-02 is "every pod's egress path and auth posture is +# DECLARED". A CDI importer pod has truthful values for both: +# egress-class: internet — it pulls the image from the public internet over https +# (cloud-images.ubuntu.com / quay.io) and must take the +# default WAN path, which is exactly what Cilium's egress +# gateway does for `internet` (ADR-0013/ADR-0015). It is NOT +# a `vpn` workload. +# auth-mode: none — the pod serves no UI and terminates when the import ends; +# there is nothing to put an IdP in front of. docs/13 permits +# `none` for internal, non-public workloads (the +# ServiceDefinition policy forbids only public + none). +# So the requirement is SATISFIED rather than bypassed: the invariant "every pod in an app +# namespace carries both labels" stays literally true cluster-wide, both label policies +# stay untouched (no `exclude`, no PolicyException, background scanning still on), and the +# labels are functional — the pod is now selectable by the Cilium egress fabric instead of +# being an unlabelled pod that silently falls through to the default path. +# +# THE MATCH IS THE GENERATING ServiceAccount, NOT A NAME PREFIX: +# `subjects: ServiceAccount cdi/cdi-sa` is the identity in the AdmissionReview's userInfo — +# the controller that issued the CREATE. Verified on prod from the Kyverno admission log of +# the blocked request: +# user={"username":"system:serviceaccount:cdi:cdi-sa", ...} roles=["cdi:cdi-deployment"] +# name=importer-prime-be3f9d1c-… namespace=unifi operation=CREATE +# A pod NAME (`importer-*`) or CDI's own labels (`cdi.kubevirt.io`) are attacker-CHOOSABLE: +# anyone who can create a pod in the namespace can name it `importer-prime-x` and copy the +# labels, and would then inherit the exemption. The requesting ServiceAccount cannot be +# chosen by the pod author — only by someone who already holds cdi-sa's token, which already +# confers "create any pod in any namespace" cluster-wide. So this adds no reachable +# privilege. It is also VERSION-ROBUST: it covers every CDI worker kind (importer, uploader, +# cloner/clone-source) without enumerating names that change between CDI releases — +# confirmed on prod that cdi-sa is the ONLY cdi-namespace ServiceAccount that can create +# pods (`kubectl auth can-i create pods -n unifi --as …`: cdi-sa yes; cdi-apiserver no; +# cdi-uploadproxy no). +# +# DELIBERATELY NOT COVERED — KubeVirt's own generated pods: +# virt-launcher-* pods INHERIT the VM's pod-template labels, so a VM authored in this repo +# already passes (verified on dev: virt-launcher-unifi-22dg7 carries egress-class=internet, +# auth-mode=native from apps/vms/unifi/30-virtualmachine.yaml). Exempting +# kubevirt-controller would REMOVE that useful enforcement on VM authors, so it is not done. +# KubeVirt's hotplug (`hp-volume-*`) and export (`virt-export-*`) pods do NOT inherit those +# labels, but nothing in this repo uses volume hotplug or VirtualMachineExport; if either is +# adopted, extend this policy then — with the same ServiceAccount-scoped shape. +# +# OPERATOR NOTE — this is the cluster's FIRST resource-MUTATING policy. Verified on prod +# 2026-09-06 that `kyverno-resource-mutating-webhook-cfg` currently holds 0 webhooks; with the +# chart's `autoUpdateWebhooks=true` Kyverno registers `mutate.kyverno.svc-fail` for Pods within +# seconds of this policy being applied. Prod runs `forceFailurePolicyIgnore=false`, so that +# webhook is failurePolicy: Fail — the same posture as the validating webhook that is ALREADY in +# the path for these pods, so no new fail-open surface and no new blocking surface. CDI retries +# the importer pod on a loop, so the stuck import proceeds once the policy lands (the detached +# `prime-…` scratch PVC left behind by the blocked attempts is a separate operator cleanup). +# +# RESIDUAL TRADE-OFF (state it plainly, it is reversible by deleting this file): +# Anything created by `cdi:cdi-sa` in an app namespace now gets `egress-class: internet` + +# `auth-mode: none` supplied for it instead of being rejected for not declaring them. Whoever +# holds cdi-sa's token can therefore run a pod in an app namespace without declaring its egress +# class. That identity can already create arbitrary pods in every namespace (ClusterRole `cdi`: +# pods create/delete cluster-wide), so it is not a new capability — but the declaration is now +# assumed rather than demanded for that one identity. Nothing else changes: the pod is still +# subject to restrict-host-namespaces, the namespace default-deny baseline, and every other +# policy, and `internet` is the LEAST-reach of the two egress classes (it does not grant the VPN +# path). +# +# PROOF (offline, no cluster — platform/kyverno/tests/controller-worker-pods/): +# (a) a CDI importer pod created by cdi:cdi-sa is mutated and then PASSES both policies; +# (b) the SAME namespace's ordinary pod — including one that FORGES the importer name and +# CDI's labels — is NOT mutated and is still DENIED by both policies. +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: label-cdi-worker-pods + annotations: + policies.kyverno.io/title: Label CDI-generated worker pods + policies.kyverno.io/category: Network Isolation + policies.kyverno.io/severity: medium + homelab.keithjackson.dev/requirements: R-NET-05,R-IDP-02 + policies.kyverno.io/description: >- + Adds homelab.keithjackson.dev/egress-class=internet and auth-mode=none to the + short-lived worker Pods that the CDI controller (ServiceAccount cdi/cdi-sa) + generates in application namespaces, so they satisfy require-egress-class-label + and require-auth-mode-label instead of being exempted from them. +spec: + # No validationFailureAction: this is a MUTATE policy, there is nothing to fail. + # background MUST be false: `match.subjects` is userInfo, which exists only on an + # AdmissionReview. Kyverno rejects a userInfo match with background scanning enabled + # ("invalid variable used at path ... policy contains 'userInfo'"), and a background scan + # of an already-created pod has no requester to match anyway. + background: false + rules: + - name: add-worker-pod-labels + match: + # One filter entry: the resource conditions AND the subject must both match. + all: + - resources: + kinds: + - Pod + operations: + - CREATE + namespaceSelector: + matchLabels: + homelab.keithjackson.dev/app-namespace: "true" + subjects: + # The CDI controller (cdi-deployment) ServiceAccount — the identity that + # CREATEs importer-*/uploader-*/cloner-* pods. Not the pods' own SA. + - kind: ServiceAccount + namespace: cdi + name: cdi-sa + mutate: + patchStrategicMerge: + metadata: + labels: + # `+(key)` is Kyverno's ADD-IF-ABSENT anchor: an explicit value already on the + # pod is never overwritten (so a future CDI worker that legitimately needs + # egress-class: vpn can be given it, and virt-launcher-style inherited labels + # would survive). + +(homelab.keithjackson.dev/egress-class): internet + +(homelab.keithjackson.dev/auth-mode): none diff --git a/platform/kyverno/tests/controller-worker-pods/kyverno-test.yaml b/platform/kyverno/tests/controller-worker-pods/kyverno-test.yaml new file mode 100644 index 0000000..24de284 --- /dev/null +++ b/platform/kyverno/tests/controller-worker-pods/kyverno-test.yaml @@ -0,0 +1,91 @@ +# L0 policy-as-code suite for the CDI worker-pod label mutation — runs in CI WITHOUT a cluster +# (discovered recursively by the gate's `kyverno test platform/kyverno`; ci/l0.sh). +# +# WHY A SEPARATE SUITE from ../kyverno-test.yaml: a `kyverno test` manifest carries ONE +# `userinfo`, and the whole point here is that the SAME pods get different outcomes for +# different requesters. Two Test documents, two identities, one file: +# 1. cdi-controller-worker-pods-admitted (requester = cdi/cdi-sa) -> mutated, ADMITTED +# 2. cdi-worker-exemption-is-narrow (requester = ordinary user) -> not mutated, DENIED +# Keeping the mutate policy OUT of ../kyverno-test.yaml is deliberate: that suite declares no +# userinfo, and with no userinfo the offline engine does not discriminate on `subjects`, which +# would silently turn its negative (result: fail) rows into passes. +apiVersion: cli.kyverno.io/v1alpha1 +kind: Test +metadata: + name: cdi-controller-worker-pods-admitted +# Namespace labels for offline namespaceSelector resolution (shared with the main suite). +variables: ../values.yaml +userinfo: userinfo-cdi-sa.yaml +policies: + - ../../policies/label-cdi-worker-pods.yaml + - ../../policies/require-egress-class-label.yaml + - ../../policies/require-auth-mode-label.yaml +resources: + - ../resources/namespaces.yaml + - resources/cdi-worker-pods.yaml +results: + # ---- the mutation fires for the CDI controller and adds exactly the two labels ---- + - policy: label-cdi-worker-pods + rule: add-worker-pod-labels + kind: Pod + resources: + - importer-prime-be3f9d1c-c3ce-455a-a450-3d9e95c7baa2 + patchedResources: patched/importer-mutated.yaml + result: pass + # ---- R-NET-05 + R-IDP-02: the mutated CDI worker pod now SATISFIES both policies, so the + # import CDI could not start is admitted. (The engine validates the mutated resource.) + - policy: require-egress-class-label + rule: require-egress-class + kind: Pod + resources: + - importer-prime-be3f9d1c-c3ce-455a-a450-3d9e95c7baa2 + result: pass + - policy: require-auth-mode-label + rule: require-auth-mode + kind: Pod + resources: + - importer-prime-be3f9d1c-c3ce-455a-a450-3d9e95c7baa2 + result: pass +--- +apiVersion: cli.kyverno.io/v1alpha1 +kind: Test +metadata: + name: cdi-worker-exemption-is-narrow +variables: ../values.yaml +userinfo: userinfo-ordinary.yaml +policies: + - ../../policies/label-cdi-worker-pods.yaml + - ../../policies/require-egress-class-label.yaml + - ../../policies/require-auth-mode-label.yaml +resources: + - ../resources/namespaces.yaml + - resources/cdi-worker-pods.yaml +results: + # ---- the mutation does NOT fire for an ordinary requester, not even for a pod that forges + # the importer's name and CDI's labels ---- + - policy: label-cdi-worker-pods + rule: add-worker-pod-labels + kind: Pod + resources: + - importer-prime-be3f9d1c-c3ce-455a-a450-3d9e95c7baa2 + - importer-prime-forged-by-an-app + - plain-app-pod-no-labels + result: skip + # ---- so every unlabelled pod in the SAME app namespace is still DENIED (R-NET-05) ---- + - policy: require-egress-class-label + rule: require-egress-class + kind: Pod + resources: + - importer-prime-be3f9d1c-c3ce-455a-a450-3d9e95c7baa2 + - importer-prime-forged-by-an-app + - plain-app-pod-no-labels + result: fail + # ---- ... and by R-IDP-02 ---- + - policy: require-auth-mode-label + rule: require-auth-mode + kind: Pod + resources: + - importer-prime-be3f9d1c-c3ce-455a-a450-3d9e95c7baa2 + - importer-prime-forged-by-an-app + - plain-app-pod-no-labels + result: fail diff --git a/platform/kyverno/tests/controller-worker-pods/patched/importer-mutated.yaml b/platform/kyverno/tests/controller-worker-pods/patched/importer-mutated.yaml new file mode 100644 index 0000000..5f154db --- /dev/null +++ b/platform/kyverno/tests/controller-worker-pods/patched/importer-mutated.yaml @@ -0,0 +1,16 @@ +# Expected result of the mutation: CDI's own labels untouched, the two platform labels ADDED. +apiVersion: v1 +kind: Pod +metadata: + name: importer-prime-be3f9d1c-c3ce-455a-a450-3d9e95c7baa2 + namespace: app-ns + labels: + app: containerized-data-importer + cdi.kubevirt.io: importer + homelab.keithjackson.dev/egress-class: internet + homelab.keithjackson.dev/auth-mode: none +spec: + restartPolicy: OnFailure + containers: + - name: importer + image: quay.io/kubevirt/cdi-importer:v1.62.0 diff --git a/platform/kyverno/tests/controller-worker-pods/resources/cdi-worker-pods.yaml b/platform/kyverno/tests/controller-worker-pods/resources/cdi-worker-pods.yaml new file mode 100644 index 0000000..e9bbda8 --- /dev/null +++ b/platform/kyverno/tests/controller-worker-pods/resources/cdi-worker-pods.yaml @@ -0,0 +1,53 @@ +# Fixtures for the CDI-worker-pod label mutation (policies/label-cdi-worker-pods.yaml). +# +# All three pods live in `app-ns` (../resources/namespaces.yaml — carries +# homelab.keithjackson.dev/app-namespace: "true"), so all three are IN SCOPE of +# require-egress-class-label + require-auth-mode-label. What separates them is WHO creates +# them, which is the whole point of the mechanism: the two "forged" pods are byte-for-byte +# as CDI-looking as the real one. +# +# ---- (a) the REAL CDI worker pod: shape copied from the pod CDI actually tried to create on +# prod (name importer-prime-, CDI's own labels, no platform labels). It is +# created by ServiceAccount cdi/cdi-sa — see userinfo-cdi-sa.yaml. +apiVersion: v1 +kind: Pod +metadata: + name: importer-prime-be3f9d1c-c3ce-455a-a450-3d9e95c7baa2 + namespace: app-ns + labels: + app: containerized-data-importer + cdi.kubevirt.io: importer + # NOTE: no homelab.keithjackson.dev/* labels — CDI generates this pod itself, nothing in + # git can label it. This is exactly why the import was blocked. +spec: + restartPolicy: OnFailure + containers: + - name: importer + image: quay.io/kubevirt/cdi-importer:v1.62.0 +--- +# ---- (b) FORGED: an ordinary pod that copies the importer's NAME PREFIX and CDI's labels. +# Proves that a name-prefix or label-based exemption would have been walkable, and +# that this ServiceAccount-scoped one is not. +apiVersion: v1 +kind: Pod +metadata: + name: importer-prime-forged-by-an-app + namespace: app-ns + labels: + app: containerized-data-importer + cdi.kubevirt.io: importer +spec: + containers: + - name: app + image: registry.example/app:1.0.0 +--- +# ---- (c) a plain unlabelled application pod in the SAME namespace. Must still be DENIED. +apiVersion: v1 +kind: Pod +metadata: + name: plain-app-pod-no-labels + namespace: app-ns +spec: + containers: + - name: app + image: registry.example/app:1.0.0 diff --git a/platform/kyverno/tests/controller-worker-pods/userinfo-cdi-sa.yaml b/platform/kyverno/tests/controller-worker-pods/userinfo-cdi-sa.yaml new file mode 100644 index 0000000..9bafd69 --- /dev/null +++ b/platform/kyverno/tests/controller-worker-pods/userinfo-cdi-sa.yaml @@ -0,0 +1,21 @@ +# The requester identity Kyverno actually saw on prod when CDI created the importer pod +# (from the kyverno-admission-controller log of the blocked request, 2026-09-06): +# user={"username":"system:serviceaccount:cdi:cdi-sa","groups":["system:serviceaccounts", +# "system:serviceaccounts:cdi","system:authenticated"], ...} +# roles=["cdi:cdi-deployment"] clusterroles=["cdi", …] +# `kyverno test` feeds this to the engine as the AdmissionReview userInfo so +# `match.subjects` can be evaluated offline. +apiVersion: cli.kyverno.io/v1alpha1 +kind: UserInfo +metadata: + name: cdi-controller +clusterRoles: + - cdi +roles: + - cdi:cdi-deployment +userInfo: + username: system:serviceaccount:cdi:cdi-sa + groups: + - system:serviceaccounts + - system:serviceaccounts:cdi + - system:authenticated diff --git a/platform/kyverno/tests/controller-worker-pods/userinfo-ordinary.yaml b/platform/kyverno/tests/controller-worker-pods/userinfo-ordinary.yaml new file mode 100644 index 0000000..608e6c7 --- /dev/null +++ b/platform/kyverno/tests/controller-worker-pods/userinfo-ordinary.yaml @@ -0,0 +1,16 @@ +# An ORDINARY requester in the application namespace — the attacker/mistake case. Anything +# that is not cdi/cdi-sa: an app's own ServiceAccount, a human's kubectl, Flux. +# The mutation must NOT fire for this identity, so both label policies must still DENY — +# even for a pod that forged the importer's name and CDI's labels. +apiVersion: cli.kyverno.io/v1alpha1 +kind: UserInfo +metadata: + name: ordinary-namespace-user +clusterRoles: + - edit +userInfo: + username: system:serviceaccount:app-ns:default + groups: + - system:serviceaccounts + - system:serviceaccounts:app-ns + - system:authenticated