jakman85andClaude Opus 5
f178d85c71
Prod's UniFi VM has no disk. `DataVolume unifi/unifi-rootdisk` stuck in
ImportScheduled for hours, because Kyverno denied CDI's own generated pod:
validate.kyverno.svc-fail denied Pod/unifi/importer-prime-<uid>
require-auth-mode-label (R-IDP-02)
require-egress-class-label (R-NET-05)
Both policies match any Pod in an `app-namespace: "true"` namespace with no
exclusion, and CDI generates `importer-*` itself — it cannot carry repo labels.
A COLD-START defect, not a prod quirk: dev's identical import Succeeded 9 days
ago, before the policies enforced, so dev has never exercised it and would hit it
on any rebuild. Fourth bug of this shape in two days.
MECHANISM: a MUTATE ClusterPolicy that ADDS
`egress-class: internet` + `auth-mode: none` (add-if-absent anchor) to Pods on
CREATE in app namespaces WHEN the requesting subject is ServiceAccount
`cdi/cdi-sa`. Both validate policies are untouched — no exclude, no exception,
`background: true` preserved on them.
Why mutate beats exempting: the requirement stays literally true ("every pod in an
app namespace declares both labels") and the values are HONEST — the importer does
pull over the default WAN path, and `internet` is the least-reach class (it does
not grant the VPN path); a transient worker pod with no UI genuinely is
`auth-mode: none`.
THE MATCH IS IDENTITY, NOT A NAME. `subjects: ServiceAccount cdi/cdi-sa` is the
AdmissionReview userInfo, taken from the real prod denial
(`system:serviceaccount:cdi:cdi-sa`, name importer-prime-…, op CREATE). A name
prefix is attacker-choosable; a requesting ServiceAccount is not — and the test
proves it, with a fixture `importer-prime-forged-by-an-app` that copies both the
name prefix AND CDI's labels and is still DENIED. Covers every CDI worker kind
(importer/uploader/cloner) without enumerating names that change per release.
Confirmed on prod that cdi-sa is the only cdi-namespace SA that can create pods.
REJECTED, and the first one is a trap worth recording:
* PolicyException — the CRD EXISTS (v2/v2beta1, from the pre-applied bundle) but
all controllers run `--enablePolicyException=false` (verified live on prod:
admission, background and reports controllers all carry the flag). An exception
would apply cleanly through Flux and be SILENTLY IGNORED. Enabling the gate is a
cluster-wide capability change for no gain over the mutate.
* `exclude` scoped to cdi-sa — sound identity, worse consequences: any userInfo
match forces `background: false`, which would drop background scanning and
policy-report coverage of R-NET-05/R-IDP-02 for ALL EXISTING PODS to fix one pod
class. It also leaves the importer with no labels at all — the unlabelled
fall-through that require-egress-class-label's own header calls a fail-open leak.
* name-prefix / CDI-label matching — attacker-choosable, and proven walkable.
TWO-WAY PROOF, non-vacuous by construction — same resource, different requester,
opposite outcomes (verified by me, not just reported):
userinfo cdi-sa -> mutate pass, require-egress-class pass, require-auth-mode pass
userinfo ordinary -> mutate SKIP, require-egress-class FAIL, require-auth-mode FAIL
Kyverno L0 went 17 -> 29 tests passed, 0 failed. The mutate policy is deliberately
kept OUT of the existing tests/kyverno-test.yaml: that suite declares no userinfo,
and without userinfo the offline engine does not discriminate on subjects — it
mutates everything, which would have silently turned its negative rows into passes.
platform-config/kustomization.yaml is the live apply path for these ClusterPolicies;
without it the policy never reaches a cluster.
RESIDUAL TRADE-OFF, reversible by deleting one file: anything created by cdi-sa in
an app namespace now has these labels supplied rather than being rejected for not
declaring them — including a pod that is not a CDI worker. That identity already
holds `pods create/delete` cluster-wide via ClusterRole `cdi`, so it is not a new
capability; what changes is that for that one identity the declaration is assumed
rather than demanded. restrict-host-namespaces, the namespace default-deny baseline
and the CiliumNetworkPolicies all still apply.
FIRST resource-mutating policy on this cluster: kyverno-resource-mutating-webhook-cfg
holds 0 webhooks today; with autoUpdateWebhooks=true Kyverno registers
mutate.kyverno.svc-fail for Pods within seconds of this applying. Prod's
forceFailurePolicyIgnore=false makes it failurePolicy: Fail — same posture as the
validating webhook already in that path.
CDI retries importer creation on a loop, so the stuck import should proceed once
this lands. The detached prime-be3f9d1c-… 40 GiB scratch PVC is separate operator
cleanup; untouched, as is the DataVolume.
VERIFIED: full CI gate PASS. Nothing applied to any cluster.
Refs R-IDP-02, R-NET-05, R-PLAT-01.
Co-Authored-By: Claude Opus 5 <[email protected]>
119 lines
8.0 KiB
Markdown
119 lines
8.0 KiB
Markdown
# platform/kyverno — L0 policy-as-code gate & admission enforcer (T-0.5)
|
||
|
||
Kyverno is the platform's **L0 static/policy gate** (in CI, no cluster) **and** its
|
||
in-cluster **admission enforcer** (ADR-0011, `docs/14`, `docs/07`). The policies here
|
||
encode spec invariants as code: they run in CI as a merge gate **today** (Phase 0, no
|
||
cluster) and the same policies are wired to run in-cluster as an admission controller
|
||
later (Phase 2, `admission/`).
|
||
|
||
> **Enforcement complements, never replaces, behavior tests.** A policy admitting a
|
||
> manifest is not proof the runtime behaves (R-TEST-01). The L2/L3 **Chainsaw**
|
||
> behavior + fault-injection tests remain mandatory. See `admission/README.md`.
|
||
|
||
## Layout
|
||
|
||
```
|
||
platform/kyverno/
|
||
├── README.md # this file: policy→requirement map + CI entrypoint
|
||
├── labels.md # label/annotation convention (PROPOSED contract — escalated)
|
||
├── policies/ # ClusterPolicies (authored once; CI + admission)
|
||
│ ├── require-default-deny-baseline.yaml # R-NET-01 (generate baseline)
|
||
│ ├── validate-default-deny-shape.yaml # R-NET-01 (reject malformed baseline)
|
||
│ ├── disallow-plaintext-secrets.yaml # R-SEC-01 (CI-only: sops block required)
|
||
│ ├── restrict-host-namespaces.yaml # R-SEC-02 (+R-NET-01 bypass guard)
|
||
│ ├── require-egress-class-label.yaml # R-NET-05
|
||
│ ├── require-auth-mode-label.yaml # R-IDP-02
|
||
│ └── label-cdi-worker-pods.yaml # R-NET-05+R-IDP-02 (mutate: CDI's own worker pods)
|
||
├── servicedefinition/ # ServiceDefinition shape validation (kyverno-json)
|
||
│ ├── validate-servicedefinition.yaml # R-GITOPS-02
|
||
│ ├── payloads/ (valid-* / invalid-*) # pass & fail fixtures
|
||
│ └── README.md
|
||
├── tests/ # `kyverno test` suite — runnable L0 tests NOW, no cluster
|
||
│ ├── kyverno-test.yaml # policies × fixtures × expected results
|
||
│ ├── resources/ (good-* / bad-* fixtures + generated golden)
|
||
│ └── controller-worker-pods/ # the CDI-worker mutation, proven BOTH ways
|
||
│ ├── kyverno-test.yaml # 2 Tests: cdi-sa requester vs ordinary requester
|
||
│ ├── userinfo-cdi-sa.yaml / userinfo-ordinary.yaml
|
||
│ └── resources/ + patched/ # real + FORGED importer pods; expected patch
|
||
├── admission/ # in-cluster enforcing bundle (scaffolding, T-2.7)
|
||
│ ├── namespace.yaml · helmrelease.yaml · kustomization.yaml
|
||
│ ├── secret-provenance.yaml # R-SEC-01 admission variant (provenance)
|
||
│ └── README.md
|
||
└── scripts/run-l0.sh # the single-command CI entrypoint for the gate
|
||
```
|
||
|
||
## Policy → requirement map
|
||
|
||
| Policy / artifact | Rule(s) | Requirement | Spec source | Behaves |
|
||
|---|---|---|---|---|
|
||
| `require-default-deny-baseline` | `generate-default-deny` | **R-NET-01** | docs/03 default-deny baseline | Every app namespace auto-gets a deny-all (ingress+egress, DNS-only) NetworkPolicy |
|
||
| `validate-default-deny-shape` | `baseline-must-deny-both-directions` | **R-NET-01** | docs/03 | A baseline that leaves a direction open is rejected |
|
||
| `disallow-plaintext-secrets` | `secrets-must-be-sops-encrypted` | **R-SEC-01** | docs/07, docs/05, ADR-0007 | Committed Secrets must be SOPS-encrypted (`sops` block); plaintext rejected. **CI-only** |
|
||
| `restrict-host-namespaces` | `disallow-host-namespaces`, `disallow-privileged-containers` | **R-SEC-02** (+R-NET-01) | docs/07 least reach | App pods can't use hostNetwork/PID/IPC or privileged (would bypass default-deny) |
|
||
| `require-egress-class-label` | `require-egress-class` | **R-NET-05** | docs/03, docs/13 | App pods must declare `egress-class` ∈ {internet,vpn} |
|
||
| `require-auth-mode-label` | `require-auth-mode` | **R-IDP-02** | docs/13, docs/15 | App pods must declare `auth-mode` ∈ {oidc,forward-auth,native,none} |
|
||
| `label-cdi-worker-pods` (mutate) | `add-worker-pod-labels` | **R-NET-05**, **R-IDP-02** | docs/03, docs/13 | CDI's **controller-generated** worker pods (`importer-*`/`uploader-*`/`cloner-*`, created by SA `cdi/cdi-sa`) get `egress-class: internet` + `auth-mode: none` **added**, so they satisfy the two policies above instead of being exempted from them. Matched on the **generating ServiceAccount**, never a pod-name prefix |
|
||
| `validate-servicedefinition` (kyverno-json) | shape / dependencies / storage | **R-GITOPS-02** | docs/13 (+03,15) | ServiceDefinition is schema/invariant valid before render; public⇒not `none` auth |
|
||
| `admission/secret-provenance` | `secrets-must-be-flux-managed` | **R-SEC-01** | docs/07 | **Admission-only** variant: app-ns Secrets must come from Flux/SOPS (see below) |
|
||
|
||
Coverage of the T-0.5 requirement set: **R-TEST-01** (behavior-tagged, reproducible
|
||
tests — this whole suite), **R-NET-01**, **R-SEC-01**, **R-SEC-02**, **R-IDP-02**,
|
||
**R-NET-05**, **R-GITOPS-02**. Every policy carries its `R-*` IDs in the
|
||
`homelab.keithjackson.dev/requirements` annotation so the coverage report can pick
|
||
them up automatically.
|
||
|
||
## CI entrypoint (the L0 gate)
|
||
|
||
CI (WS-GITOPS `.gitea/workflows`, then Gitea Actions) runs the **single command**:
|
||
|
||
```bash
|
||
platform/kyverno/scripts/run-l0.sh
|
||
```
|
||
|
||
which performs, with pinned tool versions (`KYVERNO_VERSION`, `KYVERNO_JSON_VERSION`):
|
||
|
||
```bash
|
||
# 1) ClusterPolicy tests — offline, no cluster
|
||
kyverno test platform/kyverno/tests
|
||
|
||
# 2) ServiceDefinition validation — offline, no cluster
|
||
kyverno-json scan --policy platform/kyverno/servicedefinition/validate-servicedefinition.yaml \
|
||
--payload <each catalog/ ServiceDefinition and the valid-* fixtures> # must pass
|
||
# plus each invalid-* fixture must be REJECTED (script asserts non-zero exit)
|
||
```
|
||
|
||
`kyverno test` emits pass/fail per policy×rule×resource and supports JUnit output for
|
||
the coverage/merge gate:
|
||
|
||
```bash
|
||
kyverno test platform/kyverno/tests --output junit > results/kyverno-l0.junit.xml
|
||
```
|
||
|
||
Exit non-zero fails the merge gate (`docs/14` gating rules). Deterministic, hermetic,
|
||
self-contained — no cluster, no external state (reproducibility contract, R-TEST-02).
|
||
|
||
## Notes / handoffs
|
||
|
||
- **Label keys are a proposed contract** (`labels.md`) shared with WS-NET / WS-SEC /
|
||
WS-GITOPS. Values are spec-fixed; keys need owner sign-off (coordination-protocol §2).
|
||
- **Controller-generated pods are a cold-start hazard.** A policy that matches *any* Pod in an
|
||
app namespace also matches pods that no manifest in git authors. Prod hit this on 2026-09-06:
|
||
`require-auth-mode-label` + `require-egress-class-label` denied CDI's own `importer-prime-*`
|
||
pod, so `DataVolume unifi/unifi-rootdisk` stuck in `ImportScheduled` and the UniFi VM had no
|
||
disk. Fix: `policies/label-cdi-worker-pods.yaml` **mutates** those pods to carry truthful
|
||
labels (matched on the generating ServiceAccount `cdi/cdi-sa`), rather than excluding them.
|
||
`virt-launcher-*` pods need nothing — they inherit the VM pod-template labels. KubeVirt
|
||
hotplug (`hp-volume-*`) and export (`virt-export-*`) pods do **not** inherit them and are
|
||
unused here; extend the same policy if either is adopted.
|
||
- **`PolicyException` is NOT usable in this install**: the Kyverno admission/background/reports
|
||
controllers all run `--enablePolicyException=false` (chart default, verified on prod
|
||
2026-09-06). The CRD exists (from the pre-applied CRD bundle) so a `PolicyException` would
|
||
APPLY cleanly and be silently ignored — a policy hole that looks like a fix and isn't.
|
||
- **Secrets CI vs admission divergence** is deliberate and documented in
|
||
`admission/README.md` and `policies/disallow-plaintext-secrets.yaml`.
|
||
- The ServiceDefinition JSON Schema (WS-GITOPS T-0.3) is the complementary structural
|
||
contract; this Kyverno policy encodes the *invariants* from `docs/13`.
|
||
- Tool availability: neither `kyverno` nor `kyverno-json` is installed on the WS-TEST
|
||
workstation, so this suite was authored but **not executed here**; the CI agent
|
||
(T-0.2/T-0.4 harness) runs `scripts/run-l0.sh` as the gate.
|