jakman85andClaude Opus 5
f178d85c71
Prod's UniFi VM has no disk. `DataVolume unifi/unifi-rootdisk` stuck in
ImportScheduled for hours, because Kyverno denied CDI's own generated pod:
validate.kyverno.svc-fail denied Pod/unifi/importer-prime-<uid>
require-auth-mode-label (R-IDP-02)
require-egress-class-label (R-NET-05)
Both policies match any Pod in an `app-namespace: "true"` namespace with no
exclusion, and CDI generates `importer-*` itself — it cannot carry repo labels.
A COLD-START defect, not a prod quirk: dev's identical import Succeeded 9 days
ago, before the policies enforced, so dev has never exercised it and would hit it
on any rebuild. Fourth bug of this shape in two days.
MECHANISM: a MUTATE ClusterPolicy that ADDS
`egress-class: internet` + `auth-mode: none` (add-if-absent anchor) to Pods on
CREATE in app namespaces WHEN the requesting subject is ServiceAccount
`cdi/cdi-sa`. Both validate policies are untouched — no exclude, no exception,
`background: true` preserved on them.
Why mutate beats exempting: the requirement stays literally true ("every pod in an
app namespace declares both labels") and the values are HONEST — the importer does
pull over the default WAN path, and `internet` is the least-reach class (it does
not grant the VPN path); a transient worker pod with no UI genuinely is
`auth-mode: none`.
THE MATCH IS IDENTITY, NOT A NAME. `subjects: ServiceAccount cdi/cdi-sa` is the
AdmissionReview userInfo, taken from the real prod denial
(`system:serviceaccount:cdi:cdi-sa`, name importer-prime-…, op CREATE). A name
prefix is attacker-choosable; a requesting ServiceAccount is not — and the test
proves it, with a fixture `importer-prime-forged-by-an-app` that copies both the
name prefix AND CDI's labels and is still DENIED. Covers every CDI worker kind
(importer/uploader/cloner) without enumerating names that change per release.
Confirmed on prod that cdi-sa is the only cdi-namespace SA that can create pods.
REJECTED, and the first one is a trap worth recording:
* PolicyException — the CRD EXISTS (v2/v2beta1, from the pre-applied bundle) but
all controllers run `--enablePolicyException=false` (verified live on prod:
admission, background and reports controllers all carry the flag). An exception
would apply cleanly through Flux and be SILENTLY IGNORED. Enabling the gate is a
cluster-wide capability change for no gain over the mutate.
* `exclude` scoped to cdi-sa — sound identity, worse consequences: any userInfo
match forces `background: false`, which would drop background scanning and
policy-report coverage of R-NET-05/R-IDP-02 for ALL EXISTING PODS to fix one pod
class. It also leaves the importer with no labels at all — the unlabelled
fall-through that require-egress-class-label's own header calls a fail-open leak.
* name-prefix / CDI-label matching — attacker-choosable, and proven walkable.
TWO-WAY PROOF, non-vacuous by construction — same resource, different requester,
opposite outcomes (verified by me, not just reported):
userinfo cdi-sa -> mutate pass, require-egress-class pass, require-auth-mode pass
userinfo ordinary -> mutate SKIP, require-egress-class FAIL, require-auth-mode FAIL
Kyverno L0 went 17 -> 29 tests passed, 0 failed. The mutate policy is deliberately
kept OUT of the existing tests/kyverno-test.yaml: that suite declares no userinfo,
and without userinfo the offline engine does not discriminate on subjects — it
mutates everything, which would have silently turned its negative rows into passes.
platform-config/kustomization.yaml is the live apply path for these ClusterPolicies;
without it the policy never reaches a cluster.
RESIDUAL TRADE-OFF, reversible by deleting one file: anything created by cdi-sa in
an app namespace now has these labels supplied rather than being rejected for not
declaring them — including a pod that is not a CDI worker. That identity already
holds `pods create/delete` cluster-wide via ClusterRole `cdi`, so it is not a new
capability; what changes is that for that one identity the declaration is assumed
rather than demanded. restrict-host-namespaces, the namespace default-deny baseline
and the CiliumNetworkPolicies all still apply.
FIRST resource-mutating policy on this cluster: kyverno-resource-mutating-webhook-cfg
holds 0 webhooks today; with autoUpdateWebhooks=true Kyverno registers
mutate.kyverno.svc-fail for Pods within seconds of this applying. Prod's
forceFailurePolicyIgnore=false makes it failurePolicy: Fail — same posture as the
validating webhook already in that path.
CDI retries importer creation on a loop, so the stuck import should proceed once
this lands. The detached prime-be3f9d1c-… 40 GiB scratch PVC is separate operator
cleanup; untouched, as is the DataVolume.
VERIFIED: full CI gate PASS. Nothing applied to any cluster.
Refs R-IDP-02, R-NET-05, R-PLAT-01.
Co-Authored-By: Claude Opus 5 <[email protected]>
3.0 KiB
Platform label/annotation convention (RATIFIED)
These label keys are consumed by the Kyverno policies in this directory. The values
are fixed by the spec; the keys are the platform convention, now ratified in the
spec — homelab.keithjackson.dev/{egress-class,auth-mode,app-namespace,baseline} are
documented in docs/03-networking-isolation-and-egress.md ("Platform label-key
convention") as the contract that network policies, Kyverno rules, and renderers key off.
Shared contract with WS-NET (egress rendering), WS-SEC (auth), and WS-GITOPS (rendering).
| Label key | Applied to | Values | Fixed by | Used by policy |
|---|---|---|---|---|
homelab.keithjackson.dev/app-namespace |
Namespace | "true" |
(convention) marks an application namespace / trust zone (docs/03) | all namespace-scoped policies |
homelab.keithjackson.dev/egress-class |
Pod (+ controller template) | internet | vpn |
docs/03, docs/13 | require-egress-class-label (R-NET-05); also the selector Cilium Egress Gateway steers on (ADR-0013) |
homelab.keithjackson.dev/auth-mode |
Pod (+ controller template) | oidc | forward-auth | native | none |
docs/13, docs/15 | require-auth-mode-label (R-IDP-02) |
homelab.keithjackson.dev/baseline |
NetworkPolicy | default-deny |
(convention) identifies the baseline NP | require-default-deny-baseline (generate), validate-default-deny-shape |
homelab.keithjackson.dev/secret-managed-by |
Secret | controller |
(convention) exempts controller-generated Secrets | disallow-plaintext-secrets (exclude), admission secret-provenance |
Controller-generated pods (who sets the labels when git cannot)
egress-class / auth-mode are normally projected onto pod templates by the renderer (below).
Some pods are generated by a controller at runtime and therefore have no manifest in git to
label — CDI's importer-* / uploader-* / cloner-* worker pods are the live example: both
label policies denied CDI's importer on prod (2026-09-06) and the UniFi VM got no disk. For those,
policies/label-cdi-worker-pods.yaml adds the labels at admission (+(…) add-if-absent),
matched on the generating ServiceAccount (cdi/cdi-sa) — never on a pod-name prefix or on
CDI's own labels, both of which any pod author in the namespace can copy. virt-launcher-* needs
nothing: KubeVirt propagates the VM pod-template labels onto it (verified on dev).
Rendering expectation (for WS-GITOPS)
The ServiceDefinition (docs/13) declares intent — egress: and auth.mode: — at the
service level. The renderer (T-0.3 / T-2.3 / T-6.1d) is expected to project those onto
the pod template labels above so that (a) Cilium selects pods by egress-class and
(b) these admission policies pass. Application namespaces are expected to be created
carrying app-namespace: "true" so the baseline is generated and the label/host
policies apply.