jakman85andClaude Opus 5
f178d85c71
Prod's UniFi VM has no disk. `DataVolume unifi/unifi-rootdisk` stuck in
ImportScheduled for hours, because Kyverno denied CDI's own generated pod:
validate.kyverno.svc-fail denied Pod/unifi/importer-prime-<uid>
require-auth-mode-label (R-IDP-02)
require-egress-class-label (R-NET-05)
Both policies match any Pod in an `app-namespace: "true"` namespace with no
exclusion, and CDI generates `importer-*` itself — it cannot carry repo labels.
A COLD-START defect, not a prod quirk: dev's identical import Succeeded 9 days
ago, before the policies enforced, so dev has never exercised it and would hit it
on any rebuild. Fourth bug of this shape in two days.
MECHANISM: a MUTATE ClusterPolicy that ADDS
`egress-class: internet` + `auth-mode: none` (add-if-absent anchor) to Pods on
CREATE in app namespaces WHEN the requesting subject is ServiceAccount
`cdi/cdi-sa`. Both validate policies are untouched — no exclude, no exception,
`background: true` preserved on them.
Why mutate beats exempting: the requirement stays literally true ("every pod in an
app namespace declares both labels") and the values are HONEST — the importer does
pull over the default WAN path, and `internet` is the least-reach class (it does
not grant the VPN path); a transient worker pod with no UI genuinely is
`auth-mode: none`.
THE MATCH IS IDENTITY, NOT A NAME. `subjects: ServiceAccount cdi/cdi-sa` is the
AdmissionReview userInfo, taken from the real prod denial
(`system:serviceaccount:cdi:cdi-sa`, name importer-prime-…, op CREATE). A name
prefix is attacker-choosable; a requesting ServiceAccount is not — and the test
proves it, with a fixture `importer-prime-forged-by-an-app` that copies both the
name prefix AND CDI's labels and is still DENIED. Covers every CDI worker kind
(importer/uploader/cloner) without enumerating names that change per release.
Confirmed on prod that cdi-sa is the only cdi-namespace SA that can create pods.
REJECTED, and the first one is a trap worth recording:
* PolicyException — the CRD EXISTS (v2/v2beta1, from the pre-applied bundle) but
all controllers run `--enablePolicyException=false` (verified live on prod:
admission, background and reports controllers all carry the flag). An exception
would apply cleanly through Flux and be SILENTLY IGNORED. Enabling the gate is a
cluster-wide capability change for no gain over the mutate.
* `exclude` scoped to cdi-sa — sound identity, worse consequences: any userInfo
match forces `background: false`, which would drop background scanning and
policy-report coverage of R-NET-05/R-IDP-02 for ALL EXISTING PODS to fix one pod
class. It also leaves the importer with no labels at all — the unlabelled
fall-through that require-egress-class-label's own header calls a fail-open leak.
* name-prefix / CDI-label matching — attacker-choosable, and proven walkable.
TWO-WAY PROOF, non-vacuous by construction — same resource, different requester,
opposite outcomes (verified by me, not just reported):
userinfo cdi-sa -> mutate pass, require-egress-class pass, require-auth-mode pass
userinfo ordinary -> mutate SKIP, require-egress-class FAIL, require-auth-mode FAIL
Kyverno L0 went 17 -> 29 tests passed, 0 failed. The mutate policy is deliberately
kept OUT of the existing tests/kyverno-test.yaml: that suite declares no userinfo,
and without userinfo the offline engine does not discriminate on subjects — it
mutates everything, which would have silently turned its negative rows into passes.
platform-config/kustomization.yaml is the live apply path for these ClusterPolicies;
without it the policy never reaches a cluster.
RESIDUAL TRADE-OFF, reversible by deleting one file: anything created by cdi-sa in
an app namespace now has these labels supplied rather than being rejected for not
declaring them — including a pod that is not a CDI worker. That identity already
holds `pods create/delete` cluster-wide via ClusterRole `cdi`, so it is not a new
capability; what changes is that for that one identity the declaration is assumed
rather than demanded. restrict-host-namespaces, the namespace default-deny baseline
and the CiliumNetworkPolicies all still apply.
FIRST resource-mutating policy on this cluster: kyverno-resource-mutating-webhook-cfg
holds 0 webhooks today; with autoUpdateWebhooks=true Kyverno registers
mutate.kyverno.svc-fail for Pods within seconds of this applying. Prod's
forceFailurePolicyIgnore=false makes it failurePolicy: Fail — same posture as the
validating webhook already in that path.
CDI retries importer creation on a loop, so the stuck import should proceed once
this lands. The detached prime-be3f9d1c-… 40 GiB scratch PVC is separate operator
cleanup; untouched, as is the DataVolume.
VERIFIED: full CI gate PASS. Nothing applied to any cluster.
Refs R-IDP-02, R-NET-05, R-PLAT-01.
Co-Authored-By: Claude Opus 5 <[email protected]>
39 lines
3.0 KiB
Markdown
39 lines
3.0 KiB
Markdown
# Platform label/annotation convention (RATIFIED)
|
|
|
|
These label keys are consumed by the Kyverno policies in this directory. The **values**
|
|
are fixed by the spec; the **keys** are the platform convention, now **ratified in the
|
|
spec** — `homelab.keithjackson.dev/{egress-class,auth-mode,app-namespace,baseline}` are
|
|
documented in `docs/03-networking-isolation-and-egress.md` ("Platform label-key
|
|
convention") as the contract that network policies, Kyverno rules, and renderers key off.
|
|
Shared contract with WS-NET (egress rendering), WS-SEC (auth), and WS-GITOPS (rendering).
|
|
|
|
| Label key | Applied to | Values | Fixed by | Used by policy |
|
|
|---|---|---|---|---|
|
|
| `homelab.keithjackson.dev/app-namespace` | Namespace | `"true"` | (convention) marks an application namespace / trust zone (docs/03) | all namespace-scoped policies |
|
|
| `homelab.keithjackson.dev/egress-class` | Pod (+ controller template) | `internet` \| `vpn` | docs/03, docs/13 | `require-egress-class-label` (R-NET-05); also the selector Cilium Egress Gateway steers on (ADR-0013) |
|
|
| `homelab.keithjackson.dev/auth-mode` | Pod (+ controller template) | `oidc` \| `forward-auth` \| `native` \| `none` | docs/13, docs/15 | `require-auth-mode-label` (R-IDP-02) |
|
|
| `homelab.keithjackson.dev/baseline` | NetworkPolicy | `default-deny` | (convention) identifies the baseline NP | `require-default-deny-baseline` (generate), `validate-default-deny-shape` |
|
|
| `homelab.keithjackson.dev/secret-managed-by` | Secret | `controller` | (convention) exempts controller-generated Secrets | `disallow-plaintext-secrets` (exclude), admission `secret-provenance` |
|
|
|
|
## Controller-generated pods (who sets the labels when git cannot)
|
|
|
|
`egress-class` / `auth-mode` are normally projected onto pod templates by the renderer (below).
|
|
Some pods are generated by a **controller at runtime** and therefore have no manifest in git to
|
|
label — CDI's `importer-*` / `uploader-*` / `cloner-*` worker pods are the live example: both
|
|
label policies denied CDI's importer on prod (2026-09-06) and the UniFi VM got no disk. For those,
|
|
`policies/label-cdi-worker-pods.yaml` **adds** the labels at admission (`+(…)` add-if-absent),
|
|
matched on the **generating ServiceAccount** (`cdi/cdi-sa`) — never on a pod-name prefix or on
|
|
CDI's own labels, both of which any pod author in the namespace can copy. `virt-launcher-*` needs
|
|
nothing: KubeVirt propagates the VM pod-template labels onto it (verified on dev).
|
|
|
|
## Rendering expectation (for WS-GITOPS)
|
|
|
|
The ServiceDefinition (`docs/13`) declares intent — `egress:` and `auth.mode:` — at the
|
|
service level. The renderer (T-0.3 / T-2.3 / T-6.1d) is expected to project those onto
|
|
the **pod template labels** above so that (a) Cilium selects pods by `egress-class` and
|
|
(b) these admission policies pass. Application namespaces are expected to be created
|
|
carrying `app-namespace: "true"` so the baseline is generated and the label/host
|
|
policies apply.
|
|
|
|
[`agents/coordination-protocol.md` §2]: ../../../home-network-k3s/agents/coordination-protocol.md
|