Files
jakman85andClaude Opus 5 a7eadd4054 feat(pm-agent): the kubelet's pull credential, and the seventh field admission had not read
The registry had two blockers. #59 fixed the first (nodes resolved the hostname to the
non-hairpinned WAN and failed TLS against the router's CN=%D cert). This is the second:
the pull is unauthenticated, and this registry refuses unauthenticated clients outright —
measured 2026-10-05, its anonymous token carries access: null and every path 401s.

pm-agent-registry is a dockerconfigjson Secret (SOPS, placeholders) referenced by
imagePullSecrets on the dispatcher Deployment and on the committed worker Job template. It
holds the SAME Gitea token as pm-agent-credentials, because the two consumers need two
formats: the application reads a raw token from env, the kubelet reads a dockerconfigjson
before any container starts. Same one-value-in-two-files pattern as the Keycloak client
secret, documented in both .example files. The operator's part is the value plus one new
scope: read:package.

ADMISSION GAINED A SEVENTH RULE. imagePullSecrets is a pod-spec field no container
references, which is exactly why it is easy to leave unread — the same shape of omission
as initContainers, ephemeralContainers, env and envFrom before it. The kubelet reads the
named Secret, in this namespace, before anything runs. Rule 1's pattern now REQUIRES the
committed entry; rule 7 forbids any other and any second one. Both halves are needed: a
foreach over an absent list yields no result.

The leverage is small and the policy says so: the image is digest-pinned, so a different
pull credential cannot change WHAT runs. Pinned anyway — 'only the committed shape' is the
premise, and the rule costs four lines.

kyverno test: 33 passed, 0 failed (was 28). Three new fixtures, each a real path: the
committed entry plus a second one, a different Secret, and none at all.

The gate also caught me shipping the Secret in plaintext — I never ran sops -e -i on the
placeholder. Recorded in the handoff entry, because the file looked finished.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-30 01:09:39 -05:00
..

platform/kyverno — L0 policy-as-code gate & admission enforcer (T-0.5)

Kyverno is the platform's L0 static/policy gate (in CI, no cluster) and its in-cluster admission enforcer (ADR-0011, docs/14, docs/07). The policies here encode spec invariants as code: they run in CI as a merge gate today (Phase 0, no cluster) and the same policies are wired to run in-cluster as an admission controller later (Phase 2, admission/).

Enforcement complements, never replaces, behavior tests. A policy admitting a manifest is not proof the runtime behaves (R-TEST-01). The L2/L3 Chainsaw behavior + fault-injection tests remain mandatory. See admission/README.md.

Layout

platform/kyverno/
├── README.md                     # this file: policy→requirement map + CI entrypoint
├── labels.md                     # label/annotation convention (PROPOSED contract — escalated)
├── policies/                     # ClusterPolicies (authored once; CI + admission)
│   ├── require-default-deny-baseline.yaml   # R-NET-01 (generate baseline)
│   ├── validate-default-deny-shape.yaml     # R-NET-01 (reject malformed baseline)
│   ├── disallow-plaintext-secrets.yaml      # R-SEC-01 (CI-only: sops block required)
│   ├── restrict-host-namespaces.yaml        # R-SEC-02 (+R-NET-01 bypass guard)
│   ├── require-egress-class-label.yaml      # R-NET-05
│   ├── require-auth-mode-label.yaml         # R-IDP-02
│   └── label-cdi-worker-pods.yaml           # R-NET-05+R-IDP-02 (mutate: CDI's own worker pods)
├── servicedefinition/            # ServiceDefinition shape validation (kyverno-json)
│   ├── validate-servicedefinition.yaml      # R-GITOPS-02
│   ├── payloads/ (valid-* / invalid-*)      # pass & fail fixtures
│   └── README.md
├── tests/                        # `kyverno test` suite — runnable L0 tests NOW, no cluster
│   ├── kyverno-test.yaml                     # policies × fixtures × expected results
│   ├── resources/ (good-* / bad-* fixtures + generated golden)
│   └── controller-worker-pods/               # the CDI-worker mutation, proven BOTH ways
│       ├── kyverno-test.yaml                 #   2 Tests: cdi-sa requester vs ordinary requester
│       ├── userinfo-cdi-sa.yaml / userinfo-ordinary.yaml
│       └── resources/ + patched/             #   real + FORGED importer pods; expected patch
├── admission/                    # in-cluster enforcing bundle (scaffolding, T-2.7)
│   ├── namespace.yaml · helmrelease.yaml · kustomization.yaml
│   ├── secret-provenance.yaml                # R-SEC-01 admission variant (provenance)
│   └── README.md
└── scripts/run-l0.sh             # the single-command CI entrypoint for the gate

Policy → requirement map

Policy / artifact Rule(s) Requirement Spec source Behaves
require-default-deny-baseline generate-default-deny R-NET-01 docs/03 default-deny baseline Every app namespace auto-gets a deny-all (ingress+egress, DNS-only) NetworkPolicy
validate-default-deny-shape baseline-must-deny-both-directions R-NET-01 docs/03 A baseline that leaves a direction open is rejected
disallow-plaintext-secrets secrets-must-be-sops-encrypted R-SEC-01 docs/07, docs/05, ADR-0007 Committed Secrets must be SOPS-encrypted (sops block); plaintext rejected. CI-only
restrict-host-namespaces disallow-host-namespaces, disallow-privileged-containers R-SEC-02 (+R-NET-01) docs/07 least reach App pods can't use hostNetwork/PID/IPC or privileged (would bypass default-deny)
require-egress-class-label require-egress-class R-NET-05 docs/03, docs/13 App pods must declare egress-class ∈ {internet,vpn}
require-auth-mode-label require-auth-mode R-IDP-02 docs/13, docs/15 App pods must declare auth-mode ∈ {oidc,forward-auth,native,none}
label-cdi-worker-pods (mutate) add-worker-pod-labels R-NET-05, R-IDP-02 docs/03, docs/13 CDI's controller-generated worker pods (importer-*/uploader-*/cloner-*, created by SA cdi/cdi-sa) get egress-class: internet + auth-mode: none added, so they satisfy the two policies above instead of being exempted from them. Matched on the generating ServiceAccount, never a pod-name prefix
validate-servicedefinition (kyverno-json) shape / dependencies / storage R-GITOPS-02 docs/13 (+03,15) ServiceDefinition is schema/invariant valid before render; public⇒not none auth
admission/secret-provenance secrets-must-be-flux-managed R-SEC-01 docs/07 Admission-only variant: app-ns Secrets must come from Flux/SOPS (see below)

Coverage of the T-0.5 requirement set: R-TEST-01 (behavior-tagged, reproducible tests — this whole suite), R-NET-01, R-SEC-01, R-SEC-02, R-IDP-02, R-NET-05, R-GITOPS-02. Every policy carries its R-* IDs in the homelab.keithjackson.dev/requirements annotation so the coverage report can pick them up automatically.

CI entrypoint (the L0 gate)

CI (WS-GITOPS .gitea/workflows, then Gitea Actions) runs the single command:

platform/kyverno/scripts/run-l0.sh

which performs, with pinned tool versions (KYVERNO_VERSION, KYVERNO_JSON_VERSION):

# 1) ClusterPolicy tests — offline, no cluster
kyverno test platform/kyverno/tests

# 2) ServiceDefinition validation — offline, no cluster
kyverno-json scan --policy platform/kyverno/servicedefinition/validate-servicedefinition.yaml \
  --payload <each catalog/ ServiceDefinition and the valid-* fixtures>   # must pass
#   plus each invalid-* fixture must be REJECTED (script asserts non-zero exit)

kyverno test emits pass/fail per policy×rule×resource and supports JUnit output for the coverage/merge gate:

kyverno test platform/kyverno/tests --output junit > results/kyverno-l0.junit.xml

Exit non-zero fails the merge gate (docs/14 gating rules). Deterministic, hermetic, self-contained — no cluster, no external state (reproducibility contract, R-TEST-02).

Notes / handoffs

  • Label keys are a proposed contract (labels.md) shared with WS-NET / WS-SEC / WS-GITOPS. Values are spec-fixed; keys need owner sign-off (coordination-protocol §2).
  • Controller-generated pods are a cold-start hazard. A policy that matches any Pod in an app namespace also matches pods that no manifest in git authors. Prod hit this on 2026-09-06: require-auth-mode-label + require-egress-class-label denied CDI's own importer-prime-* pod, so DataVolume unifi/unifi-rootdisk stuck in ImportScheduled and the UniFi VM had no disk. Fix: policies/label-cdi-worker-pods.yaml mutates those pods to carry truthful labels (matched on the generating ServiceAccount cdi/cdi-sa), rather than excluding them. virt-launcher-* pods need nothing — they inherit the VM pod-template labels. KubeVirt hotplug (hp-volume-*) and export (virt-export-*) pods do not inherit them and are unused here; extend the same policy if either is adopted.
  • PolicyException is NOT usable in this install: the Kyverno admission/background/reports controllers all run --enablePolicyException=false (chart default, verified on prod 2026-09-06). The CRD exists (from the pre-applied CRD bundle) so a PolicyException would APPLY cleanly and be silently ignored — a policy hole that looks like a fix and isn't.
  • Secrets CI vs admission divergence is deliberate and documented in admission/README.md and policies/disallow-plaintext-secrets.yaml.
  • The ServiceDefinition JSON Schema (WS-GITOPS T-0.3) is the complementary structural contract; this Kyverno policy encodes the invariants from docs/13.
  • Tool availability: neither kyverno nor kyverno-json is installed on the WS-TEST workstation, so this suite was authored but not executed here; the CI agent (T-0.2/T-0.4 harness) runs scripts/run-l0.sh as the gate.