jakman85andClaude Opus 4.8
28a34c4477
Admission controller was crashlooping (9 restarts/4h, exit 255, NOT OOM: live usage ~65Mi/384Mi limit, 7m cpu). Root cause: chart-default 100m CPU request starved the process under node contention, so client-go calls hit context-deadline-exceeded, leader-election lease renewal timed out, leadership was lost and the webhook server closed. Raise the CPU request to 250m (no CPU limit, to avoid CFS throttling) with modest memory headroom (256Mi/512Mi), and raise dev admission replicas 1->2 so a single fail-closed pod restart no longer 502s cluster-wide admission-gated writes. Co-Authored-By: Claude Opus 4.8 <[email protected]>
91 lines
3.7 KiB
YAML
91 lines
3.7 KiB
YAML
# SCAFFOLDING for a later phase (Phase 2, task T-2.7) -- NOT applied in Phase 0
|
|
# (no cluster exists yet). Installs Kyverno so the SAME L0 policies run in-cluster
|
|
# as an admission controller (defense in depth per docs/07, ADR-0011). Flux
|
|
# HelmRelease skeleton; chart/app versions are pinned and bumped via the update path.
|
|
---
|
|
apiVersion: source.toolkit.fluxcd.io/v1
|
|
kind: HelmRepository
|
|
metadata:
|
|
name: kyverno
|
|
namespace: kyverno
|
|
spec:
|
|
interval: 24h
|
|
url: https://kyverno.github.io/kyverno/
|
|
---
|
|
apiVersion: helm.toolkit.fluxcd.io/v2
|
|
kind: HelmRelease
|
|
metadata:
|
|
name: kyverno
|
|
namespace: kyverno
|
|
spec:
|
|
interval: 1h
|
|
chart:
|
|
spec:
|
|
chart: kyverno
|
|
version: "3.3.x" # pin to the exact chart the operator vendors
|
|
sourceRef:
|
|
kind: HelmRepository
|
|
name: kyverno
|
|
namespace: kyverno
|
|
install:
|
|
remediation:
|
|
retries: 3
|
|
upgrade:
|
|
remediation:
|
|
retries: 3
|
|
values:
|
|
# The chart's cleanup Jobs default to bitnami/kubectl:1.30.2, which Bitnami PURGED from Docker Hub
|
|
# (2024 catalog change) → the jobs ImagePullBackOff. policyReportsCleanup runs as a Helm
|
|
# post-upgrade HOOK, so its failure STALLS every helm upgrade (5m timeout → HelmRelease not-Ready →
|
|
# platform Kustomization blocks). Point both cleanup images at Bitnami's relocated `bitnamilegacy`
|
|
# mirror — the identical image (has /bin/bash + awk/wc/xargs + kubectl the jobs need). Pinned tag.
|
|
policyReportsCleanup:
|
|
image:
|
|
repository: bitnamilegacy/kubectl
|
|
webhooksCleanup:
|
|
image:
|
|
repository: bitnamilegacy/kubectl
|
|
# Admission controller replicas — cluster-vars knob (prod 3 / dev 1). HA so a single pod
|
|
# restart cannot open an enforcement gap; dev drops to 1 to save RAM during bootstrap.
|
|
admissionController:
|
|
replicas: ${KYVERNO_ADMISSION_REPLICAS}
|
|
# Resources — chart default (limits.memory 384Mi / requests.cpu 100m,memory 128Mi) starves this
|
|
# latency-sensitive, fail-closed webhook: under node CPU contention the 100m request gave too few
|
|
# CFS shares, so client-go calls hit "context deadline exceeded" → leader-election lease renewal
|
|
# timed out → "leadership lost, stopped leading" → server closed (exit 255, NOT OOM: live usage
|
|
# was ~65Mi/7m). Raise the CPU REQUEST for scheduling headroom and keep NO CPU limit (a limit would
|
|
# reintroduce throttling and make lease-renewal timeouts worse). Memory bumped for headroom only.
|
|
container:
|
|
resources:
|
|
requests:
|
|
cpu: 250m
|
|
memory: 256Mi
|
|
limits:
|
|
memory: 512Mi
|
|
backgroundController:
|
|
enabled: true # needed for generate (default-deny baseline) + sync
|
|
reportsController:
|
|
enabled: true
|
|
cleanupController:
|
|
enabled: true
|
|
# config.webhooks is a MAP in the Kyverno chart (NOT a list — a list breaks the chart's
|
|
# webhook helper). namespaceSelector excludes system + platform namespaces at the webhook layer
|
|
# (belt-and-suspenders: the policies already scope to app-namespace=true). forceFailurePolicyIgnore
|
|
# is the chart's global fail-open switch — a cluster-vars knob: dev enables it (a cold/unhealthy
|
|
# Kyverno can't brick admission during bootstrap); prod disables it (fail-closed per
|
|
# coordination-protocol.md #8).
|
|
config:
|
|
forceFailurePolicyIgnore:
|
|
enabled: ${KYVERNO_FORCE_FAILURE_POLICY_IGNORE}
|
|
webhooks:
|
|
namespaceSelector:
|
|
matchExpressions:
|
|
- key: kubernetes.io/metadata.name
|
|
operator: NotIn
|
|
values:
|
|
- kube-system
|
|
- flux-system
|
|
- kyverno
|
|
- longhorn-system
|
|
- monitoring
|