fix(storage): bound the caches filling each node's /var
The act-runner's Docker cache was an unbounded emptyDir on the 11.6 GiB /var partition and evicted k3s-prod-2 on 2026-11-12. It moves to a generic ephemeral volume on longhorn-scratch (off /var, pod-lifetime, single replica), with a threshold-triggered docker prune and Velero opt-out. vuln-scanner's cache sizeLimit and ephemeral-storage limit are cut to fit the partition, and it prefers not to share a node with buildkitd. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5.5 committed
1 parent
93c4994bf1
commit
3ae7fb74f8
3 files changed
+253
-5
No files matched your search
@@ -135,6 +135,13 @@ spec:
|
||||
# Bump when act-runner-config changes without any other pod-template change, so the
|
||||
# Deployment rolls and act_runner re-reads its config (see the ConfigMap header).
|
||||
homelab.keithjackson.dev/act-runner-config-revision: "1" # capacity 2
|
||||
# Velero's daily schedule is opt-OUT (defaultVolumesToFsBackup: true) and its volume policy
|
||||
# skips only emptyDir and NFS (platform/velero/volume-policies.yaml). dind-storage is now a
|
||||
# Longhorn-backed generic ephemeral volume (see `volumes:`), which that policy does NOT
|
||||
# match, so without this the nightly backup would push the whole Docker image cache to R2
|
||||
# and race this pod's replacement (the PartiallyFailed failure that file documents). It is a
|
||||
# disposable cache: nothing in it is state.
|
||||
backup.velero.io/backup-volumes-excludes: dind-storage
|
||||
spec:
|
||||
# CI holds no cluster credentials — do not mount a SA token into the runner or
|
||||
# (via the shared netns) the job containers (R-SEC-02, docs/14).
|
||||
@@ -197,12 +204,18 @@ spec:
|
||||
# memory request 1Gi / limit 4Gi: two jobs at today's peak is ~0.9Gi. An OOM kill here
|
||||
# takes down dockerd and with it BOTH running jobs, so the limit carries ~4x headroom
|
||||
# over that for the parallelised gate. Nodes have ~31Gi allocatable.
|
||||
# ephemeral-storage: /var/lib/docker (images, job containers, their logs) is on the
|
||||
# dind-storage PVC, NOT the node's /var, so this bounds only dockerd's own rootfs
|
||||
# writes and its container log. Before 2026-11-14 the image cache was an unbounded
|
||||
# emptyDir on /var (see `volumes:`).
|
||||
resources:
|
||||
requests:
|
||||
cpu: "2"
|
||||
memory: 1Gi
|
||||
ephemeral-storage: 256Mi
|
||||
limits:
|
||||
memory: 4Gi
|
||||
ephemeral-storage: 2Gi
|
||||
startupProbe:
|
||||
exec:
|
||||
command:
|
||||
@@ -266,16 +279,132 @@ spec:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
ephemeral-storage: 128Mi
|
||||
limits:
|
||||
memory: 1Gi
|
||||
ephemeral-storage: 1Gi
|
||||
# ---- docker-prune: keeps the dind image cache bounded ----------------------------------
|
||||
#
|
||||
# dockerd never garbage-collects images. Every new renovate/toolbox digest (~2 GB each) used
|
||||
# to stack up in /var/lib/docker until the node's /var filled: on 2026-11-12 that tipped
|
||||
# k3s-prod-2 into DiskPressure and the act-runner pod was evicted 63 times. Moving the cache
|
||||
# to its own volume (see `volumes:`) stops it hurting the node; this stops it filling that
|
||||
# volume and failing jobs with ENOSPC instead.
|
||||
#
|
||||
# Every 10 minutes: if the volume is at or above 45% used, `docker system prune --all
|
||||
# --force` removes every image, stopped container, unused network and build cache entry NOT
|
||||
# in use by a container. Images a running job uses are never removed. 45% (not 85%) so the
|
||||
# volume stays above the 50%-free gate of PersistentVolumeFillingUpPredicted
|
||||
# (platform/monitoring/pvc-capacity-alerts.yaml) and a normal refill never pages anyone.
|
||||
# Threshold-triggered, not age-based: `--filter until=` on images is image CREATION time, so
|
||||
# an age-based prune would delete the long-lived job images (catthehacker, the toolbox) between
|
||||
# every pair of jobs and force a re-pull each time.
|
||||
#
|
||||
# Accepted race: a prune landing in the milliseconds between a job's image pull and its
|
||||
# container create removes that image and fails that one job. It needs the volume over the
|
||||
# threshold AT that instant; a re-run passes.
|
||||
#
|
||||
# NO DOLLAR SIGNS, deliberately: clusters/base/apps-wave-1.yaml runs Flux postBuild
|
||||
# substitution over this whole file, comments included, which rewrites shell variables and
|
||||
# command substitutions (conformance/tests/l0-static/flux-postbuild-substitution). The
|
||||
# percentage test is therefore
|
||||
# a regex over `df -P` output rather than arithmetic on a captured value.
|
||||
- name: docker-prune
|
||||
image: docker:29-cli
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- |
|
||||
trap 'exit 0' TERM INT
|
||||
echo "docker-prune: started; prunes when /var/lib/docker is at or above 45% used"
|
||||
while true; do
|
||||
if df -P /var/lib/docker | grep -qE ' (4[5-9]|[5-9][0-9]|100)% '; then
|
||||
echo "docker-prune: threshold reached, before:"
|
||||
df -P /var/lib/docker | tail -n 1
|
||||
docker system prune --all --force || echo "docker-prune: prune failed; retrying next cycle"
|
||||
echo "docker-prune: after:"
|
||||
df -P /var/lib/docker | tail -n 1
|
||||
fi
|
||||
sleep 600 &
|
||||
wait
|
||||
done
|
||||
env:
|
||||
- name: DOCKER_HOST
|
||||
value: tcp://localhost:2376
|
||||
- name: DOCKER_CERT_PATH
|
||||
value: /certs/client
|
||||
- name: DOCKER_TLS_VERIFY
|
||||
value: "1"
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
volumeMounts:
|
||||
- name: certs
|
||||
mountPath: /certs
|
||||
readOnly: true
|
||||
# Mounted only so `df` can read the volume's usage; the prune itself goes through the
|
||||
# daemon's API.
|
||||
- name: dind-storage
|
||||
mountPath: /var/lib/docker
|
||||
readOnly: true
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 32Mi
|
||||
ephemeral-storage: 16Mi
|
||||
limits:
|
||||
memory: 128Mi
|
||||
ephemeral-storage: 128Mi
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: act-runner-data
|
||||
- name: certs
|
||||
emptyDir: {}
|
||||
emptyDir:
|
||||
sizeLimit: 16Mi # a handful of PEM files
|
||||
- name: config
|
||||
configMap:
|
||||
name: act-runner-config
|
||||
# The Docker daemon's data root: every job image and job container. OFF the node's /var.
|
||||
#
|
||||
# WHY NOT an emptyDir (what this was until 2026-11-14). kubelet emptyDirs live on /var, a
|
||||
# separate 11.8 GiB partition on every node, and containerd's image store is NOT there
|
||||
# (k3s --data-dir is /data/k3s, infra/ansible/inventory/group_vars/k3s_prod/vars.yml), so
|
||||
# kubelet image GC never frees /var. Measured 2026-11-14: this emptyDir held 3.75 GiB on
|
||||
# k3s-prod-2 (renovate 1.99 GB + toolbox 2.04 GB) with ~1.9 GiB of /var left; the
|
||||
# 2026-11-12 eviction storm was it tipping the node into DiskPressure. A sizeLimit alone
|
||||
# would only trade that for this pod evicting itself, and /var is too small to give the
|
||||
# cache the room it needs next to vuln-scanner's and buildkit's.
|
||||
#
|
||||
# WHY a GENERIC EPHEMERAL volume on `longhorn-scratch` and not a standalone PVC.
|
||||
# * Off /var: Longhorn's dataPath is /data/longhorn on the ~424 GB partition (~170 GB free
|
||||
# per node, measured 2026-11-14).
|
||||
# * Single replica (`longhorn-scratch`, dataLocality best-effort): a disposable cache gains
|
||||
# nothing from 3x replication, and replica writes of multi-GB image pulls would land on
|
||||
# the device etcd's WAL shares (docs/04 accepted risk). Same reasoning as CDI's scratch,
|
||||
# which is what that class exists for.
|
||||
# * Pod-lifetime, not persistent: the claim is created with the pod and deleted with it.
|
||||
# That keeps this out of R-STATE-02 ("persistent storage MUST be replicated"), which a
|
||||
# long-lived single-replica PVC would violate. It also means a replacement pod never
|
||||
# waits on a lone replica stranded on a dead node: it gets a fresh, empty volume. The
|
||||
# cost, accepted: the cache is cold after every pod replacement (~4 GB of pulls), which
|
||||
# is exactly what the emptyDir already did.
|
||||
# * Bounded twice: the claim size is a hard ceiling (ENOSPC for jobs, never a node
|
||||
# eviction), and the docker-prune container keeps it under 45%.
|
||||
# Velero: excluded by the pod annotation above AND the claim label below.
|
||||
- name: dind-storage
|
||||
emptyDir: {}
|
||||
ephemeral:
|
||||
volumeClaimTemplate:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: act-runner
|
||||
velero.io/exclude-from-backup: "true"
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
storageClassName: longhorn-scratch
|
||||
resources:
|
||||
requests:
|
||||
storage: 30Gi
|
||||
Reference in new issue
Block a user