--- date: 2026-11-14 branch: fix/bound-var-ephemeral-caches title: Bound the three caches filling each node's /var (act-runner off /var, vuln-scanner fitted) --- ## What changed Each node's `/var` is a separate ~11.6 GiB partition holding kubelet emptyDirs and pod logs; containerd's image store is on `/data/k3s`, so kubelet image GC never frees `/var`. On 2026-11-12 the act-runner's unbounded Docker cache tipped k3s-prod-2 into DiskPressure (63 act-runner evictions). - `apps/wave-1/gitea-actions.yaml` (hand-authored: a renderer `EXTRA_UNITS` file, placed by `catalog/render.py` but never generated, so no re-render was needed or run): - `dind-storage` is now a **generic ephemeral volume** (30Gi, `longhorn-scratch`) instead of an unbounded `emptyDir {}`. Off `/var`, onto `/data/longhorn`. - New `docker-prune` container (`docker:29-cli`): every 10 min, if the volume is >=45% used, `docker system prune --all --force`. Threshold-based, because image `until=` filters use image creation time and would re-pull long-lived job images between jobs. No `$` in the script (Flux postBuild substitution runs over the file). - ephemeral-storage requests/limits on dind (256Mi/2Gi), runner (128Mi/1Gi) and prune (16Mi/128Mi); `certs` emptyDir `sizeLimit: 16Mi`. - Velero opt-out: pod annotation `backup.velero.io/backup-volumes-excludes: dind-storage` and the claim label `velero.io/exclude-from-backup: "true"`. Daily schedule is opt-out and its volume policy skips only emptyDir/NFS, so without this the cache would go to R2 nightly. - `platform/vuln-scanner/deployment.yaml`: `/cache` emptyDir sizeLimit 10Gi -> 5Gi; scanner ephemeral-storage request 256Mi -> 3584Mi (measured steady state), limit 12Gi -> 6Gi (covers cache + tools + logs, since kubelet compares pod usage with the sum of container limits). Added soft pod anti-affinity (weight 100) from `buildkit/buildkitd`. The prod-2 node preference is kept: with the runner off `/var`, sharing prod-2 no longer stacks caches. - buildkit: unchanged (maxUsedSpace 4GB, sizeLimit 6Gi, bounded). It has no anti-affinity back to the scanner; the scanner side alone carries it. ### Why not what the brief preferred - **Persistent PVC for the runner cache** (survives pod moves): not done. A long-lived single-replica PVC is persistent storage that is not replicated, against R-STATE-02. A 3-replica PVC would triple multi-GB pull writes onto the device etcd's WAL shares. A generic ephemeral volume is pod-lifetime, the same class as the emptyDir it replaces. It is also never stranded on a dead node: a replacement pod gets a fresh claim. Cost: the cache is cold after a pod replacement, which is what the emptyDir already did. - **vuln-scanner cache onto a PVC**: not done. docs/17 "Load" says scan caches MUST NOT be on a replicated Longhorn volume *because etcd and Longhorn share a partition*. A single-replica volume meets the letter but not the reason. L0 `vuln-scanner-supply-chain` also requires the cache to be a size-limited emptyDir. ### Spec escalations (for the architect, not acted on) 1. docs/04 calls CDI scratch "the one non-replicated use of Longhorn". `longhorn-scratch` now also backs the act-runner's pod-lifetime Docker cache. The sentence should say "throwaway, pod-lifetime scratch", or name this use. 2. docs/04's rebuild rule and R-STATE-02 don't say whether a generic ephemeral volume counts as "persistent storage". This change reads it as not. ## What I verified - Live, read-only, 2026-11-14 (kubelet `stats/summary`): nodefs capacity/available prod-1 11.60/8.89 GiB, prod-2 11.85/1.93, prod-3 11.60/4.75. prod-2 `dind-storage` 3.75 GiB, vuln `cache` 3.12 GiB; prod-3 buildkit `cache` 3.86 GiB. Non-cache `/var` use ~2.1-2.5 GiB per node. - StorageClasses live: `longhorn-scratch` = `numberOfReplicas: "1"`, `dataLocality: best-effort`, reclaim Delete. Longhorn disk available per node 177-187 GB; over-provisioning 200%, minimal available 25%. - Node allocatable ephemeral-storage 11.8-12.1 GB; current requests prod-2 256Mi, prod-3 2Gi, so the 3584Mi scanner request schedules. - No Kyverno policy selects the non-app `gitea-actions` namespace for volume shape. None of the touched paths are in `infra/gitea/approval-policy.json` `sensitive_paths`. - Toolbox gate from this worktree (DEV_KUBECONFIG=/dev/null, ENGINE shim mounting the main `.git`, spec repo mounted by dev.sh as a sibling), on the committed HEAD: **rc 3, UNDETERMINED, not green.** "Stages that did not run: gitea-settings". `conformance` (L2/L3) logged SKIPPED, no cluster. spec-immutability, approval-policy (N/A offline, proves nothing), decisions-log, lint, l0 (incl. `flux-postbuild-substitution` and `vuln-scanner-supply-chain` PASS), render-golden and l1 green. ## What I did NOT verify Nothing here has been deployed. This is only provable after merge, under Flux. - That the generic ephemeral claim provisions and attaches on `longhorn-scratch`, and that dind starts on it within apps-wave-1's 5m health timeout (`wait: true`). If it doesn't, wave 1 stalls, and with it every wave above it. - That `docker system prune` runs: TLS to dind from the sidecar, BusyBox `df -P` output matching the regex, cert file permissions readable by a capability-less root process. UNVERIFIED. - That Velero skips the volume and the claim. The annotation and label are documented Velero opt-outs but were not exercised here. - That 5Gi is enough for the largest image Trivy scans. Peak extraction size was not measured. An overrun evicts only the scanner, and the image sits out per the in-flight marker. - `runbooks/node-disk.md` still points at `/var/lib/rancher/k3s` for containerd. That is stale (it's on `/data/k3s`) and not fixed here.