home-network-k3s-impl/decisions/2026-11-14-bound-var-ephemeral-caches.md

date, branch, title

date	branch	title
2026-11-14	fix/bound-var-ephemeral-caches	Bound the three caches filling each node's /var (act-runner off /var, vuln-scanner fitted)

What changed

Each node's /var is a separate ~11.6 GiB partition holding kubelet emptyDirs and pod logs; containerd's image store is on /data/k3s, so kubelet image GC never frees /var. On 2026-11-12 the act-runner's unbounded Docker cache tipped k3s-prod-2 into DiskPressure (63 act-runner evictions).

• apps/wave-1/gitea-actions.yaml (hand-authored: a renderer EXTRA_UNITS file, placed by catalog/render.py but never generated, so no re-render was needed or run):
  • dind-storage is now a generic ephemeral volume (30Gi, longhorn-scratch) instead of an unbounded emptyDir {}. Off /var, onto /data/longhorn.
  • New docker-prune container (docker:29-cli): every 10 min, if the volume is >=45% used, docker system prune --all --force. Threshold-based, because image until= filters use image creation time and would re-pull long-lived job images between jobs. No $ in the script (Flux postBuild substitution runs over the file).
  • ephemeral-storage requests/limits on dind (256Mi/2Gi), runner (128Mi/1Gi) and prune (16Mi/128Mi); certs emptyDir sizeLimit: 16Mi.
  • Velero opt-out: pod annotation backup.velero.io/backup-volumes-excludes: dind-storage and the claim label velero.io/exclude-from-backup: "true". Daily schedule is opt-out and its volume policy skips only emptyDir/NFS, so without this the cache would go to R2 nightly.
• platform/vuln-scanner/deployment.yaml: /cache emptyDir sizeLimit 10Gi -> 5Gi; scanner ephemeral-storage request 256Mi -> 3584Mi (measured steady state), limit 12Gi -> 6Gi (covers cache + tools + logs, since kubelet compares pod usage with the sum of container limits). Added soft pod anti-affinity (weight 100) from buildkit/buildkitd. The prod-2 node preference is kept: with the runner off /var, sharing prod-2 no longer stacks caches.
• buildkit: unchanged (maxUsedSpace 4GB, sizeLimit 6Gi, bounded). It has no anti-affinity back to the scanner; the scanner side alone carries it.

Why not what the brief preferred

• Persistent PVC for the runner cache (survives pod moves): not done. A long-lived single-replica PVC is persistent storage that is not replicated, against R-STATE-02. A 3-replica PVC would triple multi-GB pull writes onto the device etcd's WAL shares. A generic ephemeral volume is pod-lifetime, the same class as the emptyDir it replaces. It is also never stranded on a dead node: a replacement pod gets a fresh claim. Cost: the cache is cold after a pod replacement, which is what the emptyDir already did.
• vuln-scanner cache onto a PVC: not done. docs/17 "Load" says scan caches MUST NOT be on a replicated Longhorn volume because etcd and Longhorn share a partition. A single-replica volume meets the letter but not the reason. L0 vuln-scanner-supply-chain also requires the cache to be a size-limited emptyDir.

Spec escalations (for the architect, not acted on)

1. docs/04 calls CDI scratch "the one non-replicated use of Longhorn". longhorn-scratch now also backs the act-runner's pod-lifetime Docker cache. The sentence should say "throwaway, pod-lifetime scratch", or name this use.
2. docs/04's rebuild rule and R-STATE-02 don't say whether a generic ephemeral volume counts as "persistent storage". This change reads it as not.

What I verified

• Live, read-only, 2026-11-14 (kubelet stats/summary): nodefs capacity/available prod-1 11.60/8.89 GiB, prod-2 11.85/1.93, prod-3 11.60/4.75. prod-2 dind-storage 3.75 GiB, vuln cache 3.12 GiB; prod-3 buildkit cache 3.86 GiB. Non-cache /var use ~2.1-2.5 GiB per node.
• StorageClasses live: longhorn-scratch = numberOfReplicas: "1", dataLocality: best-effort, reclaim Delete. Longhorn disk available per node 177-187 GB; over-provisioning 200%, minimal available 25%.
• Node allocatable ephemeral-storage 11.8-12.1 GB; current requests prod-2 256Mi, prod-3 2Gi, so the 3584Mi scanner request schedules.
• No Kyverno policy selects the non-app gitea-actions namespace for volume shape. None of the touched paths are in infra/gitea/approval-policy.json sensitive_paths.
• Toolbox gate from this worktree (DEV_KUBECONFIG=/dev/null, ENGINE shim mounting the main .git, spec repo mounted by dev.sh as a sibling), on the committed HEAD: rc 3, UNDETERMINED, not green. "Stages that did not run: gitea-settings". conformance (L2/L3) logged SKIPPED, no cluster. spec-immutability, approval-policy (N/A offline, proves nothing), decisions-log, lint, l0 (incl. flux-postbuild-substitution and vuln-scanner-supply-chain PASS), render-golden and l1 green.

What I did NOT verify

Nothing here has been deployed. This is only provable after merge, under Flux.

• That the generic ephemeral claim provisions and attaches on longhorn-scratch, and that dind starts on it within apps-wave-1's 5m health timeout (wait: true). If it doesn't, wave 1 stalls, and with it every wave above it.
• That docker system prune runs: TLS to dind from the sidecar, BusyBox df -P output matching the regex, cert file permissions readable by a capability-less root process. UNVERIFIED.
• That Velero skips the volume and the claim. The annotation and label are documented Velero opt-outs but were not exercised here.
• That 5Gi is enough for the largest image Trivy scans. Peak extraction size was not measured. An overrun evicts only the scanner, and the image sits out per the in-flight marker.
• runbooks/node-disk.md still points at /var/lib/rancher/k3s for containerd. That is stale (it's on /data/k3s) and not fixed here.