fix(storage): bound the caches filling each node's /var

The act-runner's Docker cache was an unbounded emptyDir on the 11.6 GiB
/var partition and evicted k3s-prod-2 on 2026-11-12. It moves to a
generic ephemeral volume on longhorn-scratch (off /var, pod-lifetime,
single replica), with a threshold-triggered docker prune and Velero
opt-out. vuln-scanner's cache sizeLimit and ephemeral-storage limit are
cut to fit the partition, and it prefers not to share a node with
buildkitd.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

decisions/2026-11-14-bound-var-ephemeral-caches.md

@@ -0,0 +1,84 @@
+---
+date: 2026-11-14
+branch: fix/bound-var-ephemeral-caches
+title: Bound the three caches filling each node's /var (act-runner off /var, vuln-scanner fitted)
+---
+
+## What changed
+
+Each node's `/var` is a separate ~11.6 GiB partition holding kubelet emptyDirs and pod logs;
+containerd's image store is on `/data/k3s`, so kubelet image GC never frees `/var`. On 2026-11-12
+the act-runner's unbounded Docker cache tipped k3s-prod-2 into DiskPressure (63 act-runner
+evictions).
+
+- `apps/wave-1/gitea-actions.yaml` (hand-authored: a renderer `EXTRA_UNITS` file, placed by
+  `catalog/render.py` but never generated, so no re-render was needed or run):
+  - `dind-storage` is now a **generic ephemeral volume** (30Gi, `longhorn-scratch`) instead of an
+    unbounded `emptyDir {}`. Off `/var`, onto `/data/longhorn`.
+  - New `docker-prune` container (`docker:29-cli`): every 10 min, if the volume is >=45% used,
+    `docker system prune --all --force`. Threshold-based, because image `until=` filters use image
+    creation time and would re-pull long-lived job images between jobs. No `$` in the script
+    (Flux postBuild substitution runs over the file).
+  - ephemeral-storage requests/limits on dind (256Mi/2Gi), runner (128Mi/1Gi) and prune
+    (16Mi/128Mi); `certs` emptyDir `sizeLimit: 16Mi`.
+  - Velero opt-out: pod annotation `backup.velero.io/backup-volumes-excludes: dind-storage` and
+    the claim label `velero.io/exclude-from-backup: "true"`. Daily schedule is opt-out and its
+    volume policy skips only emptyDir/NFS, so without this the cache would go to R2 nightly.
+- `platform/vuln-scanner/deployment.yaml`: `/cache` emptyDir sizeLimit 10Gi -> 5Gi; scanner
+  ephemeral-storage request 256Mi -> 3584Mi (measured steady state), limit 12Gi -> 6Gi (covers
+  cache + tools + logs, since kubelet compares pod usage with the sum of container limits). Added
+  soft pod anti-affinity (weight 100) from `buildkit/buildkitd`. The prod-2 node preference is kept:
+  with the runner off `/var`, sharing prod-2 no longer stacks caches.
+- buildkit: unchanged (maxUsedSpace 4GB, sizeLimit 6Gi, bounded). It has no anti-affinity back to
+  the scanner; the scanner side alone carries it.
+
+### Why not what the brief preferred
+
+- **Persistent PVC for the runner cache** (survives pod moves): not done. A long-lived
+  single-replica PVC is persistent storage that is not replicated, against R-STATE-02. A 3-replica
+  PVC would triple multi-GB pull writes onto the device etcd's WAL shares. A generic ephemeral
+  volume is pod-lifetime, the same class as the emptyDir it replaces. It is also never stranded on a
+  dead node: a replacement pod gets a fresh claim. Cost: the cache is cold after a pod replacement,
+  which is what the emptyDir already did.
+- **vuln-scanner cache onto a PVC**: not done. docs/17 "Load" says scan caches MUST NOT be on a
+  replicated Longhorn volume *because etcd and Longhorn share a partition*. A single-replica volume
+  meets the letter but not the reason. L0 `vuln-scanner-supply-chain` also requires the cache to be
+  a size-limited emptyDir.
+
+### Spec escalations (for the architect, not acted on)
+
+1. docs/04 calls CDI scratch "the one non-replicated use of Longhorn". `longhorn-scratch` now also
+   backs the act-runner's pod-lifetime Docker cache. The sentence should say "throwaway,
+   pod-lifetime scratch", or name this use.
+2. docs/04's rebuild rule and R-STATE-02 don't say whether a generic ephemeral volume counts as
+   "persistent storage". This change reads it as not.
+
+## What I verified
+
+- Live, read-only, 2026-11-14 (kubelet `stats/summary`): nodefs capacity/available prod-1
+  11.60/8.89 GiB, prod-2 11.85/1.93, prod-3 11.60/4.75. prod-2 `dind-storage` 3.75 GiB, vuln
+  `cache` 3.12 GiB; prod-3 buildkit `cache` 3.86 GiB. Non-cache `/var` use ~2.1-2.5 GiB per node.
+- StorageClasses live: `longhorn-scratch` = `numberOfReplicas: "1"`, `dataLocality: best-effort`,
+  reclaim Delete. Longhorn disk available per node 177-187 GB; over-provisioning 200%, minimal
+  available 25%.
+- Node allocatable ephemeral-storage 11.8-12.1 GB; current requests prod-2 256Mi, prod-3 2Gi, so the
+  3584Mi scanner request schedules.
+- No Kyverno policy selects the non-app `gitea-actions` namespace for volume shape. None of the
+  touched paths are in `infra/gitea/approval-policy.json` `sensitive_paths`.
+- Toolbox gate: see the PR description for the exact rc and stages that did not run.
+
+## What I did NOT verify
+
+Nothing here has been deployed. This is only provable after merge, under Flux.
+
+- That the generic ephemeral claim provisions and attaches on `longhorn-scratch`, and that dind
+  starts on it within apps-wave-1's 5m health timeout (`wait: true`). If it doesn't, wave 1 stalls,
+  and with it every wave above it.
+- That `docker system prune` runs: TLS to dind from the sidecar, BusyBox `df -P` output matching
+  the regex, cert file permissions readable by a capability-less root process. UNVERIFIED.
+- That Velero skips the volume and the claim. The annotation and label are documented Velero
+  opt-outs but were not exercised here.
+- That 5Gi is enough for the largest image Trivy scans. Peak extraction size was not measured. An
+  overrun evicts only the scanner, and the image sits out per the in-flight marker.
+- `runbooks/node-disk.md` still points at `/var/lib/rancher/k3s` for containerd. That is stale
+  (it's on `/data/k3s`) and not fixed here.