fix(storage): bound the caches filling each node's /var

The act-runner's Docker cache was an unbounded emptyDir on the 11.6 GiB
/var partition and evicted k3s-prod-2 on 2026-11-12. It moves to a
generic ephemeral volume on longhorn-scratch (off /var, pod-lifetime,
single replica), with a threshold-triggered docker prune and Velero
opt-out. vuln-scanner's cache sizeLimit and ephemeral-storage limit are
cut to fit the partition, and it prefers not to share a node with
buildkitd.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

platform/vuln-scanner/deployment.yaml

@@ -14,6 +14,15 @@
 #
 # SOFT affinity toward k3s-prod-2: prod-1 is the node whose etcd I/O starved historically, and scans
 # are CPU- and I/O-heavy. Soft on purpose: a scanner that cannot be placed must still run elsewhere.
+#
+# SOFT anti-affinity from buildkitd (added 2026-11-14). The /cache emptyDir lives on the node's /var,
+# a separate ~11.6 GiB partition kubelet evicts from at <5% free, and buildkitd's own /var cache is
+# bounded at 6Gi. Measured 2026-11-14: ~2.2-2.5 GiB of /var is already used on each node by
+# everything else, so this cache (<=5Gi) and buildkit's (<=6Gi) cannot both fit on one node. The
+# third big /var tenant, the act-runner Docker cache, moved off /var the same day
+# (apps/wave-1/gitea-actions.yaml, `dind-storage`), so sharing prod-2 with Gitea and the runner no
+# longer stacks caches and the prod-2 preference stays. Weight 100 on both: when prod-2 is
+# unavailable the scanner prefers prod-1 over buildkit's node. Soft, for the same reason as above.
 apiVersion: apps/v1
 kind: Deployment
 metadata:
@@ -61,6 +70,16 @@ spec:
                     operator: In
                     values:
                       - k3s-prod-2
+        podAntiAffinity:
+          preferredDuringSchedulingIgnoredDuringExecution:
+            - weight: 100
+              podAffinityTerm:
+                topologyKey: kubernetes.io/hostname
+                namespaces:
+                  - buildkit
+                labelSelector:
+                  matchLabels:
+                    app.kubernetes.io/name: buildkitd
       initContainers:
         - name: trivy-tools
           # TRIVY 0.76.1 — pinned by digest (R-VULN-09). Not 0.69.4/5/6 (compromised release, March
@@ -148,15 +167,24 @@ spec:
             - { name: metrics, containerPort: 9090, protocol: TCP }
           # Sized to trivy, the only heavy thing here: scans are sequential, the limit is the peak of
           # one large image; the emptyDir sizeLimit bounds disk.
+          #
+          # ephemeral-storage, RESIZED 2026-11-14. It was request 256Mi / limit 12Gi with a 10Gi
+          # cache sizeLimit, on a node /var of ~11.6 GiB: neither bound could fire before kubelet's
+          # node-level DiskPressure eviction did, which takes other pods down with it. kubelet
+          # compares the POD's usage (its emptyDirs + container writable layers + logs) against the
+          # SUM of its containers' limits, and only this container carries one, so this limit must
+          # cover cache (5Gi) + tools (512Mi) + logs: 6Gi. The request is the measured steady state
+          # (3.12 GiB of cache: Trivy DB 1.7G + Java DB 1.5G, 2026-11-14) so the scheduler stops
+          # treating this pod as ~free of local disk.
           resources:
             requests:
               cpu: 100m
               memory: 256Mi
-              ephemeral-storage: 256Mi
+              ephemeral-storage: 3584Mi
             limits:
               cpu: "1"
               memory: 2Gi
-              ephemeral-storage: 12Gi
+              ephemeral-storage: 6Gi
           securityContext:
             allowPrivilegeEscalation: false
             readOnlyRootFilesystem: true
@@ -280,6 +308,13 @@ spec:
             claimName: vuln-scanner-data
         # Trivy DB + per-image scan cache + TMPDIR (trivy extracts layers here). Size-limited; never
         # on a replicated volume or in etcd (docs/17 "Load"). A full run cleans it between images.
+        #
+        # 5Gi, down from 10Gi (2026-11-14): 10Gi was most of the node's whole ~11.6 GiB /var, so it
+        # could never fire before node DiskPressure. 5Gi = the 3.12 GiB of DBs measured 2026-11-14
+        # plus ~1.9 GiB for one image's extraction (TMPDIR points here, so extraction is inside the
+        # bound) and a DB refresh. Overrunning it evicts THIS pod only; main.py records the
+        # in-flight image as failed and sits it out, so an oversized image cannot loop the scanner.
+        # If the DBs grow past ~4 GiB, raise this AND the scanner's ephemeral-storage limit together.
         - name: cache
           emptyDir:
-            sizeLimit: 10Gi
+            sizeLimit: 5Gi