Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

fix(storage): bound the caches filling each node's /var #174

Merged
jakman85 merged 2 commits from fix/bound-var-ephemeral-caches into main 2026-11-14 21:56:43 +00:00
Conversation 0 Commits 2 Files Changed 3
+258 -5
gitea-admin commented 2026-11-14 20:17:42 +00:00
Owner
Copy link
Copy source

Problem

Each node's /var is a separate ~11.6 GiB partition. It holds kubelet emptyDirs and pod logs, and kubelet evicts at <5% free. The image store is on /data/k3s, so image GC never frees /var. Measured 2026-11-14 (kubelet stats/summary, read-only):

node /var avail big emptyDirs
prod-1 8.89 / 11.60 GiB none
prod-2 1.93 / 11.85 GiB act-runner dind-storage 3.75 GiB, vuln-scanner cache 3.12 GiB
prod-3 4.75 / 11.60 GiB buildkitd cache 3.86 GiB

On 2026-11-12 the unbounded act-runner Docker cache tipped prod-2 into DiskPressure (63 act-runner evictions).

Changes

  • apps/wave-1/gitea-actions.yaml (hand-authored EXTRA_UNITS file; the renderer only places it and does not generate it, so nothing was re-rendered):
    • dind-storage: unbounded emptyDir {} → generic ephemeral volume, 30Gi, longhorn-scratch (1 replica, best-effort locality). Off /var, onto /data/longhorn.
    • New docker-prune container (docker:29-cli): every 10 min, if the volume is ≥45% used, runs docker system prune --all --force. The 45% keeps the volume above the 50%-free gate of PersistentVolumeFillingUpPredicted. The script has no $, because Flux postBuild substitution runs over the file.
    • ephemeral-storage requests and limits on dind, runner and prune; certs emptyDir sizeLimit: 16Mi.
    • Velero opt-out: pod annotation backup.velero.io/backup-volumes-excludes: dind-storage, plus the claim label velero.io/exclude-from-backup: "true". The daily schedule is opt-out and only skips emptyDir and NFS.
  • platform/vuln-scanner/deployment.yaml:
    • /cache sizeLimit 10Gi → 5Gi.
    • Scanner ephemeral-storage request 256Mi → 3584Mi (measured steady state) and limit 12Gi → 6Gi (cache + tools + logs). The old limits were bigger than the whole partition, so they could never fire.
    • Soft pod anti-affinity from buildkit/buildkitd.
    • The prod-2 preference is kept. With the runner off /var, sharing prod-2 no longer stacks caches.
  • buildkit: unchanged (maxUsedSpace 4GB, sizeLimit 6Gi; already bounded).
  • decisions/2026-11-14-bound-var-ephemeral-caches.md: the handoff.

Why not the brief's preferred options

  • A persistent PVC for the runner cache: a long-lived single-replica PVC is unreplicated persistent storage, which R-STATE-02 rules out. A 3-replica PVC would triple multi-GB pull writes onto the device etcd's WAL shares. A generic ephemeral volume lives only as long as the pod, so it is never stranded on a dead node. The cost: the cache starts cold after each pod replacement, same as the emptyDir did.
  • A PVC for the vuln-scanner cache: docs/17 "Load" forbids scan caches on replicated Longhorn because etcd and Longhorn share a partition. L0 vuln-scanner-supply-chain requires a size-limited emptyDir.

Spec escalations (for the architect, not acted on):

  • docs/04 calls CDI scratch "the one non-replicated use of Longhorn". That is no longer true.
  • R-STATE-02 doesn't say whether a generic ephemeral volume counts as persistent storage.

Gate

Toolbox gate run from the worktree on the committed HEAD 89ed1af: rc 3, UNDETERMINED (not green).

  • Stages that did not run: gitea-settings
  • conformance (L2/L3): SKIPPED (no cluster)
  • Green: spec-immutability, approval-policy (N/A offline), decisions-log, lint, l0, render-golden, l1

Sensitive paths

None. No touched path matches infra/gitea/approval-policy.json sensitive_paths.

NOT verified (only provable after merge, under Flux)

  • Whether the ephemeral claim provisions on longhorn-scratch, and whether dind starts on it within apps-wave-1's 5m wait: true timeout. If it doesn't, wave 1 stalls, and with it every wave above.
  • Whether docker-prune actually reaches dind over TLS, whether the BusyBox df -P regex matches, and whether the certs are readable without capabilities.
  • Whether Velero actually skips the volume.
  • Whether 5Gi covers the largest image Trivy extracts.

🤖 Generated with Claude Code

## Problem Each node's `/var` is a separate ~11.6 GiB partition. It holds kubelet emptyDirs and pod logs, and kubelet evicts at <5% free. The image store is on `/data/k3s`, so image GC never frees `/var`. Measured 2026-11-14 (kubelet stats/summary, read-only): | node | /var avail | big emptyDirs | |---|---|---| | prod-1 | 8.89 / 11.60 GiB | none | | prod-2 | **1.93** / 11.85 GiB | act-runner `dind-storage` 3.75 GiB, vuln-scanner `cache` 3.12 GiB | | prod-3 | 4.75 / 11.60 GiB | buildkitd `cache` 3.86 GiB | On 2026-11-12 the unbounded act-runner Docker cache tipped prod-2 into DiskPressure (63 act-runner evictions). ## Changes - **`apps/wave-1/gitea-actions.yaml`** (hand-authored `EXTRA_UNITS` file; the renderer only places it and does not generate it, so nothing was re-rendered): - `dind-storage`: unbounded `emptyDir {}` → **generic ephemeral volume, 30Gi, `longhorn-scratch`** (1 replica, best-effort locality). Off `/var`, onto `/data/longhorn`. - New **`docker-prune`** container (`docker:29-cli`): every 10 min, if the volume is ≥45% used, runs `docker system prune --all --force`. The 45% keeps the volume above the 50%-free gate of `PersistentVolumeFillingUpPredicted`. The script has no `$`, because Flux postBuild substitution runs over the file. - ephemeral-storage requests and limits on dind, runner and prune; `certs` emptyDir `sizeLimit: 16Mi`. - Velero opt-out: pod annotation `backup.velero.io/backup-volumes-excludes: dind-storage`, plus the claim label `velero.io/exclude-from-backup: "true"`. The daily schedule is opt-out and only skips emptyDir and NFS. - **`platform/vuln-scanner/deployment.yaml`**: - `/cache` sizeLimit 10Gi → 5Gi. - Scanner ephemeral-storage request 256Mi → 3584Mi (measured steady state) and limit 12Gi → 6Gi (cache + tools + logs). The old limits were bigger than the whole partition, so they could never fire. - Soft pod anti-affinity from `buildkit/buildkitd`. - The prod-2 preference is kept. With the runner off `/var`, sharing prod-2 no longer stacks caches. - **buildkit**: unchanged (maxUsedSpace 4GB, sizeLimit 6Gi; already bounded). - `decisions/2026-11-14-bound-var-ephemeral-caches.md`: the handoff. ## Why not the brief's preferred options - **A persistent PVC for the runner cache:** a long-lived single-replica PVC is unreplicated persistent storage, which R-STATE-02 rules out. A 3-replica PVC would triple multi-GB pull writes onto the device etcd's WAL shares. A generic ephemeral volume lives only as long as the pod, so it is never stranded on a dead node. The cost: the cache starts cold after each pod replacement, same as the emptyDir did. - **A PVC for the vuln-scanner cache:** docs/17 "Load" forbids scan caches on replicated Longhorn *because* etcd and Longhorn share a partition. L0 `vuln-scanner-supply-chain` requires a size-limited emptyDir. **Spec escalations** (for the architect, not acted on): - docs/04 calls CDI scratch "the one non-replicated use of Longhorn". That is no longer true. - R-STATE-02 doesn't say whether a generic ephemeral volume counts as persistent storage. ## Gate Toolbox gate run from the worktree on the committed HEAD `89ed1af`: **rc 3, UNDETERMINED (not green).** - `Stages that did not run: gitea-settings` - `conformance` (L2/L3): SKIPPED (no cluster) - Green: spec-immutability, approval-policy (N/A offline), decisions-log, lint, l0, render-golden, l1 ## Sensitive paths None. No touched path matches `infra/gitea/approval-policy.json` `sensitive_paths`. ## NOT verified (only provable after merge, under Flux) - Whether the ephemeral claim provisions on `longhorn-scratch`, and whether dind starts on it within apps-wave-1's 5m `wait: true` timeout. If it doesn't, wave 1 stalls, and with it every wave above. - Whether `docker-prune` actually reaches dind over TLS, whether the BusyBox `df -P` regex matches, and whether the certs are readable without capabilities. - Whether Velero actually skips the volume. - Whether 5Gi covers the largest image Trivy extracts. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
gitea-admin added 2 commits 2026-11-14 20:17:42 +00:00
fix(storage): bound the caches filling each node's /var 3ae7fb74f8 
The act-runner's Docker cache was an unbounded emptyDir on the 11.6 GiB
/var partition and evicted k3s-prod-2 on 2026-11-12. It moves to a
generic ephemeral volume on longhorn-scratch (off /var, pod-lifetime,
single replica), with a threshold-triggered docker prune and Velero
opt-out. vuln-scanner's cache sizeLimit and ephemeral-storage limit are
cut to fit the partition, and it prefers not to share a node with
buildkitd.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
docs(decisions): record gate result for bound-var-ephemeral-caches
gate / gate (pull_request) Successful in 56s
Details
gate / conformance (pull_request) Skipped
Details
89ed1af282 
Co-Authored-By: Claude Opus 5.5 <[email protected]>
jakman85 merged commit 6868e8d9a5 into main 2026-11-14 21:56:43 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
agent
automated-update
blast-radius
breaking
ci-tooling
cve-dashboard
dependencies
needs-review
own-image
range-widen
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
gitea-admin jakman85 (Keith Jackson) pm-agent renovate-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Home-Network/home-network-k3s-impl#174