home-network-k3s-impl/runbooks/node-memory.md

node-memory — Memory pressure high on a node

Alert: NodeMemoryPressureHigh (warning, 10m) File: platform/monitoring/alerts.yaml, group platform-health.nodes Expression: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 0.90

What this means here

Available memory has been under 10% for ten minutes. These are HP EliteDesk 800 G5 Minis with ~22 GiB usable, running the whole platform plus KubeVirt VMs on three nodes, so the headroom is genuinely thin and this alert is a capacity signal rather than a bug signal most of the time.

Two cluster-specific facts change how you read it:

SwapTotal == 0 is not evidence swap is off. On 2026-09-07 all three prod nodes came back from a reboot with 977 MiB of swap ACTIVE (/dev/sda4 on prod-1 and prod-3, /dev/nvme0n1p4 on prod-2) while the Ansible role's commented-out fstab line reported success. systemd-gpt-auto-generator claims any GPT partition typed 0657fd6d-a4ab-43c4-84e5-0933c84b4f4f straight from the partition table and synthesises a .swap unit into /run/systemd/generator.late/ — it never reads /etc/fstab. The fix masks swap.target (the single choke point for gpt-auto, fstab-generator and cryptsetup-generator). If a node is swapping, the kubelet's memory accounting is lying to you and eviction will misbehave. Check explicitly with swapon --show, never SwapTotal.

There is a kernel memory-reporting bug on all three nodes. infra/PROD-PROVISIONING.md §0b records an EFI memattr defect that once presented k3s-prod-1 as having 2.6 GiB usable of 16 GB. It needs kernel ≥ 6.13 and all three nodes are exposed (prod-3 publishes MEMATTR=0xbeece018 too). If MemTotal looks absurd rather than merely tight, suspect this before suspecting a leak.

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two dev.sh invocations at once; they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git. main is a protected branch — a direct push is rejected. A change lands by pushing a branch, opening a PR, and merging it once the gate is green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod top nodes; echo; kubectl --context k3s-prod get nodes -o custom-columns=NAME:.metadata.name,MEM:.status.capacity.memory,PRESSURE:.status.conditions[?\(@.type==\"MemoryPressure\"\)].status'

Who is using it — and is swap actually off?

ssh k3s-prod-1 'free -h; echo; swapon --show; echo "(empty output above = swap really off)"; echo; cat /proc/pressure/memory; echo; ps -eo pid,rss,comm --sort=-rss | head -15'

Biggest consumers in the cluster, and anything that has been OOM-killed:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod top pods -A --sort-by=memory | head -20; echo; \
   kubectl --context k3s-prod get events -A --field-selector reason=OOMKilling --sort-by=.lastTimestamp | tail -20'

Likely causes, most likely first

1. A monitoring component grew past its limit

This is the one with real history. When kube-prometheus-stack went 65 → 88 it shipped Grafana 13 and Grafana OOM-looped — "the hidden blocker that stalled the whole kps upgrade and the platform tier". Limits went 256 → 512 → 768Mi for Grafana and Prometheus 2 → 3Gi (commit b2b1aa5). Prod keeps 30d retention / 45GiB retentionSize on a 50Gi PVC, so Prometheus is legitimately the largest single consumer. Remediation: raise the limit in platform/monitoring/kube-prometheus-stack.yaml and push; do not restart it in a loop.

2. A KubeVirt VM, especially a TCG-emulated one

VMs are the largest discretionary consumer. useEmulation: false is set deliberately (on v1.9 true silently forces TCG), but leaked test namespaces running emulated VMs have caused API blips before and were deleted. Check kubectl --context k3s-prod get vmi -A and the virt-launcher-* pods' memory.

3. Swap silently back on after a reboot

See above. Verify with swapon --show and systemctl is-enabled swap.target (should be masked). node_health now reports memory.swap_persistence for this. The fix is the Ansible role; an operator runs it.

4. Genuine capacity exhaustion

If nothing is anomalous, the node is simply full. The node vCPU count was already raised 6 → 8 for CPU pressure; memory is next. A bpg/OpenTofu memory change reboots the VM, so roll one node at a time and wait for Longhorn robustness=healthy between each — the same discipline as every other reboot here.

5. A leak in an app-tier workload

Nextcloud, Plex and the *arr stack are the usual suspects. Compare top pods against the previous day in Grafana rather than guessing.

When to escalate or stop

• Escalate immediately if this is firing together with node-io or node-down on the same node. Memory pressure plus I/O pressure on an etcd member is the shape of the 2026-09-07 outage.
• Do not add swap to relieve this. Kubernetes does not want it here and the platform has already been bitten by swap re-enabling itself.
• Do not evict or drain to free memory: this cluster does not drain (Longhorn rebuild cost). Kill the specific offending pod instead, and only if you know what it is.
• Hand to the operator for anything requiring a VM resize or an Ansible run.

---

Source of truth: runbooks/node-memory.md. Index: /runbooks/ Records: DECISIONS-LOG.md (Grafana OOM, kps 65→88); commits 0e92d25/9d00a3f (swap persistence); infra/PROD-PROVISIONING.md §0b (EFI memattr).