jakman85andClaude Opus 5
8e017fad91
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
126 lines
7.3 KiB
Markdown
126 lines
7.3 KiB
Markdown
# node-memory — Memory pressure high on a node
|
|
|
|
**Alert:** `NodeMemoryPressureHigh` (warning, 10m)
|
|
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes`
|
|
**Expression:** `(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 0.90`
|
|
|
|
## What this means here
|
|
|
|
Available memory has been under 10% for ten minutes. These are HP EliteDesk 800 G5 Minis with
|
|
~22 GiB usable, running the whole platform plus KubeVirt VMs on three nodes, so the headroom is
|
|
genuinely thin and this alert is a capacity signal rather than a bug signal most of the time.
|
|
|
|
Two cluster-specific facts change how you read it:
|
|
|
|
**`SwapTotal == 0` is not evidence swap is off.** On 2026-09-07 all three prod nodes came back from a
|
|
reboot with **977 MiB of swap ACTIVE** (`/dev/sda4` on prod-1 and prod-3, `/dev/nvme0n1p4` on prod-2)
|
|
while the Ansible role's commented-out fstab line reported success.
|
|
`systemd-gpt-auto-generator` claims any GPT partition typed
|
|
`0657fd6d-a4ab-43c4-84e5-0933c84b4f4f` straight from the partition table and synthesises a `.swap`
|
|
unit into `/run/systemd/generator.late/` — it never reads `/etc/fstab`. The fix masks `swap.target`
|
|
(the single choke point for gpt-auto, fstab-generator and cryptsetup-generator). If a node is
|
|
swapping, the kubelet's memory accounting is lying to you and eviction will misbehave. Check
|
|
explicitly with `swapon --show`, never `SwapTotal`.
|
|
|
|
**There is a kernel memory-reporting bug on all three nodes.** `infra/PROD-PROVISIONING.md` §0b
|
|
records an EFI memattr defect that once presented `k3s-prod-1` as having **2.6 GiB usable of 16 GB**.
|
|
It needs kernel ≥ 6.13 and all three nodes are exposed (prod-3 publishes `MEMATTR=0xbeece018` too).
|
|
If `MemTotal` looks absurd rather than merely tight, suspect this before suspecting a leak.
|
|
## Running the commands in this runbook
|
|
|
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
|
that hang is not the cluster being down. Every command below runs as:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
|
|
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
|
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
|
|
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
|
|
the same thing next month as it does today; one that relies on the default does not, and the failure is
|
|
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
|
|
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
|
|
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
|
|
interfere.
|
|
|
|
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
|
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
|
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
|
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
|
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
|
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
|
|
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
|
and it says so.
|
|
|
|
## First commands
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod top nodes; echo; kubectl --context k3s-prod get nodes -o custom-columns=NAME:.metadata.name,MEM:.status.capacity.memory,PRESSURE:.status.conditions[?\(@.type==\"MemoryPressure\"\)].status'
|
|
```
|
|
|
|
Who is using it — and is swap actually off?
|
|
|
|
```
|
|
ssh k3s-prod-1 'free -h; echo; swapon --show; echo "(empty output above = swap really off)"; echo; cat /proc/pressure/memory; echo; ps -eo pid,rss,comm --sort=-rss | head -15'
|
|
```
|
|
|
|
Biggest consumers in the cluster, and anything that has been OOM-killed:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod top pods -A --sort-by=memory | head -20; echo; \
|
|
kubectl --context k3s-prod get events -A --field-selector reason=OOMKilling --sort-by=.lastTimestamp | tail -20'
|
|
```
|
|
|
|
## Likely causes, most likely first
|
|
|
|
### 1. A monitoring component grew past its limit
|
|
This is the one with real history. When kube-prometheus-stack went 65 → 88 it shipped Grafana 13 and
|
|
**Grafana OOM-looped** — "the hidden blocker that stalled the whole kps upgrade and the platform
|
|
tier". Limits went 256 → 512 → **768Mi** for Grafana and Prometheus 2 → **3Gi** (commit `b2b1aa5`).
|
|
Prod keeps 30d retention / 45GiB retentionSize on a 50Gi PVC, so Prometheus is legitimately the
|
|
largest single consumer. Remediation: raise the limit in
|
|
`platform/monitoring/kube-prometheus-stack.yaml` and push; do not restart it in a loop.
|
|
|
|
### 2. A KubeVirt VM, especially a TCG-emulated one
|
|
VMs are the largest discretionary consumer. `useEmulation: false` is set deliberately (on v1.9
|
|
`true` silently forces TCG), but leaked test namespaces running emulated VMs have caused API blips
|
|
before and were deleted. Check `kubectl --context k3s-prod get vmi -A` and the `virt-launcher-*` pods'
|
|
memory.
|
|
|
|
### 3. Swap silently back on after a reboot
|
|
See above. Verify with `swapon --show` and `systemctl is-enabled swap.target` (should be `masked`).
|
|
`node_health` now reports `memory.swap_persistence` for this. The fix is the Ansible role; an
|
|
operator runs it.
|
|
|
|
### 4. Genuine capacity exhaustion
|
|
If nothing is anomalous, the node is simply full. The node vCPU count was already raised 6 → 8 for
|
|
CPU pressure; memory is next. A bpg/OpenTofu memory change **reboots the VM**, so roll **one node at
|
|
a time and wait for Longhorn `robustness=healthy` between each** — the same discipline as every other
|
|
reboot here.
|
|
|
|
### 5. A leak in an app-tier workload
|
|
Nextcloud, Plex and the *arr stack are the usual suspects. Compare `top pods` against the previous
|
|
day in Grafana rather than guessing.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **Escalate immediately if this is firing together with `node-io` or `node-down`** on the same node.
|
|
Memory pressure plus I/O pressure on an etcd member is the shape of the 2026-09-07 outage.
|
|
- **Do not add swap** to relieve this. Kubernetes does not want it here and the platform has already
|
|
been bitten by swap re-enabling itself.
|
|
- **Do not evict or drain** to free memory: this cluster does not drain (Longhorn rebuild cost). Kill
|
|
the specific offending pod instead, and only if you know what it is.
|
|
- Hand to the operator for anything requiring a VM resize or an Ansible run.
|
|
|
|
---
|
|
Source of truth: `runbooks/node-memory.md`. Index: `/runbooks/`
|
|
Records: `DECISIONS-LOG.md` (Grafana OOM, kps 65→88); commits `0e92d25`/`9d00a3f` (swap persistence);
|
|
`infra/PROD-PROVISIONING.md` §0b (EFI memattr).
|