jakman85andClaude Opus 5
8e017fad91
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
159 lines
8.9 KiB
Markdown
159 lines
8.9 KiB
Markdown
# node-down — A node's node-exporter is unscrapable (the node may be gone)
|
|
|
|
**Alert:** `NodeExporterDown` (critical, 5m)
|
|
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes`
|
|
**Expression:** `up{job="node-exporter"} == 0`
|
|
|
|
## What this means here
|
|
|
|
Prometheus has not scraped node-exporter on that instance for 5 minutes. The alert is worded
|
|
carefully — it says the *exporter* is down — because on this cluster the two cases look identical
|
|
from Prometheus and have opposite urgency:
|
|
|
|
- **The node is genuinely gone** (powered off, kernel panic, network loss).
|
|
- **The node is up and the kubelet stopped posting status.** This is what actually happened on
|
|
**2026-09-07 at 06:05:21Z**: `k3s-prod-1` went NotReady with "Kubelet stopped posting node status"
|
|
while the host was up, pingable and SSH-able. Root cause was disk-I/O starvation of the embedded
|
|
etcd, not a dead machine. If this alert is firing alongside `node-io`, read `/runbooks/node-io`
|
|
first — that is the cause and this is the symptom.
|
|
|
|
**Why it is critical rather than a warning.** All three nodes are control-plane + etcd members
|
|
(`k3s-prod-1` .10, `k3s-prod-2` .11, `k3s-prod-3` .12, kube-vip API VIP `192.168.2.9`). Losing one
|
|
leaves a 2-member etcd with no fault tolerance: the next loss ends quorum and the API server with it.
|
|
Losing a node also degrades every Longhorn volume that had a replica there, and every degraded volume
|
|
starts rebuilding — which loads the same disks etcd fsyncs to. One node down is not a routine event
|
|
on a 3-node cluster; it is one step from an outage.
|
|
|
|
Two node-specific roles matter when you read which node it is:
|
|
- `k3s-prod-3` is the **sole VPN egress gateway**. Cilium OSS has **no egress-gateway failover**, so
|
|
while that node is down, VPN-classed egress is an outage, not a degradation. See `/runbooks/vpn-down`.
|
|
- `k3s-prod-2` is the cluster **founder** (`[k3s_prod_primary]` in `inventory/hosts.ini`). Do not
|
|
re-provision it from the inventory without reading `/runbooks/node-io` — re-provisioning a node
|
|
still listed as primary founds a **second cluster**.
|
|
## Running the commands in this runbook
|
|
|
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
|
that hang is not the cluster being down. Every command below runs as:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
|
|
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
|
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
|
|
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
|
|
the same thing next month as it does today; one that relies on the default does not, and the failure is
|
|
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
|
|
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
|
|
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
|
|
interfere.
|
|
|
|
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
|
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
|
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
|
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
|
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
|
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
|
|
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
|
and it says so.
|
|
|
|
## First commands
|
|
|
|
Is the node actually NotReady, or is only the exporter unscrapable?
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get nodes -o wide; echo; kubectl --context k3s-prod -n monitoring get pods -l app.kubernetes.io/name=prometheus-node-exporter -o wide'
|
|
```
|
|
|
|
Is it the API server or the node? (If this hangs, you have a bigger problem than one node.)
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get --raw /readyz?verbose 2>&1 | tail -20'
|
|
```
|
|
|
|
Is the host alive at all? Prod is directly on `192.168.2.0/24` — no NAT hop:
|
|
|
|
```
|
|
ping -c3 192.168.2.10 ; ssh k3s-prod-1 'uptime; systemctl is-active k3s; cat /proc/pressure/io'
|
|
```
|
|
|
|
The three lines that distinguish "etcd starved" from "machine dead":
|
|
|
|
```
|
|
ssh k3s-prod-1 'journalctl -u k3s --since "30 min ago" | grep -Ei "fdatasync|apply request took|too many requests|Kubelet stopped|leader"'
|
|
```
|
|
|
|
Etcd membership and leader — do this before touching anything:
|
|
|
|
```
|
|
ssh k3s-prod-2 'curl -s http://127.0.0.1:2381/metrics | grep -E "etcd_server_is_leader|etcd_server_has_leader"'
|
|
```
|
|
|
|
What did the node take with it?
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
|
|
-o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy; \
|
|
kubectl --context k3s-prod get pods -A --field-selector spec.nodeName=k3s-prod-1 -o wide | head -30'
|
|
```
|
|
|
|
## Likely causes, most likely first
|
|
|
|
### 1. Disk I/O starvation wedged the kubelet (host still up)
|
|
The signature is `slow fdatasync took=3.56s` / `apply request took too long took=8.30s` /
|
|
`etcdserver: too many requests` in `journalctl -u k3s`, with `/proc/pressure/io` full avg300 well
|
|
above 1. On 2026-09-07 node 1 read **11.36** against 0.75 on node 2 and 1.29 on node 3.
|
|
Remediation: follow `/runbooks/node-io`. Recovery that day required a manual
|
|
`systemctl restart k3s` on the node (operator, on-host), after which 18 Longhorn volumes rebuilt and
|
|
pressure only fell when the rebuilds finished.
|
|
|
|
### 2. The node rebooted (planned or otherwise)
|
|
Check `uptime`. A reboot is not harmless here: **every node reboot degrades all ~18 Longhorn volumes
|
|
and triggers a full rebuild**. If a reboot is intentional, roll one node at a time and wait for
|
|
`robustness=healthy` on every volume before the next, and freeze rebuilds first (see
|
|
`/runbooks/longhorn-degraded`). Note two hardware quirks recorded in `infra/PROD-PROVISIONING.md`
|
|
§0b: `k3s-prod-2` **hangs on a warm reboot while staying powered**, and all three nodes carry the EFI
|
|
memattr kernel bug that once presented `k3s-prod-1` as having 2.6 GiB usable of 16 GB (needs kernel
|
|
≥ 6.13).
|
|
|
|
### 3. node-exporter itself is unhealthy, node fine
|
|
If `kubectl get nodes` says Ready and the pod is CrashLoopBackOff or Pending, this is a monitoring
|
|
fault, not a node fault. Downgrade urgency, but do not ignore it — while that exporter is down you
|
|
are blind to `node-io`, `node-memory`, `node-disk` and `node-cpu` on that node, which is exactly the
|
|
blindness that let 2026-09-07 arrive as a surprise. Related: `/runbooks/targets-down`.
|
|
|
|
### 4. Network loss to the node
|
|
`ping` fails, SSH fails. Check the switch and the node's `eno1` (prod-3's
|
|
`/etc/network/interfaces` also holds a commented-out wireless stanza — inert, but worth scrubbing).
|
|
A node reachable by ping but not by the API is usually Cilium, not the network: heavy sustained node
|
|
load has been observed to degrade the Cilium datapath into intermittent LB-VIP unreachability,
|
|
cleared by a restart.
|
|
|
|
### 5. Hardware
|
|
`k3s-prod-1`'s original 2011 SSD had 61,440 reallocated sectors and was replaced 2026-09-07. Check
|
|
SMART (`/runbooks/node-io` has the command). Hardware is an operator job.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **STOP if two nodes are affected.** Two of three etcd members down means no quorum, no API server,
|
|
and no GitOps. Nothing in this runbook applies; get the operator.
|
|
- **Do not reboot, drain or cordon a second node** while any Longhorn volume is not `healthy`. This
|
|
cluster deliberately does not `kubectl drain` — the auto-drain of system-upgrade-controller was
|
|
rejected precisely because it fights slow-disk Longhorn.
|
|
- **Do not delete the Node object** to "clean up". That triggers replica deletion in Longhorn.
|
|
- Escalate to the operator for: a power-cycle, a disk replacement, or re-provisioning a node (the
|
|
inventory's `[k3s_prod_primary]` trap above makes re-provisioning a two-person job).
|
|
- If the node returns but volumes stay degraded for more than an hour, go to
|
|
`/runbooks/longhorn-degraded`.
|
|
|
|
---
|
|
Source of truth: `runbooks/node-down.md` in `home-network-k3s-impl`. Index: `/runbooks/`
|
|
Incident record: `DECISIONS-LOG.md` "2026-09-07 — k3s-prod-1 NotReady"; `infra/PROD-PROVISIONING.md` §0b/§0c.
|