jakman85andClaude Opus 5
b9f3143f42
NodeIOPressureCritical has now fired twice on k3s-prod-1 for opposite causes and
said the same thing both times. On 2026-09-14 02:40-02:45Z it fired at PSI(full)
20.98% — twice the threshold, four times the 2026-09-07 level — while every local
device was idle (0.09-0.27 utilisation) and fast (1.4-4.8 ms per write), NFS
retransmissions were zero and no Longhorn volume was degraded. Establishing that
took eight ad-hoc PromQL queries after the event had already recovered.
Adds the group platform-health.node-io-cause: eight rules, one per cause class,
each naming its cause in its own summary.
NodeDiskDeviceSaturated device saturation >0.85 busy, 10m
NodeDiskWriteLatencyHigh device latency >20ms/write, 10m
NodeDiskWriteLatencyCritical device latency >50ms/write, 5m
NodeDiskReadLatencyHigh device latency >250ms/read, 15m
NodeNFSRetransmissionRateHigh NFS transport >1% retransmitted, 10m
NodeIOPressureNotLocalDisk not local disk PSI>10% with all of the above absent
NodeEtcdRequestsSlow etcd >0.5% of etcd requests over 1s
ContainerIOStallHigh which workload per-container PSI >15%
Every threshold is measured on prod over 24-48h, not guessed, and each rule
records its headroom. The PSI thresholds themselves are deliberately unchanged:
the 21% stall was real, it fired twice in 24h for two real events on the one node
that has a problem, and raising the bar would also have raised it above the 11.36%
at which prod-1 was already losing etcd on 2026-09-07.
Context reaches the operator at page time three ways: the summary states the cause
class; every rule carries a diagnostic_query annotation with the exact companion
PromQL; and node-pressure-dashboard.yaml gains a WHY row (seven panels) plotting
saturation, write/read service time, NFS retransmissions, our own NFS op rate,
the etcd slow-request fraction and per-container PSI side by side. runbooks/node-io.md
gains a 10-row triage table with the real healthy / 2026-09-14 / 2026-09-07 columns,
which turns those eight queries into a two-minute read.
etcd and NFS metric availability, verified on prod rather than assumed:
* etcd SERVER disk metrics do NOT exist. etcd_disk_wal_fsync_duration_seconds and
etcd_disk_backend_commit_duration_seconds are absent (kubeEtcd disabled for k3s);
a query for etcd_disk_*/etcd_server_*/etcd_mvcc_* returns nothing at all.
NodeEtcdRequestsSlow uses the apiserver's client-side view of its own local
member instead, and says in its own description that it is a proxy.
* node_nfs_* DOES exist — the nfs collector is on by default, host-wide via
/proc/net/rpc/nfs, confirmed on all three nodes. It gives op counts and
retransmissions and NO latency, so it can see a dropping export but not a slow
one. The mountstats collector is enabled here to close that gap; NO rule
references node_mountstats_* yet, because the series do not exist until the
HelmRelease rolls out and the collector's visibility through the node-exporter
mount namespace must be verified on prod first. The runbook records the check
and says to revert if it comes back empty.
Proof, per the repo's two shipped alerting failure modes:
* platform/monitoring/tests/node-io-cause.test.yaml — 7 fixtures, all from
measured profiles: healthy (all 8 quiet), 2026-09-14 (NotLocalDisk fires and
names sonarr), 2026-09-07 re-queried at offset 164h (write-latency and etcd
fire, NotLocalDisk correctly silent), rebuild storm, NFS transport, read
latency, and the `unless` semantics.
* 10 mutations of the shipped rules, all killed. The mutation run found a real
gap: the etcd threshold could be changed without failing any test, because it
was only pinned by a promql_expr_test that restated the threshold in the test
file. An alert_rule_test now pins it.
* All 11 referenced metrics verified to return series on prod with
`promtool query instant`, and all 8 expressions verified quiet on today's state.
Also extends the alert-metrics-have-scrape-targets owner table for container_*
(the kubelet ServiceMonitor's cAdvisor path, which this entry already claimed in
its title) and for etcd_request_duration_seconds_* by EXACT NAME, deliberately not
by an etcd_ prefix, so a future rule on the unscraped etcd_disk_* names still fails
the gate.
Co-Authored-By: Claude Opus 5 <[email protected]>
410 lines
24 KiB
YAML
410 lines
24 KiB
YAML
---
|
|
# kube-prometheus-stack — Prometheus + Alertmanager + Grafana + kube-state-metrics +
|
|
# node-exporter + the standard Kubernetes exporters/rules (docs/10). Serves R-OBS-01/02/03.
|
|
#
|
|
# Retention/PVC sizes are overlay knobs (R-ENV-02) resolved from cluster-vars via Flux
|
|
# postBuild.substituteFrom. Everything else is DEV-SCOPED for the ~22 GiB workstation ceiling:
|
|
# single replicas, modest requests/limits, ephemeral (emptyDir) Alertmanager/Grafana state, and
|
|
# the k3s-embedded control-plane scrape jobs disabled (they are not separately reachable on k3s and
|
|
# would otherwise show as permanently-down targets and burn RAM/alert noise).
|
|
apiVersion: helm.toolkit.fluxcd.io/v2
|
|
kind: HelmRelease
|
|
metadata:
|
|
name: kube-prometheus-stack
|
|
namespace: monitoring
|
|
spec:
|
|
interval: 30m
|
|
# The 65->88 catch-up is a large multi-major jump (operator + KSM + node-exporter roll, plus the
|
|
# Prometheus/Alertmanager StatefulSets). The default 5m Helm timeout is too short for the full rollout
|
|
# to reach Ready, so Flux logged "context deadline exceeded" and rolled back to 65.8.1 four times.
|
|
# A generous timeout lets one attempt finish; remediateLastFailure:false keeps partial progress instead
|
|
# of yo-yoing back to the old chart (a re-reconcile then drives it forward). CRDs are pre-applied
|
|
# server-side (Helm never updates crds/ on upgrade), so schema is already at operator v0.93.1.
|
|
timeout: 20m0s
|
|
chart:
|
|
spec:
|
|
chart: kube-prometheus-stack
|
|
version: ">=88.0.0 <89.0.0" # k8s-1.36 catch-up (po v0.93; CRDs pre-applied server-side, Helm won't update them). Renovate maintains
|
|
sourceRef:
|
|
kind: HelmRepository
|
|
name: prometheus-community
|
|
namespace: monitoring
|
|
install:
|
|
remediation:
|
|
retries: 3
|
|
upgrade:
|
|
remediation:
|
|
retries: 2
|
|
remediateLastFailure: false
|
|
cleanupOnFail: true
|
|
values:
|
|
# --- k3s trim: control-plane components are embedded in the k3s server process and are not
|
|
# scrapable as discrete endpoints. Disabling avoids permanently-firing "target down" noise.
|
|
kubeControllerManager:
|
|
enabled: false
|
|
kubeScheduler:
|
|
enabled: false
|
|
kubeEtcd:
|
|
enabled: false
|
|
kubeProxy:
|
|
enabled: false
|
|
|
|
# --- Prometheus ---
|
|
prometheus:
|
|
prometheusSpec:
|
|
replicas: 1
|
|
retention: ${MONITORING_PROMETHEUS_RETENTION} # overlay knob (dev 3d / prod long)
|
|
# SIZE-based retention cap (docs/10, R-OBS). `retention` alone prunes only by AGE — on a young
|
|
# cluster (< retention window) NOTHING is pruned yet, so a busy TSDB fills the PVC before the
|
|
# time window elapses. A full /prometheus wedges ALL WAL commits ("no space left on device"):
|
|
# scrapes still 200 but samples are never stored, the head freezes, and every absent()/==0
|
|
# platform alert (Keycloak/KubeAPI/Kubelet/VPN/KubeVirt/CDI…) fires falsely because instant
|
|
# queries at "now" return empty. retentionSize makes Prometheus drop the OLDEST blocks once the
|
|
# block set approaches this cap, so the disk never fills regardless of cluster age or churn.
|
|
# Kept BELOW the PVC size (leaves headroom for the WAL + head + compaction scratch, which are
|
|
# NOT counted by retentionSize). Prometheus size units are *iB (GiB), distinct from the K8s PVC
|
|
# `Gi`, so it is a separate overlay knob rather than a reuse of the PVC-size knob.
|
|
retentionSize: ${MONITORING_PROMETHEUS_RETENTION_SIZE} # overlay knob (dev 5GiB / base 8GiB / prod 45GiB)
|
|
# Discover ServiceMonitors/PodMonitors/Rules from ALL namespaces (flux, cert-manager,
|
|
# longhorn, velero, keycloak, cilium/hubble ship their own) — not just this release's.
|
|
ruleSelectorNilUsesHelmValues: false
|
|
serviceMonitorSelectorNilUsesHelmValues: false
|
|
podMonitorSelectorNilUsesHelmValues: false
|
|
probeSelectorNilUsesHelmValues: false
|
|
# Modest footprint for a home dev cluster (few hundred series). Limit gives headroom for
|
|
# WAL replay / compaction spikes without letting Prometheus eat the node.
|
|
resources:
|
|
requests:
|
|
cpu: 100m
|
|
memory: 512Mi
|
|
limits:
|
|
memory: 3Gi # 1Gi OOMKilled (exit 137) during TSDB compaction/checkpoint;
|
|
# 2Gi ran pinned at ~98% (2002Mi) with no room for compaction/WAL-replay
|
|
# spikes (the classic Prometheus OOM). Nodes are 24Gi at 38-44% used
|
|
# (~14Gi free each), so 3Gi gives real headroom.
|
|
storageSpec:
|
|
volumeClaimTemplate:
|
|
spec:
|
|
storageClassName: longhorn # replicated tier (ADR-0003)
|
|
accessModes: ["ReadWriteOnce"]
|
|
resources:
|
|
requests:
|
|
storage: ${MONITORING_PROMETHEUS_PVC_SIZE} # overlay knob (dev 5Gi / prod large)
|
|
|
|
# --- Alertmanager: routes by severity to email (R-OBS-02) ---
|
|
alertmanager:
|
|
alertmanagerSpec:
|
|
replicas: 1
|
|
# DEV TRIM: no PVC — silences/notification-log live in emptyDir. Acceptable in dev (a
|
|
# restart re-fires active alerts). Prod overlay can add a small storage block.
|
|
resources:
|
|
requests:
|
|
cpu: 10m
|
|
memory: 32Mi
|
|
limits:
|
|
memory: 96Mi
|
|
# SMTP credentials are an OPERATOR HAND-OFF: a SOPS-encrypted Secret, referenced BY NAME
|
|
# only (docs/07). Mounted at /etc/alertmanager/secrets/<name>/ so the config below can read
|
|
# the password from a file (never inline plaintext).
|
|
secrets:
|
|
- alertmanager-smtp
|
|
# Routing/receivers. Non-secret SMTP endpoint fields (smarthost/from/to/username) are
|
|
# operator-supplied placeholders (REPLACE-ME, same convention as velero); the PASSWORD is
|
|
# read from the mounted SOPS secret via *_file. Both severities email; subject/route differ.
|
|
config:
|
|
global:
|
|
resolve_timeout: 5m
|
|
smtp_smarthost: "smtp.gmail.com:587" # operator hand-off (e.g. smtp.gmail.com:587)
|
|
smtp_from: "[email protected]" # operator hand-off (From: address)
|
|
smtp_auth_username: "[email protected]" # operator hand-off (SMTP login)
|
|
smtp_auth_password_file: /etc/alertmanager/secrets/alertmanager-smtp/smtp-password
|
|
smtp_require_tls: true
|
|
route:
|
|
group_by: ["alertname", "namespace"]
|
|
group_wait: 30s
|
|
group_interval: 5m
|
|
repeat_interval: 12h
|
|
receiver: email-warning
|
|
routes:
|
|
- matchers: ['severity = "critical"']
|
|
receiver: email-critical
|
|
# page-worthy — notify fast, remind often
|
|
group_wait: 10s
|
|
repeat_interval: 1h
|
|
receivers:
|
|
- name: email-warning
|
|
email_configs:
|
|
- to: "[email protected]" # alert inbox (operator: change if a different inbox is wanted)
|
|
send_resolved: true
|
|
headers:
|
|
subject: '[WARNING] {{ .CommonLabels.alertname }} ({{ .CommonLabels.namespace }})'
|
|
- name: email-critical
|
|
email_configs:
|
|
- to: "[email protected]" # alert inbox (operator: change if a different inbox is wanted)
|
|
send_resolved: true
|
|
headers:
|
|
subject: '[CRITICAL] {{ .CommonLabels.alertname }} ({{ .CommonLabels.namespace }})'
|
|
|
|
# --- Grafana: dashboards (R-OBS-03), surfaced in the C2 UI (docs/12) behind Keycloak SSO ---
|
|
grafana:
|
|
enabled: true
|
|
# Force a pod roll when the OIDC wiring changes so Grafana loads the grafana-oidc-credentials
|
|
# client-secret (env valueFrom is resolved at pod start; the secret was added after first boot, so
|
|
# generic_oauth started disabled). Bump this value to re-roll Grafana on future OIDC secret changes.
|
|
podAnnotations:
|
|
homelab.keithjackson.dev/oidc-rollout: "1"
|
|
# Admin bootstrap + (later) OIDC client are SOPS secrets referenced BY NAME (WS-SEC/ADR-0014).
|
|
admin:
|
|
existingSecret: grafana-admin-credentials
|
|
userKey: admin-user
|
|
passwordKey: admin-password
|
|
# DEV TRIM: emptyDir (default). Dashboards are provisioned from ConfigMaps (sidecar), so no
|
|
# durable Grafana DB is needed to iterate. Prod can enable persistence.
|
|
persistence:
|
|
enabled: false
|
|
# Auto-discover dashboard ConfigMaps labeled grafana_dashboard across namespaces (per-service
|
|
# dashboards rendered from ServiceDefinitions land here — docs/10 onboarding output).
|
|
sidecar:
|
|
dashboards:
|
|
enabled: true
|
|
searchNamespace: ALL
|
|
label: grafana_dashboard
|
|
datasources:
|
|
enabled: true
|
|
resources:
|
|
requests:
|
|
cpu: 50m
|
|
memory: 384Mi # Grafana 13.x (image 13.2.0, kps 88) baseline is well above Grafana 11.x (kps 65)
|
|
limits:
|
|
memory: 768Mi # 256Mi OOMKilled (exit 137) Grafana 13.x during apiserver/unified-storage
|
|
# startup — the silent SIGKILL that made the whole 65->88 Helm upgrade
|
|
# time out (the new Grafana pod never went Ready). 512Mi then ran pinned at
|
|
# ~98% (499Mi) with no headroom — one render/GC spike from OOM. 768Mi holds
|
|
# with real headroom (nodes 24Gi, ~14Gi free each).
|
|
# --- Keycloak native-OIDC SSO (ADR-0014, deliverable #4 SSO proof) ---
|
|
# Grafana is a confidential OIDC client of the Keycloak `homelab` realm. The client SECRET comes
|
|
# from the SOPS-encrypted grafana-oidc-credentials Secret via env (referenced by name only,
|
|
# R-IDP-04). optional:true so the RUNNING Grafana keeps starting before the operator fills the
|
|
# secret — local admin login (below) stays available as the fallback until SSO is proven.
|
|
envValueFrom:
|
|
GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET:
|
|
secretKeyRef:
|
|
name: grafana-oidc-credentials
|
|
key: client-secret
|
|
optional: true
|
|
grafana.ini:
|
|
server:
|
|
root_url: https://grafana.${BASE_DOMAIN}
|
|
auth.generic_oauth:
|
|
enabled: true
|
|
name: Keycloak
|
|
client_id: grafana
|
|
scopes: "openid email profile"
|
|
auth_url: https://keycloak.${BASE_DOMAIN}/realms/homelab/protocol/openid-connect/auth
|
|
token_url: https://keycloak.${BASE_DOMAIN}/realms/homelab/protocol/openid-connect/token
|
|
api_url: https://keycloak.${BASE_DOMAIN}/realms/homelab/protocol/openid-connect/userinfo
|
|
login_attribute_path: preferred_username
|
|
email_attribute_path: email
|
|
# Map Keycloak groups (the `groups` claim) to Grafana roles (R-IDP-06): platform-admins ->
|
|
# Admin, media-users -> Editor, everyone else -> Viewer.
|
|
role_attribute_path: "contains(groups[*], 'platform-admins') && 'Admin' || contains(groups[*], 'media-users') && 'Editor' || 'Viewer'"
|
|
# Keep local admin login enabled as break-glass until SSO is proven end-to-end; flip to
|
|
# disable_login_form:true afterwards (docs/15 break-glass, R-IDP-05).
|
|
# TODO(WS-SEC, post-proof): set auth.disable_login_form=true once SSO login is verified.
|
|
|
|
# --- Bundled exporters / operator: keep small ---
|
|
prometheusOperator:
|
|
resources:
|
|
requests:
|
|
cpu: 50m
|
|
memory: 96Mi
|
|
limits:
|
|
memory: 256Mi
|
|
kube-state-metrics:
|
|
# Expose the egress-gateway NODE LABEL as a metric label so alerts can ask "does this cluster
|
|
# have a VPN egress gateway at all?". KSM exposes NO custom labels by default, so without this
|
|
# `kube_node_labels` does not exist and any join against it silently matches nothing.
|
|
# Consumed by `VPNTunnelDown` (platform/monitoring/alerts.yaml): the alert is armed only on a
|
|
# cluster that actually has a labelled gateway node, and ARMS ITSELF the moment one is labelled
|
|
# — which is how prod (no gateway today) stops emailing a permanently-firing critical alert
|
|
# without anyone having to remember to re-enable it later. Scoped to one label on one resource:
|
|
# 3 extra series, no cardinality concern. Label name is sanitized by KSM to
|
|
# `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`).
|
|
metricLabelsAllowlist:
|
|
- "nodes=[homelab.keithjackson.dev/egress-gateway]"
|
|
# --- VELERO SCHEDULE OBJECTS AS METRICS (added 2026-09-13) ---------------------------------
|
|
# Velero exports NOTHING that enumerates Schedule objects and nothing that exposes
|
|
# `spec.paused`. Verified on prod against velero v1.18.1: all 35 velero_* metric names that
|
|
# carry a `schedule` label are created by a backup ATTEMPT, so a Schedule that has never run
|
|
# has NO series at all — a throwaway `paused: true` Schedule produced zero lines in
|
|
# `curl velero:8085/metrics`. Consequence: `VeleroNoRecentBackup`
|
|
# (`time() - velero_backup_last_successful_timestamp{schedule!=""}`) was structurally INERT for
|
|
# a never-successful schedule, because `time() - <absent series>` yields no series. prod sat
|
|
# with `velero-daily` (the VOLUME-DATA schedule) paused and unbacked-up, and nothing alerted.
|
|
#
|
|
# No PromQL can fix that — the fact is not in the TSDB — so the Schedule CRs themselves are
|
|
# exported here. KSM's custom-resource-state collector is the right mechanism precisely
|
|
# because it is ALREADY the platform's CRD-to-metrics path (this same block exports
|
|
# kube_node_labels for VPNTunnelDown); it needs no new exporter, image, ServiceMonitor or
|
|
# scrape job, and the series land on the existing kube-state-metrics target.
|
|
#
|
|
# Consumed by VeleroScheduleNeverBackedUp / VeleroSchedulePaused / VeleroNoVolumeDataBackup
|
|
# (platform/monitoring/alerts.yaml). Cost: 4 series per Schedule — 8 on prod today.
|
|
#
|
|
# Chart mechanics, verified by rendering kube-prometheus-stack 88.6.5 (KSM subchart 8.4.1,
|
|
# KSM v2.20.0) and by running that exact KSM image against prod's API:
|
|
# * `customResourceState.enabled` mounts this `config` as a ConfigMap at
|
|
# /etc/customresourcestate/config.yaml and adds --custom-resource-state-config-file.
|
|
# `--custom-resource-state-only` is NOT set, so the standard kube_* metrics keep flowing.
|
|
# * the chart auto-adds the apiextensions.k8s.io/customresourcedefinitions list+watch rule
|
|
# (rbac.customResourceState.createClusterRoleRules, default true). That rule is MANDATORY,
|
|
# not cosmetic: without it KSM logs "customresourcedefinitions ... is forbidden" and emits
|
|
# the CRS metrics as an EMPTY SET — silent, exactly the inert-alert failure mode again.
|
|
# Measured: the first probe run, with velero.io access but no CRD access, produced 0 series.
|
|
# * `rbac.extraRules` below is the only RBAC this repo must supply (read-only, one CRD).
|
|
# * `metricNamePrefix` deliberately yields `kube_velero_schedule_*`, NOT `velero_*`: these are
|
|
# kube-state-metrics series about Kubernetes objects, and must not be mistaken for
|
|
# something Velero itself exports.
|
|
# * `nilIsZero` matters — a Schedule that simply omits `spec.paused` (the chart renders
|
|
# `velero-daily-resources-only` that way) must still publish paused=0 rather than nothing,
|
|
# or `== 1` would be evaluated against a missing series.
|
|
# * the KSM Deployment carries NO checksum annotation over this ConfigMap, so a change to
|
|
# `config` alone does not restart the pod: after editing it, `kubectl -n monitoring rollout
|
|
# restart deploy/kube-prometheus-stack-kube-state-metrics`. Enabling it the first time
|
|
# changes the pod spec (new arg + volume), so the initial rollout is automatic.
|
|
rbac:
|
|
extraRules:
|
|
- apiGroups: ["velero.io"]
|
|
resources: ["schedules"]
|
|
verbs: ["get", "list", "watch"]
|
|
customResourceState:
|
|
enabled: true
|
|
config:
|
|
spec:
|
|
resources:
|
|
- groupVersionKind:
|
|
group: velero.io
|
|
version: v1
|
|
kind: Schedule
|
|
metricNamePrefix: kube_velero_schedule
|
|
labelsFromPath:
|
|
# `schedule` (not `name`) so these join 1:1 with Velero's own `schedule` label.
|
|
schedule: [metadata, name]
|
|
namespace: [metadata, namespace]
|
|
metrics:
|
|
- name: created
|
|
help: "Unix creation time of the Velero Schedule object."
|
|
each:
|
|
type: Gauge
|
|
gauge:
|
|
# KSM parses the RFC3339 timestamp to a unix float (verified live).
|
|
# This is what gives VeleroScheduleNeverBackedUp a grace period, so a
|
|
# newly added schedule is not alerted before its first run is even due.
|
|
path: [metadata, creationTimestamp]
|
|
- name: paused
|
|
help: "1 if the Velero Schedule is paused and creating no backups."
|
|
each:
|
|
type: Gauge
|
|
gauge:
|
|
path: [spec, paused]
|
|
nilIsZero: true
|
|
- name: protects_volume_data
|
|
help: "1 if the Schedule captures pod volume DATA, not just Kubernetes objects."
|
|
each:
|
|
type: Gauge
|
|
gauge:
|
|
# THE anti-masking signal: read from the field that defines it, so a
|
|
# resources-only schedule can never stand in for a volume-data one.
|
|
path: [spec, template, defaultVolumesToFsBackup]
|
|
nilIsZero: true
|
|
- name: info
|
|
help: "Descriptive metadata for the Velero Schedule (cron, phase)."
|
|
each:
|
|
type: Info
|
|
info:
|
|
# NOTE: KSM OMITS a label whose path is absent (a Schedule with no
|
|
# status.phase yet yields no `phase` label at all), so label sets differ
|
|
# between series here. Context for dashboards — never join on this metric.
|
|
labelsFromPath:
|
|
cron: [spec, schedule]
|
|
phase: [status, phase]
|
|
# MEMORY RAISED 2026-09-13 after 91 OOMKills. Measured, not guessed: working set 111 MiB
|
|
# with a 6h peak of 123.3 MiB against the old 128Mi limit — KSM was dying on every spike,
|
|
# roughly every 11 minutes (exit 137). That matters beyond the restarts: THIRTEEN alert
|
|
# rules read kube_* series, so each kill made them intermittently inert, and an alert that
|
|
# is only sometimes able to fire is the quiet version of the inert-alert class this repo
|
|
# keeps getting bitten by (KubePodNotReady and KubeDeploymentReplicasMismatch were both
|
|
# stuck pending on it). The trigger was enabling customResourceState below for the Velero
|
|
# Schedule metrics — it adds a CRD informer plus the mandatory CRD list/watch, and that
|
|
# marginal cost crossed a limit already sitting ~4 MiB above the working set.
|
|
# 384Mi is ~3x the observed peak and costs nothing here: the nodes report 18.7-22.3 GiB
|
|
# available. Re-measure with
|
|
# max_over_time(container_memory_working_set_bytes{container="kube-state-metrics"}[6h])
|
|
# before trimming it back.
|
|
resources:
|
|
requests:
|
|
cpu: 20m
|
|
memory: 192Mi
|
|
limits:
|
|
memory: 384Mi
|
|
prometheus-node-exporter:
|
|
# --- NFS OBSERVABILITY (added 2026-09-14) -------------------------------------------------
|
|
# WHY. Five in-tree NFS PVs on 192.168.2.102 carry the whole media library and the legacy
|
|
# Nextcloud dataset, and I/O PSI counts an NFS stall as an I/O stall. That is why prod-1's
|
|
# 20.98% PSI spike at 02:40Z on 2026-09-14 looked inexplicable from the block layer: devices
|
|
# at 0.09-0.21 utilisation, 1.4-4.8 ms per write, zero NFS retransmissions. The one signal
|
|
# that moved was 114 NFS reads/s on prod-1 against 0 on the other two nodes.
|
|
#
|
|
# WHAT IS ALREADY THERE, SO NOBODY "ENABLES" IT TWICE. The `nfs` collector is ON BY DEFAULT
|
|
# and its series are confirmed present on prod (node_nfs_rpcs_total, node_nfs_requests_total
|
|
# by method/proto, node_nfs_rpc_retransmissions_total — all three nodes, 2026-09-14). It is
|
|
# host-wide, read from /proc/net/rpc/nfs, which works because this DaemonSet is hostNetwork.
|
|
# It gives op COUNTS and retransmissions. It gives NO LATENCY and no per-mount breakdown, so
|
|
# it can say "the server dropped requests" but not "the export is slow".
|
|
#
|
|
# WHAT THIS ARGUMENT ADDS. `mountstats` reads /proc/.../mountstats and emits per-mount,
|
|
# per-operation counters including cumulative RPC queue time and round-trip time, which
|
|
# divided by the request count is the per-operation NFS latency this platform has never had.
|
|
# That is the missing half of the I/O-cause question: with it, "PSI is high and the export is
|
|
# answering slowly" and "PSI is high because WE are asking for a lot" become distinguishable,
|
|
# and so does the case where neither holds and the load is coming from the Proxmox media
|
|
# stack that still has operational control of the same files.
|
|
#
|
|
# COST, MEASURED-ADJACENT NOT ASSUMED. mountstats is per-mount and per-operation, so it is
|
|
# the highest-cardinality collector in node-exporter. NFSv4.1 exposes ~70 operations and the
|
|
# collector emits ~8 counters per operation plus ~15 mount-level series, so roughly 575
|
|
# series per mounted export, and the mounts are per POD (kubelet mounts an in-tree NFS PV
|
|
# once per pod), so they churn on every restart. With five exports that is ~2.9k series
|
|
# against a head of 451,650 measured on prod 2026-09-14 — about 0.6%, inside the 3Gi
|
|
# Prometheus limit and the 45GiB retentionSize. Re-measure after rollout with
|
|
# count({__name__=~"node_mountstats_.*"})
|
|
# and if it lands materially above ~5k series, drop the collector rather than growing the PVC.
|
|
#
|
|
# VERIFY BEFORE TRUSTING — THIS MAY LAND INERT, AND THAT MUST BE CHECKED NOT ASSUMED.
|
|
# node-exporter runs with --path.procfs=/host/proc and hostPID, so it reads ITS OWN mount
|
|
# namespace, not the host's. The kubelet-created NFS mounts should still be visible because
|
|
# the `root` volume is a hostPath of / with mountPropagation: HostToContainer, which
|
|
# propagates them in — they will appear with mount points prefixed /host/root/var/lib/kubelet/
|
|
# (the `export` label, e.g. 192.168.2.102:/mnt/Storage/plex, is the stable thing to aggregate
|
|
# by). The node-exporter image is distroless with no shell, so this could not be tested from
|
|
# outside the collector; note that node_filesystem_* currently reports only ext4 and tmpfs,
|
|
# which is explained by the kubelet mount-point exclusion above but is NOT proof either way.
|
|
# AFTER THIS ROLLS OUT, RUN:
|
|
# count({__name__=~"node_mountstats_.*"}) -> must be non-zero
|
|
# count(node_mountstats_nfs_operations_requests_total) by (export)
|
|
# If it is empty, REVERT this argument — a collector that is on and produces nothing is the
|
|
# same lie as an alert rule on a metric nobody scrapes. No alert rule references
|
|
# node_mountstats_* yet, deliberately, for exactly that reason; the per-export latency rule
|
|
# is the follow-up once the series are confirmed, and runbooks/node-io.md records the gap.
|
|
extraArgs:
|
|
- --collector.mountstats
|
|
resources:
|
|
requests:
|
|
cpu: 20m
|
|
memory: 32Mi
|
|
limits:
|
|
# mountstats parses a per-mount, per-operation table on every scrape. 64Mi was sized for
|
|
# the default collector set; raised with headroom for that parse rather than discovering
|
|
# the ceiling as an OOMKill on the exporter that every node alert depends on.
|
|
memory: 96Mi
|