jakman85andClaude Opus 5
a3f4ddb971
Brings the two alerts for the silent egress-gateway failure: VPNEgressSteeringStale
(critical) and VPNEgressUplinkLabelMissing (warning), plus the kube-state-metrics
allowlist entry that makes them non-inert.
WHY THE BRANCH SAT IN THE BACKLOG: it could not pass the merge gate. Three
whole-repo checks failed on it, and each one was a real gap rather than a nuisance:
* alert-runbooks-resolvable — both alerts link to /runbooks/vpn-egress-stale,
which did not exist. An alert firing now would hand the operator a 404.
* alert-rules-unit-tested — alerts.yaml is already a covered file, so new rules
in it must be tested or a "tested" file quietly grows untested rules.
* alert-metrics-have-scrape-targets — see the guard bug below.
WHAT THE ALERTS CATCH. egress-class=vpn pods leave through one gateway node's tun0;
Cilium resolves it to an ifindex and caches it. An OpenVPN reconnect destroys and
re-creates tun0 with a NEW ifindex, and the node label
homelab.keithjackson.dev/egress-uplink=<addr>-<ifindex> is what forces Cilium to
re-resolve. If the label does not follow the tunnel, the cache points at a destroyed
interface and every vpn-class pod is silently dropped — while tun0 stays up and
VPNTunnelDown says nothing. That ran ~80 minutes undetected on the dev gateway.
Verified this is live on prod, not inert: k3s-prod-3 carries egress-gateway=true and
egress-uplink=10.150.192.86-139, and vpn-egress-steering has been in place 14 days.
The label was exported on 0 series before this change, so the allowlist entry is
what switches the alerts on.
runbooks/vpn-egress-stale.md — written and wired into the configMapGenerator, with
the four required operator sections. It leads with proving egress from a pod the
policy selects, because the symptom is "outbound traffic silently disappears", and
it says plainly not to "fix" this by deleting the policy: that does not restore
egress, it sends vpn-class traffic out of the real ISP connection, which is the leak
the class exists to prevent.
platform/monitoring/tests/vpn-egress.test.yaml — four promtool fixtures. 1 and 2 are
the SAME tunnel reconnect with and without the label following it, which isolates the
mechanism the rule actually keys on: a label VALUE change is not a changing sample,
it retires one series and starts another. 3 is the inert case (gateway node, no uplink
label) where the critical rule CANNOT fire, which is why the warning exists.
Two fixes the branch needed that were defects in the checks themselves:
1. alert-metrics-have-scrape-targets asserted the KSM allowlist matched
`nodes=[...egress-gateway]` with a literal closing bracket — i.e. that the list
contained EXACTLY that one label. Adding a second label correctly (egress-uplink,
which these alerts need) therefore FAILED the check while the property it guards
was perfectly satisfied. Loosened to `[,\]]`: in the list, not the whole list. A
guard that breaks when you correctly extend the thing it guards teaches people to
edit the guard, which is how it stops guarding.
2. UNTESTED_ALERTS listed VPNTunnelDown, and the check flagged it as stale because
"asserted on" is a bare word match over the test file — my header COMMENT
mentioning VPNTunnelDown was enough to count as coverage. Rather than delete the
register entry and claim coverage that did not exist, the fixtures now assert
VPNTunnelDown QUIET in all four cases. That required making them faithful: they
had no node_network_receive_bytes_total{device="tun0"} at all, so VPNTunnelDown
would have been firing throughout unnoticed, because promtool only checks the
alertnames you name. A healthy tun0 passes traffic, and that silence IS the claim
these rules exist for.
NOTE FOR LATER, not fixed here: that word-match means any alert named in prose counts
as tested. Tightening it to `alertname:` positions could reveal other prose-only
"coverage", and the register is shrink-only, so the cascade wants doing deliberately
rather than at 01:00 unattended.
Conflicts resolved: kept HEAD's Velero custom-resource-state block and took only the
branch's one-line allowlist change (14 lines down to 1); inserted the branch's
2026-09-12 DECISIONS-LOG entry in date order rather than appending it out of sequence.
Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips,
0 unwired. The new unit tests pass under the pinned promtool 3.14.0.
Co-Authored-By: Claude Opus 5 <[email protected]>
422 lines
25 KiB
YAML
422 lines
25 KiB
YAML
---
|
|
# kube-prometheus-stack — Prometheus + Alertmanager + Grafana + kube-state-metrics +
|
|
# node-exporter + the standard Kubernetes exporters/rules (docs/10). Serves R-OBS-01/02/03.
|
|
#
|
|
# Retention/PVC sizes are overlay knobs (R-ENV-02) resolved from cluster-vars via Flux
|
|
# postBuild.substituteFrom. Everything else is DEV-SCOPED for the ~22 GiB workstation ceiling:
|
|
# single replicas, modest requests/limits, ephemeral (emptyDir) Alertmanager/Grafana state, and
|
|
# the k3s-embedded control-plane scrape jobs disabled (they are not separately reachable on k3s and
|
|
# would otherwise show as permanently-down targets and burn RAM/alert noise).
|
|
apiVersion: helm.toolkit.fluxcd.io/v2
|
|
kind: HelmRelease
|
|
metadata:
|
|
name: kube-prometheus-stack
|
|
namespace: monitoring
|
|
spec:
|
|
interval: 30m
|
|
# The 65->88 catch-up is a large multi-major jump (operator + KSM + node-exporter roll, plus the
|
|
# Prometheus/Alertmanager StatefulSets). The default 5m Helm timeout is too short for the full rollout
|
|
# to reach Ready, so Flux logged "context deadline exceeded" and rolled back to 65.8.1 four times.
|
|
# A generous timeout lets one attempt finish; remediateLastFailure:false keeps partial progress instead
|
|
# of yo-yoing back to the old chart (a re-reconcile then drives it forward). CRDs are pre-applied
|
|
# server-side (Helm never updates crds/ on upgrade), so schema is already at operator v0.93.1.
|
|
timeout: 20m0s
|
|
chart:
|
|
spec:
|
|
chart: kube-prometheus-stack
|
|
version: ">=88.0.0 <89.0.0" # k8s-1.36 catch-up (po v0.93; CRDs pre-applied server-side, Helm won't update them). Renovate maintains
|
|
sourceRef:
|
|
kind: HelmRepository
|
|
name: prometheus-community
|
|
namespace: monitoring
|
|
install:
|
|
remediation:
|
|
retries: 3
|
|
upgrade:
|
|
remediation:
|
|
retries: 2
|
|
remediateLastFailure: false
|
|
cleanupOnFail: true
|
|
values:
|
|
# --- k3s trim: control-plane components are embedded in the k3s server process and are not
|
|
# scrapable as discrete endpoints. Disabling avoids permanently-firing "target down" noise.
|
|
kubeControllerManager:
|
|
enabled: false
|
|
kubeScheduler:
|
|
enabled: false
|
|
kubeEtcd:
|
|
enabled: false
|
|
kubeProxy:
|
|
enabled: false
|
|
|
|
# --- Prometheus ---
|
|
prometheus:
|
|
prometheusSpec:
|
|
replicas: 1
|
|
retention: ${MONITORING_PROMETHEUS_RETENTION} # overlay knob (dev 3d / prod long)
|
|
# SIZE-based retention cap (docs/10, R-OBS). `retention` alone prunes only by AGE — on a young
|
|
# cluster (< retention window) NOTHING is pruned yet, so a busy TSDB fills the PVC before the
|
|
# time window elapses. A full /prometheus wedges ALL WAL commits ("no space left on device"):
|
|
# scrapes still 200 but samples are never stored, the head freezes, and every absent()/==0
|
|
# platform alert (Keycloak/KubeAPI/Kubelet/VPN/KubeVirt/CDI…) fires falsely because instant
|
|
# queries at "now" return empty. retentionSize makes Prometheus drop the OLDEST blocks once the
|
|
# block set approaches this cap, so the disk never fills regardless of cluster age or churn.
|
|
# Kept BELOW the PVC size (leaves headroom for the WAL + head + compaction scratch, which are
|
|
# NOT counted by retentionSize). Prometheus size units are *iB (GiB), distinct from the K8s PVC
|
|
# `Gi`, so it is a separate overlay knob rather than a reuse of the PVC-size knob.
|
|
retentionSize: ${MONITORING_PROMETHEUS_RETENTION_SIZE} # overlay knob (dev 5GiB / base 8GiB / prod 45GiB)
|
|
# Discover ServiceMonitors/PodMonitors/Rules from ALL namespaces (flux, cert-manager,
|
|
# longhorn, velero, keycloak, cilium/hubble ship their own) — not just this release's.
|
|
ruleSelectorNilUsesHelmValues: false
|
|
serviceMonitorSelectorNilUsesHelmValues: false
|
|
podMonitorSelectorNilUsesHelmValues: false
|
|
probeSelectorNilUsesHelmValues: false
|
|
# Modest footprint for a home dev cluster (few hundred series). Limit gives headroom for
|
|
# WAL replay / compaction spikes without letting Prometheus eat the node.
|
|
resources:
|
|
requests:
|
|
cpu: 100m
|
|
memory: 512Mi
|
|
limits:
|
|
memory: 3Gi # 1Gi OOMKilled (exit 137) during TSDB compaction/checkpoint;
|
|
# 2Gi ran pinned at ~98% (2002Mi) with no room for compaction/WAL-replay
|
|
# spikes (the classic Prometheus OOM). Nodes are 24Gi at 38-44% used
|
|
# (~14Gi free each), so 3Gi gives real headroom.
|
|
storageSpec:
|
|
volumeClaimTemplate:
|
|
spec:
|
|
storageClassName: longhorn # replicated tier (ADR-0003)
|
|
accessModes: ["ReadWriteOnce"]
|
|
resources:
|
|
requests:
|
|
storage: ${MONITORING_PROMETHEUS_PVC_SIZE} # overlay knob (dev 5Gi / prod large)
|
|
|
|
# --- Alertmanager: routes by severity to email (R-OBS-02) ---
|
|
alertmanager:
|
|
alertmanagerSpec:
|
|
replicas: 1
|
|
# DEV TRIM: no PVC — silences/notification-log live in emptyDir. Acceptable in dev (a
|
|
# restart re-fires active alerts). Prod overlay can add a small storage block.
|
|
resources:
|
|
requests:
|
|
cpu: 10m
|
|
memory: 32Mi
|
|
limits:
|
|
memory: 96Mi
|
|
# SMTP credentials are an OPERATOR HAND-OFF: a SOPS-encrypted Secret, referenced BY NAME
|
|
# only (docs/07). Mounted at /etc/alertmanager/secrets/<name>/ so the config below can read
|
|
# the password from a file (never inline plaintext).
|
|
secrets:
|
|
- alertmanager-smtp
|
|
# Routing/receivers. Non-secret SMTP endpoint fields (smarthost/from/to/username) are
|
|
# operator-supplied placeholders (REPLACE-ME, same convention as velero); the PASSWORD is
|
|
# read from the mounted SOPS secret via *_file. Both severities email; subject/route differ.
|
|
config:
|
|
global:
|
|
resolve_timeout: 5m
|
|
smtp_smarthost: "smtp.gmail.com:587" # operator hand-off (e.g. smtp.gmail.com:587)
|
|
smtp_from: "[email protected]" # operator hand-off (From: address)
|
|
smtp_auth_username: "[email protected]" # operator hand-off (SMTP login)
|
|
smtp_auth_password_file: /etc/alertmanager/secrets/alertmanager-smtp/smtp-password
|
|
smtp_require_tls: true
|
|
route:
|
|
group_by: ["alertname", "namespace"]
|
|
group_wait: 30s
|
|
group_interval: 5m
|
|
repeat_interval: 12h
|
|
receiver: email-warning
|
|
routes:
|
|
- matchers: ['severity = "critical"']
|
|
receiver: email-critical
|
|
# page-worthy — notify fast, remind often
|
|
group_wait: 10s
|
|
repeat_interval: 1h
|
|
receivers:
|
|
- name: email-warning
|
|
email_configs:
|
|
- to: "[email protected]" # alert inbox (operator: change if a different inbox is wanted)
|
|
send_resolved: true
|
|
headers:
|
|
subject: '[WARNING] {{ .CommonLabels.alertname }} ({{ .CommonLabels.namespace }})'
|
|
- name: email-critical
|
|
email_configs:
|
|
- to: "[email protected]" # alert inbox (operator: change if a different inbox is wanted)
|
|
send_resolved: true
|
|
headers:
|
|
subject: '[CRITICAL] {{ .CommonLabels.alertname }} ({{ .CommonLabels.namespace }})'
|
|
|
|
# --- Grafana: dashboards (R-OBS-03), surfaced in the C2 UI (docs/12) behind Keycloak SSO ---
|
|
grafana:
|
|
enabled: true
|
|
# Force a pod roll when the OIDC wiring changes so Grafana loads the grafana-oidc-credentials
|
|
# client-secret (env valueFrom is resolved at pod start; the secret was added after first boot, so
|
|
# generic_oauth started disabled). Bump this value to re-roll Grafana on future OIDC secret changes.
|
|
podAnnotations:
|
|
homelab.keithjackson.dev/oidc-rollout: "1"
|
|
# Admin bootstrap + (later) OIDC client are SOPS secrets referenced BY NAME (WS-SEC/ADR-0014).
|
|
admin:
|
|
existingSecret: grafana-admin-credentials
|
|
userKey: admin-user
|
|
passwordKey: admin-password
|
|
# DEV TRIM: emptyDir (default). Dashboards are provisioned from ConfigMaps (sidecar), so no
|
|
# durable Grafana DB is needed to iterate. Prod can enable persistence.
|
|
persistence:
|
|
enabled: false
|
|
# Auto-discover dashboard ConfigMaps labeled grafana_dashboard across namespaces (per-service
|
|
# dashboards rendered from ServiceDefinitions land here — docs/10 onboarding output).
|
|
sidecar:
|
|
dashboards:
|
|
enabled: true
|
|
searchNamespace: ALL
|
|
label: grafana_dashboard
|
|
datasources:
|
|
enabled: true
|
|
resources:
|
|
requests:
|
|
cpu: 50m
|
|
memory: 384Mi # Grafana 13.x (image 13.2.0, kps 88) baseline is well above Grafana 11.x (kps 65)
|
|
limits:
|
|
memory: 768Mi # 256Mi OOMKilled (exit 137) Grafana 13.x during apiserver/unified-storage
|
|
# startup — the silent SIGKILL that made the whole 65->88 Helm upgrade
|
|
# time out (the new Grafana pod never went Ready). 512Mi then ran pinned at
|
|
# ~98% (499Mi) with no headroom — one render/GC spike from OOM. 768Mi holds
|
|
# with real headroom (nodes 24Gi, ~14Gi free each).
|
|
# --- Keycloak native-OIDC SSO (ADR-0014, deliverable #4 SSO proof) ---
|
|
# Grafana is a confidential OIDC client of the Keycloak `homelab` realm. The client SECRET comes
|
|
# from the SOPS-encrypted grafana-oidc-credentials Secret via env (referenced by name only,
|
|
# R-IDP-04). optional:true so the RUNNING Grafana keeps starting before the operator fills the
|
|
# secret — local admin login (below) stays available as the fallback until SSO is proven.
|
|
envValueFrom:
|
|
GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET:
|
|
secretKeyRef:
|
|
name: grafana-oidc-credentials
|
|
key: client-secret
|
|
optional: true
|
|
grafana.ini:
|
|
server:
|
|
root_url: https://grafana.${BASE_DOMAIN}
|
|
auth.generic_oauth:
|
|
enabled: true
|
|
name: Keycloak
|
|
client_id: grafana
|
|
scopes: "openid email profile"
|
|
auth_url: https://keycloak.${BASE_DOMAIN}/realms/homelab/protocol/openid-connect/auth
|
|
token_url: https://keycloak.${BASE_DOMAIN}/realms/homelab/protocol/openid-connect/token
|
|
api_url: https://keycloak.${BASE_DOMAIN}/realms/homelab/protocol/openid-connect/userinfo
|
|
login_attribute_path: preferred_username
|
|
email_attribute_path: email
|
|
# Map Keycloak groups (the `groups` claim) to Grafana roles (R-IDP-06): platform-admins ->
|
|
# Admin, media-users -> Editor, everyone else -> Viewer.
|
|
role_attribute_path: "contains(groups[*], 'platform-admins') && 'Admin' || contains(groups[*], 'media-users') && 'Editor' || 'Viewer'"
|
|
# Keep local admin login enabled as break-glass until SSO is proven end-to-end; flip to
|
|
# disable_login_form:true afterwards (docs/15 break-glass, R-IDP-05).
|
|
# TODO(WS-SEC, post-proof): set auth.disable_login_form=true once SSO login is verified.
|
|
|
|
# --- Bundled exporters / operator: keep small ---
|
|
prometheusOperator:
|
|
resources:
|
|
requests:
|
|
cpu: 50m
|
|
memory: 96Mi
|
|
limits:
|
|
memory: 256Mi
|
|
kube-state-metrics:
|
|
# Expose the egress-gateway NODE LABEL as a metric label so alerts can ask "does this cluster
|
|
# have a VPN egress gateway at all?". KSM exposes NO custom labels by default, so without this
|
|
# `kube_node_labels` does not exist and any join against it silently matches nothing.
|
|
# Consumed by `VPNTunnelDown` (platform/monitoring/alerts.yaml): the alert is armed only on a
|
|
# cluster that actually has a labelled gateway node, and ARMS ITSELF the moment one is labelled
|
|
# — which is how prod (no gateway today) stops emailing a permanently-firing critical alert
|
|
# without anyone having to remember to re-enable it later. Scoped to one label on one resource:
|
|
# 3 extra series, no cardinality concern. Label name is sanitized by KSM to
|
|
# `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`).
|
|
metricLabelsAllowlist:
|
|
- "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]"
|
|
# --- VELERO SCHEDULE OBJECTS AS METRICS (added 2026-09-13) ---------------------------------
|
|
# Velero exports NOTHING that enumerates Schedule objects and nothing that exposes
|
|
# `spec.paused`. Verified on prod against velero v1.18.1: all 35 velero_* metric names that
|
|
# carry a `schedule` label are created by a backup ATTEMPT, so a Schedule that has never run
|
|
# has NO series at all — a throwaway `paused: true` Schedule produced zero lines in
|
|
# `curl velero:8085/metrics`. Consequence: `VeleroNoRecentBackup`
|
|
# (`time() - velero_backup_last_successful_timestamp{schedule!=""}`) was structurally INERT for
|
|
# a never-successful schedule, because `time() - <absent series>` yields no series. prod sat
|
|
# with `velero-daily` (the VOLUME-DATA schedule) paused and unbacked-up, and nothing alerted.
|
|
#
|
|
# No PromQL can fix that — the fact is not in the TSDB — so the Schedule CRs themselves are
|
|
# exported here. KSM's custom-resource-state collector is the right mechanism precisely
|
|
# because it is ALREADY the platform's CRD-to-metrics path (this same block exports
|
|
# kube_node_labels for VPNTunnelDown); it needs no new exporter, image, ServiceMonitor or
|
|
# scrape job, and the series land on the existing kube-state-metrics target.
|
|
#
|
|
# Consumed by VeleroScheduleNeverBackedUp / VeleroSchedulePaused / VeleroNoVolumeDataBackup
|
|
# (platform/monitoring/alerts.yaml). Cost: 4 series per Schedule — 8 on prod today.
|
|
#
|
|
# Chart mechanics, verified by rendering kube-prometheus-stack 88.6.5 (KSM subchart 8.4.1,
|
|
# KSM v2.20.0) and by running that exact KSM image against prod's API:
|
|
# * `customResourceState.enabled` mounts this `config` as a ConfigMap at
|
|
# /etc/customresourcestate/config.yaml and adds --custom-resource-state-config-file.
|
|
# `--custom-resource-state-only` is NOT set, so the standard kube_* metrics keep flowing.
|
|
# * the chart auto-adds the apiextensions.k8s.io/customresourcedefinitions list+watch rule
|
|
# (rbac.customResourceState.createClusterRoleRules, default true). That rule is MANDATORY,
|
|
# not cosmetic: without it KSM logs "customresourcedefinitions ... is forbidden" and emits
|
|
# the CRS metrics as an EMPTY SET — silent, exactly the inert-alert failure mode again.
|
|
# Measured: the first probe run, with velero.io access but no CRD access, produced 0 series.
|
|
# * `rbac.extraRules` below is the only RBAC this repo must supply (read-only, one CRD).
|
|
# * `metricNamePrefix` deliberately yields `kube_velero_schedule_*`, NOT `velero_*`: these are
|
|
# kube-state-metrics series about Kubernetes objects, and must not be mistaken for
|
|
# something Velero itself exports.
|
|
# * `nilIsZero` matters — a Schedule that simply omits `spec.paused` (the chart renders
|
|
# `velero-daily-resources-only` that way) must still publish paused=0 rather than nothing,
|
|
# or `== 1` would be evaluated against a missing series.
|
|
# * the KSM Deployment carries NO checksum annotation over this ConfigMap, so a change to
|
|
# `config` alone does not restart the pod: after editing it, `kubectl -n monitoring rollout
|
|
# restart deploy/kube-prometheus-stack-kube-state-metrics`. Enabling it the first time
|
|
# changes the pod spec (new arg + volume), so the initial rollout is automatic.
|
|
rbac:
|
|
extraRules:
|
|
- apiGroups: ["velero.io"]
|
|
resources: ["schedules"]
|
|
verbs: ["get", "list", "watch"]
|
|
customResourceState:
|
|
enabled: true
|
|
config:
|
|
spec:
|
|
resources:
|
|
- groupVersionKind:
|
|
group: velero.io
|
|
version: v1
|
|
kind: Schedule
|
|
metricNamePrefix: kube_velero_schedule
|
|
labelsFromPath:
|
|
# `schedule` (not `name`) so these join 1:1 with Velero's own `schedule` label.
|
|
schedule: [metadata, name]
|
|
namespace: [metadata, namespace]
|
|
metrics:
|
|
- name: created
|
|
help: "Unix creation time of the Velero Schedule object."
|
|
each:
|
|
type: Gauge
|
|
gauge:
|
|
# KSM parses the RFC3339 timestamp to a unix float (verified live).
|
|
# This is what gives VeleroScheduleNeverBackedUp a grace period, so a
|
|
# newly added schedule is not alerted before its first run is even due.
|
|
path: [metadata, creationTimestamp]
|
|
- name: paused
|
|
help: "1 if the Velero Schedule is paused and creating no backups."
|
|
each:
|
|
type: Gauge
|
|
gauge:
|
|
path: [spec, paused]
|
|
nilIsZero: true
|
|
- name: protects_volume_data
|
|
help: "1 if the Schedule captures pod volume DATA, not just Kubernetes objects."
|
|
each:
|
|
type: Gauge
|
|
gauge:
|
|
# THE anti-masking signal: read from the field that defines it, so a
|
|
# resources-only schedule can never stand in for a volume-data one.
|
|
path: [spec, template, defaultVolumesToFsBackup]
|
|
nilIsZero: true
|
|
- name: info
|
|
help: "Descriptive metadata for the Velero Schedule (cron, phase)."
|
|
each:
|
|
type: Info
|
|
info:
|
|
# NOTE: KSM OMITS a label whose path is absent (a Schedule with no
|
|
# status.phase yet yields no `phase` label at all), so label sets differ
|
|
# between series here. Context for dashboards — never join on this metric.
|
|
labelsFromPath:
|
|
cron: [spec, schedule]
|
|
phase: [status, phase]
|
|
# MEMORY RAISED 2026-09-13 after 91 OOMKills. Measured, not guessed: working set 111 MiB
|
|
# with a 6h peak of 123.3 MiB against the old 128Mi limit — KSM was dying on every spike,
|
|
# roughly every 11 minutes (exit 137). That matters beyond the restarts: THIRTEEN alert
|
|
# rules read kube_* series, so each kill made them intermittently inert, and an alert that
|
|
# is only sometimes able to fire is the quiet version of the inert-alert class this repo
|
|
# keeps getting bitten by (KubePodNotReady and KubeDeploymentReplicasMismatch were both
|
|
# stuck pending on it). The trigger was enabling customResourceState below for the Velero
|
|
# Schedule metrics — it adds a CRD informer plus the mandatory CRD list/watch, and that
|
|
# marginal cost crossed a limit already sitting ~4 MiB above the working set.
|
|
# 384Mi is ~3x the observed peak and costs nothing here: the nodes report 18.7-22.3 GiB
|
|
# available. Re-measure with
|
|
# max_over_time(container_memory_working_set_bytes{container="kube-state-metrics"}[6h])
|
|
# before trimming it back.
|
|
resources:
|
|
requests:
|
|
cpu: 20m
|
|
memory: 192Mi
|
|
limits:
|
|
memory: 384Mi
|
|
prometheus-node-exporter:
|
|
# --- NFS OBSERVABILITY (added 2026-09-14) -------------------------------------------------
|
|
# WHY. Five in-tree NFS PVs on 192.168.2.102 carry the whole media library and the legacy
|
|
# Nextcloud dataset, and I/O PSI counts an NFS stall as an I/O stall. That is why prod-1's
|
|
# 20.98% PSI spike at 02:40Z on 2026-09-14 looked inexplicable from the block layer: devices
|
|
# at 0.09-0.21 utilisation, 1.4-4.8 ms per write, zero NFS retransmissions. The one signal
|
|
# that moved was 114 NFS reads/s on prod-1 against 0 on the other two nodes.
|
|
#
|
|
# WHAT IS ALREADY THERE, SO NOBODY "ENABLES" IT TWICE. The `nfs` collector is ON BY DEFAULT
|
|
# and its series are confirmed present on prod (node_nfs_rpcs_total, node_nfs_requests_total
|
|
# by method/proto, node_nfs_rpc_retransmissions_total — all three nodes, 2026-09-14). It is
|
|
# host-wide, read from /proc/net/rpc/nfs, which works because this DaemonSet is hostNetwork.
|
|
# It gives op COUNTS and retransmissions. It gives NO LATENCY and no per-mount breakdown, so
|
|
# it can say "the server dropped requests" but not "the export is slow".
|
|
#
|
|
# WHAT THIS ARGUMENT ADDS. `mountstats` reads /proc/.../mountstats and emits per-mount,
|
|
# per-operation counters including cumulative RPC queue time and round-trip time, which
|
|
# divided by the request count is the per-operation NFS latency this platform has never had.
|
|
# That is the missing half of the I/O-cause question: with it, "PSI is high and the export is
|
|
# answering slowly" and "PSI is high because WE are asking for a lot" become distinguishable,
|
|
# and so does the case where neither holds and the load is coming from the Proxmox media
|
|
# stack that still has operational control of the same files.
|
|
#
|
|
# COST, MEASURED-ADJACENT NOT ASSUMED. mountstats is per-mount and per-operation, so it is
|
|
# the highest-cardinality collector in node-exporter. NFSv4.1 exposes ~70 operations and the
|
|
# collector emits ~8 counters per operation plus ~15 mount-level series, so roughly 575
|
|
# series per mounted export, and the mounts are per POD (kubelet mounts an in-tree NFS PV
|
|
# once per pod), so they churn on every restart. With five exports that is ~2.9k series
|
|
# against a head of 451,650 measured on prod 2026-09-14 — about 0.6%, inside the 3Gi
|
|
# Prometheus limit and the 45GiB retentionSize. Re-measure after rollout with
|
|
# count({__name__=~"node_mountstats_.*"})
|
|
# and if it lands materially above ~5k series, drop the collector rather than growing the PVC.
|
|
#
|
|
# VERIFY BEFORE TRUSTING — THIS MAY LAND INERT, AND THAT MUST BE CHECKED NOT ASSUMED.
|
|
# node-exporter runs with --path.procfs=/host/proc and hostPID, so it reads ITS OWN mount
|
|
# namespace, not the host's. The kubelet-created NFS mounts should still be visible because
|
|
# the `root` volume is a hostPath of / with mountPropagation: HostToContainer, which
|
|
# propagates them in — they will appear with mount points prefixed /host/root/var/lib/kubelet/
|
|
# (the `export` label, e.g. 192.168.2.102:/mnt/Storage/plex, is the stable thing to aggregate
|
|
# by). The node-exporter image is distroless with no shell, so this could not be tested from
|
|
# outside the collector; note that node_filesystem_* currently reports only ext4 and tmpfs,
|
|
# which is explained by the kubelet mount-point exclusion above but is NOT proof either way.
|
|
# AFTER THIS ROLLS OUT, RUN:
|
|
# count({__name__=~"node_mountstats_.*"}) -> must be non-zero
|
|
# count(node_mountstats_nfs_operations_requests_total) by (export)
|
|
# If it is empty, REVERT this argument — a collector that is on and produces nothing is the
|
|
# same lie as an alert rule on a metric nobody scrapes. No alert rule references
|
|
# node_mountstats_* yet, deliberately, for exactly that reason; the per-export latency rule
|
|
# is the follow-up once the series are confirmed, and runbooks/node-io.md records the gap.
|
|
#
|
|
# extraArgs REPLACES, IT DOES NOT APPEND — and the chart's DEFAULT for this key is not empty.
|
|
# prometheus-node-exporter's own values.yaml ships two filesystem excludes in `extraArgs`
|
|
# (confirmed against the chart, and against the live DaemonSet's argv on prod). Listing only
|
|
# `--collector.mountstats` here would silently DROP both, so the filesystem collector would
|
|
# start reporting every container overlay and every kubelet bind mount: a large cardinality
|
|
# jump and a behaviour change in NodeDiskFillingUp, which alerts per mountpoint. Both
|
|
# defaults are therefore restated verbatim below. If the chart's defaults change on an
|
|
# upgrade, this list must be re-reconciled against
|
|
# helm show values prometheus-community/prometheus-node-exporter | grep -A3 '^extraArgs:'
|
|
extraArgs:
|
|
- --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|run/containerd/.+|var/lib/docker/.+|var/lib/kubelet/.+)($|/)
|
|
- --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs|erofs)$
|
|
- --collector.mountstats
|
|
resources:
|
|
requests:
|
|
cpu: 20m
|
|
memory: 32Mi
|
|
limits:
|
|
# mountstats parses a per-mount, per-operation table on every scrape. 64Mi was sized for
|
|
# the default collector set; raised with headroom for that parse rather than discovering
|
|
# the ceiling as an OOMKill on the exporter that every node alert depends on.
|
|
memory: 96Mi
|